OnPath Testing

Please describe your company and your position there.

CodeLogic is building the worlds first comprehensive system of record for tracking and storing relationships and dependencies between Applications, Data Stores, Infrastructure and Platforms. This includes discovering, in fine-grained detail, application structure, dynamic relationships and other complex systemic elements, and surfacing them in easy to understand ways that deliver value and drive action within development and IT organizations. As CTO, I'm responsible for the technical strategy, development and delivery of the product. As such, comprehensive, continuous test, as part of our DevOps pipeline, is exceptionally important.

For what projects/services did your company hire OnPath Testing?

As CodeLogic is a significantly complex technical undertaking, it was important for our development organization to have a well defined, structured approach to Quality Assurance and Testing. OnPath, on an ongoing basis, provides our core managed test team, including the development of automated test processes, test definitions and project management for these activities.

How did you select this vendor and what were the deciding factors?

Having worked successfully with OnPath in the past, and knowing the depth of their expertise, both in technical testing and in managing test infrastructure development, OnPath was an obvious choice for us, given our complex demands.

Describe the project in detail and walk through the stages of the project.

The project, defined across multiple phases, extends current unit and integration testing, with automated functional test harness development across our core product and agent frameworks. In latter phases we are extending comprehensive API based testing, to address simulated user acceptance testing. Additionally, this is being done in conjunction with DevOps CI/CB/CT/CD (CT being continuous test) integration. This includes eventual continuous comprehensive test automation across feature branches, in conjunction with in-line unit and automation tests that are already in place.

How many resources from the vendor's team worked with you, and what were their positions?

We have worked with 2 test development engineers, 2 test engineers and 1 team project manager.

Can you share any outcomes from the project that demonstrate progress or success?

In conjunction with unit and integration testing, the automated continuous functional test platform, implemented by OnPath, has provided critical insight and significant QA improvements. This is exceptionally important as CodeLogic delivers releases every two weeks, using a disciplined Continuous Delivery model. OnPath is one of the few Test Development, Automation and Outsourcing providers I'd trust to work in such a demanding environment.

How effective was the workflow between your team and theirs?

Given our tight, continuous delivery pattern, hard deadlines and highly technical product, OnPath has provided excellent, tightly coupled workflow and execution with our development engineering teams.

What did you find most impressive or unique about this company?

The quality of the resources, sourced from both North America and Asia, has been consistently impressive. Additionally, with team/project management located in North America, OnPath has managed to effectively bridge issues with the (usually) very challenging time-zone issues exhibited by other service providers. Finally, OnPath (in both my experiences working with them) requires remarkably little corrective oversight. Provided structured touchpoints and lines of communications are established, they get the job done with a minimum of fuss.

Are there any areas for improvement or something they could have done differently?

My only recommendation would be to broaden their offerings into core DevOps process development, specifically with an eye toward earlier integration of comprehensive testing as part of Continuous Integration and Build. That said, this isn't a current deficiency, but a recommendation due to the evolving nature of development processes.

Introduce your business and what you do there.

I am the director of product management at ClearCaptions, a transcription service for people that are hard of hearing.

What challenge were you trying to address with OnPath Testing?

We needed OnPath Testing to test our software applications and hardware to confirm that our phones were meeting the product requirements, both from feature presentation and from a nonfunctional perspective as well. At the time, we were a part of another company called Purple Communications. We were in a transition period and developing our own solution.

What was the scope of their involvement?

It has really been a test, fix, and recast process where everything is done in a ticketing system where we document various cases and acceptance criteria.  Every week we typically have a build that is coming from our development team and then heads to the test team. They validate it based on the issues and acceptance criteria that are included in that sprint.

Depending on where we are in the development cycle, if it’s a released candidate, for instance, then we will go through regression testing and they will document everything within that same Jira ticket and then the process repeats. We have modified this process overtime from trying to communicate across externals versus using an internal resource, which has been very different.

What is the team composition?

I have worked a lot with Brian (President & Chief QA Engineer, OnPath Testing) and 3–4 testers. Over the last few years, we have had anywhere from 2–8 testers. 

How did you come to work with OnPath Testing?

I originally found them online. We reached out to various companies and did interviews to understand their process, how we integrate cost quotes, and sent out an RFE (Request for Evidence). Ultimately, OnPath Testing was the one that we gelled with most and in terms of both understanding the scope as well as culturally. This process took a few months.

How much have you invested with them?

We spend roughly $500,000–$1,000,000 annually. So, we have spent $1 million–$2 million in total. 

What is the status of this engagement?

We have been working together since January 2017 and the relationship is currently ongoing.

What evidence can you share that demonstrates the impact of the engagement? 

They’ve enabled us to get the product out the door and to catch issues and make assessments on whether we want to fix something now or later. We are also able to pinpoint, address, and resolve issues that we are seeing in the field very quickly.

How did OnPath Testing perform from a project management standpoint

They are able to focus on this particular product, which is really good. There are never distractions in terms of deprioritizing us based on other responsibilities they may have. They immediately take care of what we need and get it turned around. They’ve exceeded my expectations in regard to timing; they are always on time. With testers that are overseas, we communicate via Jira issue tracking, Confluence, or email. We had a little bit of a rocky start because we got a new project manager, but the relationship has integrated.

What did you find most impressive about them?

Their flexibility has been really beneficial. Brian and I have sat down and gone through product requirements, how various features and functionalities work and finding the acceptance criteria around those, and then how we will test them. It has been really great to reach out and talk through things. It’s a lot more interaction and collaboration. My impression with other companies was they wanted everything predefined, so we really appreciated the collaboration with OnPath Testing and being integrated with our teams.

Are there any areas they could improve?

There are some individuals in the team that haven’t kept up with the changes from our product requirements, so there’s some work that needs to be done there. There’s been some turnover here and there, but it’s been managed fairly well. 

Do you have any advice for potential customers?

Spending a lot of time working on the frontend, defining the process such as test cases, and understanding what test suite you’re going to use is important. Communicate the metrics you capture in order to understand how those cadences are going to work. Examine the frontend in the beginning so that you can just tweak the relationship as it continues.

Please describe your organization.

We are a Scrum Team working on developing software products for online registration of athletic events. This can include, for example, the New York City Marathon. Registration for the event is typically done online currently. We provide the necessary software for it. We also provide scoring systems for these races.

What is your position?

I am the product owner.

What business challenge were you trying to address with OnPath Testing?

The business goal was making sure that we had sufficient regression testing for the code we had developed. We initially used Selenium for the process, trying to automate the testing for our registration and scoring systems.

Please describe the scope of their involvement.

OnPath Testing wrote test scripts in Selenium and Capybara for us. We moved to the latter because we needed Ruby code support. They became familiar enough with our systems to be able to write tests scripts themselves, without the need of doing step-by-step scenarios.

OnPath built our entire test suite for Selenium in 2013, over the span of three to six months. They continued to write automated tests as we developed new features, in order to make sure that everything was covered. In late 2014, our automated test suite became too cumbersome, so we decided to switch from Selenium to Cucumber. This involved rewriting all of our application tests. OnPath were the ones who undertook the task in its entirety.

They had to learn the new technology, as they didn't have past Cucumber experience. They not only did this, but they also learned to write for Capybara, the Ruby code that runs Cucumber. It was a long project, but they stuck to it. The total time was about six months for complete code coverage. They continued to write new parts as we released new features.

OnPath was willing to learn new technologies, which we seriously appreciated. We had a remote relationship. The dev team we worked with is based in India.

How did you come to work with OnPath Testing?

They were referred to us by a collaborator who had worked with them in the past.

Could you provide a sense of the size of this initiative in financial terms?

The total expense for our collaboration was around $50,000.

What is the status of this engagement?

The collaboration ended in December 2015. Our company is moving to a different direction, and the Scrum developers are now responsible for writing tests. Ending our relationship had nothing to do with OnPath Testing’s performance.

We only gave them two weeks’ notice of the termination, and OnPath Testing continued the work up until the last day. It was impressive to see this attitude.

Could you share any statistics or metrics from this engagement?

We consider the relationship with OnPath to have been a very good investment. 

How did OnPath Testing perform from a project management standpoint?

We found them very reliable and great collaborators. They don’t need to be micromanaged and they’ve always been able to deliver the specs requested of them.

What distinguishes OnPath Testing from other providers?

We appreciate their desire to learn new technologies and their ability to put in long hours. OnPath Testing worked harder than most of our employees, which was impressive.

Is there anything OnPath Testing could have improved or done differently?

They were very willing to learn new technologies, but it took them a while to obtain the domain knowledge necessary to actually write out the test scenarios themselves. I had to take care of this for longer than I would have wanted.

Please describe your organization.

We work in the endurance event space, and we were building an online registration tool to facilitate transactions for participants in the events.

What is your position?

I was a project manager when we brought OnPath Testing onboard. 

What business challenge were you trying to address with OnPath Testing?

We were building out our integration tests, trying to improve them. We needed more manpower to help us ramp up the creation of specific quality assurance tests. Then, in time, we changed the technologies that we were using to write the tests. We knew this would add significantly to our time. We leveraged OnPath to help guide us to new systems and help build the foundation of those systems. They restructured our testing environment and kept us on pace with our growth. 

Please describe the scope of their involvement.

We moved to Cucumber, which is what they're using right now. Before that, we had a couple changes. We were always using Jenkins server, then we migrated, and now we're using Cucumber tests. OnPath helped us with that migration and took all of our old tests and rewrote them so they could be utilized again with the new system.

Our software developers also write tests. But, we needed a starting point and we wanted our software developers to focus on writing tests for new code in features recently developed. OnPath helped us catch up with the existing code that was not covered at the moment. They became an extension of our team.

We've used just two of their resources, continuously. They were like full-time members of our team. Since we started our relationship, it's always been full-time with those two. We never used anyone else.

We didn't have a choice regarding who joined our project team. That was up to OnPath. At one point, they brought in a project manager who was going to help and add to that and after working with him for a couple weeks it wasn't a good fit so we just kept with the two testing engineers. It was easier for them to work as members of our team than be managed through an intermediary. 

How did you come to work with OnPath Testing?

One of our senior software engineers had worked with Brian [Borg], the founder of OnPath Testing. He vouched for Brian. We spoke to Brian and decided we would give OnPath a shot starting out with just some basics, nothing too major. I'll say we were in our infancy in terms of our understanding as to what we truly needed. We didn't have a well-defined project scope when we first engaged with them. Since then, they have grown with us and gone through some growing pains with us.

Could you provide a sense of the size of this initiative in financial terms?

They were being billed out the two engineers each for $35,000 to $40,000 a year, which was higher than we would've expected. That's why we relied on them as full-time team members.

Could you share any statistics or metrics from this engagement?

We went from weekly to every other week releases. We were working on a live web application. We handled about $300,000 in transactions a day. We wanted to go to a continuous employment setup, but we didn't have the testing infrastructure to do so. As a result, we went from very large releases every two weeks that we littered with regressions to a rapid release to anywhere between three and seven deployments to production a day with a regression rate of less than 3 percent. That was all based on the work they helped facilitate. They also led the charge in training our software developers and bringing them up to speed. Our reliability increased a great deal.

Our own developers are more than impressed with the engineers we have from OnPath Testing. I don't know if we would switch those engineers if we would be as confident. Those two engineers became part of the team. They worked more than your 40 hours a week just as if they were members of the team. That was fantastic. I got the sense that we were the first company that brought them in as part of the team. The energy and enthusiasm they brought was awesome. It was a very good relationship. I know this was a higher price point than most people would expect to pay for offshore testing. We brought in a new vice president of technology about a year ago. That was one of the first things they said. IT [information technology] seemed expensive. Our software developers love working with them, though. There's a little bit of a learning curve to start, but once they're familiar with the process and standards, things to very smoothly. Our project managers love working with them, too. They're very consistent. 

How did OnPath Testing perform from a project management standpoint?

We managed everything through JIRA. OnPath Testing is well versed in agile development. I don't know if the whole organization is or not, but it's worked very well from our perspective. We never had any issues with delayed work or anything like that.

Is there anything OnPath Testing could have improved or done differently?

I don't have great judgment as to the value of the entire organization. I know these two test engineers are fantastic. One time they brought in a new person, and it wasn't a good fit. That project manager didn't work out. I can't really attest to whether or not other members would have produced the exact same results as the two we have. There were several times in which Brian offered to bring other people on board to help. But, we were satisfied with the people we had and the amount of output we were getting, so we felt like it would have been unnecessary. 

Do you have any suggestions for future clients of theirs?

We use offshore resources a lot and for us it's always been about making them feel like an integral part of the team. Have them partake in the wins and losses of the team. We've gotten more out of our contractors than I've seen other organizations just because they were part of our culture. Just get them as involved as possible so they understand the context around the tasks and can take some degree of ownership of the project success.

Please describe your organization.

We have a web platform that helps language teachers use the web for what's called blended learning, or hybrid learning. This is where part of the traditional language teaching class, part of it is the normal face to face in the classroom, and then part of it is online, where the student works on their own. It's not a virtual classroom where you're connecting with a teacher online, rather it's what they call a learning management system, where the student does different tasks to work with the language that they're learning.

The platform is a learning management system and a content repository. We license content from publishers in the English language teaching field, and then we do some fancy stuff with the content, including disaggregating it so that teacher can take bits and pieces from different courses and recombine them into a learning sequence that is personalized for each student. There's a learning management aspect to the platform, there is a content repository, there are authoring tools, and then there are some social features, where students can collaborate together on projects, and that type of thing.

We sell student subscriptions to language schools and universities. We don't sell directly to students. This is not a "learn on your own product" like Rosetta Stone would be. Rather, it's for language schools, universities, and even corporate training departments. There is a tremendous amount of corporate language training that goes on as well. It's technology geared to help teachers do a better job as opposed to technology that's supposed to replace teachers. It's a typical software as a service business model. We sell student subscriptions. It's about $12 per month, per student, and most of our customers are in Europe and Latin America. We're just getting started in Asia, and we have a fair amount in the Middle East as well.

What is your position?

I am the founder and CEO.

What business challenge were you trying to address with OnPath Testing?

We went through quite a dramatic evolution in terms of the coding talent that we were hiring, which might be typical in situations where you have a non-coder as the CEO because I'm not a tech guy. As a result, it was difficult for me to judge talent and manage the project. We went through some ups and downs with different firms that were building our software and then we finally decided to go ahead and just build our own.

We were working with our lead engineer at the time, a gentleman by the name of Aaron Zakowski, a very qualified and competent engineer. He recognized that we needed some good, hardcore quality assurance and brought testing in as an integral part of the processes we use to build our product, and we ended up hiring Brian [Borg, Founder at OnPath Testing] and his firm.

Please describe the scope of their involvement.

The development of our actual system was done in-house as well as with the help of other people we hired. We didn't get any really decent results until we hired our own. That's not to say it's the only way you can do it. It just worked out that way. I did hire a couple of firms and, in both cases, it did not work out in a satisfactory way. It was our in-house team that ended up doing it, although they inherited some buggy legacy code from the firms that had started on the project.

In a way, a lot of the development ended up being an undoing of what people had done before them. It was through that that we ended up working with Brian. Although he headed up the OnPath firm, at that point, he only had maybe five or six guys. But, Brian was able to fit in as a part of the internal team somehow. I think that was one of the reasons that it worked out so well with him. He was very good at the communication side of things, understanding the dynamic of the software team, and how to fulfill his responsibilities in a way that kept the whole thing moving along nicely.

Reworking some of the buggy legacy code was a nightmare. We used a rapid development framework, and had mixed results with a couple developers that we hired, and finally decided to hire the firm that invented this rapid development framework, thinking they would be great. All they wanted to do was deliver software that was so abstracted that they could then reuse it for other projects, and it was just nightmarish. There were just layers and layers on things. This meant that you had a core element that you could reuse for any project in the future, regardless of context or specific use. Once we cut ties with them and hired our in-house team, they had to pull some of those layers of abstraction out, so that everything ran a lot more efficiently.

OnPath was integrated directly into our development cycle. They would participate in the sprints we would run. We might also take some time and have a period with more comprehensive testing on a specific new piece that we were releasing. Brian was on all the stand-ups we would have each day, and he worked not only with our lead engineer but also with all the other coders on our team.

We did primarily performance and functional testing. We didn't do a lot of load testing. We really didn't do a lot of usability testing, so it was mostly just making sure that things were working the way they should. We had some complicated stuff happening on these web pages with the authoring tools. The authoring tools produce a range of almost 20 different question templates, including some drag-and-drop stuff, picture labeling, and stuff like that. Those are aspects characteristic of language teaching, but tough to actually implement in HTML and JavaScript.

Brian would work directly with our lead engineer, looking at some of the basic architectural questions as well. That's probably less traditional testing, but more of just his feedback on anticipating directions we didn't want to take. He could say, "Hey, I can see that causing problems down the road." That's an example of him being involved directly in the team in a way that was very much not how you would traditionally view a vendor, especially one running their firm.

Anyway, we didn't do a lot of load testing. It wasn't automated. A couple of times we were going to go ahead and put in some automated testing tools, and then, for a variety of reasons, and usually just the immense time pressure we were under, we decided not to actually do that. We were talking about whether we should go with Selenium, but it ended up being manual. Brian was actually the one pushing for that because he said, "Guys, let's do this right." I think I kept overruling him, to his displeasure, but he worked well with everybody, even though sometimes as part of the team he wasn't running the show.

How did you come to work with OnPath Testing?

I'm not sure. My guess is that we put something in Stack Overflow. Or, maybe it wasn't. I don't even know if Stack Overflow was around back then. This would have been 2009, so it was something like that. This was mostly run by our own lead engineer, and I wasn't really involved with the decision myself. Aaron probably interviewed two or three different guys. He's a very tough and demanding interviewer. I remember him being enthusiastic about hiring OnPath.

Could you provide a sense of the size of this initiative in financial terms?

We probably spent around a quarter of a million dollars annually.

What is the status of this engagement?

The last milestone completed was about 18 months ago. At the time, we were half-owned by Cambridge University in the United Kingdom. Just after I launched and got a beta version of my product, they came along and liked the technology, so we set up a joint venture that was 50/50. For a variety of reasons, the university publisher, Cambridge University Press, decided to go in a different direction in terms of technology. We had a very amicable separation about a year and a half ago. It was at that point that we dramatically downsized our team, and we don't have a dedicated testing unit right now. When we get going again, we will absolutely be calling Brian right away.

Could you share any statistics or metrics from this engagement?

I don't have anything specific to offer you. There wasn't really a before and after because he was there building it from scratch. I can't tell you anything on page load times, uptime, or anything like that. In terms of specific numbers, I can't really help you out there.

In terms of internal stakeholders, I do know that occasionally we would pull in people from the university, subject matter experts on something and have them be involved in certain projects where they would have actually been in meetings with Brian there. They were external to the company, but Brian has a way of getting to the crux of an issue very calmly in a way that doesn't step on anyone's toes. He is very diplomatic. He comes in very calm, gets right to the essence of the matter, and unravels the knots, so to speak. I do remember our Cambridge partners occasionally saying, "Wow, that guy really contributed to that meeting." These were sometimes technical people at the university as well because we had a couple joint projects with them.

In terms of customers, we have a reputation for doing some complicated stuff in a way that works. The platform is very stable, and we do some fancy stuff that really no other platform does. If you look at other learning management systems and the authoring tools that they come with, ours is really second to none in terms of the complexity of the types of tasks that we let the end-users create by themselves. Many times, if you go in and custom build something, you can make some fantastic learning tasks on the web, but we went beyond that and created a template that allowed any user to do it. We had a bottom-up, user-generated content element that's critical and a big differentiator for us as well.

That stuff just works, so in terms of customer reputation I think you can see Brian's work in that recognition of just the fact that it does work. People really trust English360 a lot because their whole work processes run through English360. If you're a small language school or a university, and you're delivering blended learning to these customers, years' worth of your work is inside the English360 platform in terms of the content that you've created. There's a trustworthiness that's important there and, if we didn't have that, we wouldn't be successful as a business. The fact that we do have it is a testament to the work that Brian did.

What distinguishes OnPath Testing from other providers?

One of the reasons that Brian is so good is he has a very high level of emotional intelligence. There are many good QA people out there, but what sets Brian apart is just fantastic communication skills. He's a great teammate; he's always very steady, and he calms everybody down. That was hugely beneficial for us because we were under a lot of pressure to release our partners at the time. We needed something out there in the market very quickly, and Brian played a big role in getting that done.

Is there anything OnPath Testing could have improved or done differently?

The only possible thing I could say on that is not anything that Brian needed to do better, that OnPath would do better, it's just that we got used to working with Brian personally, so that when we would have one of his guys jump in, I would email Brian and say, "Dude, we need you." Sometimes, Brian and I would go back and forth on this, he'd say, "I've got other stuff going on as well. I'm trying to do as much as I can to be dedicated to English360, but I do have other customers." At the end of the day, he has a business to run, too. It's understandable, but it's also a bummer because he was such a huge asset to us.

As it turned out, his guys are fantastic. I forget the other guy we used to work with a lot, who reported to Brian inside OnPath. He was a very high-quality tester. What Brian brought to everything was that emotional intelligence aspect of it. That dedication and expertise that he brings became essential to the whole build, so if he couldn't be in a meeting, we felt the absence. I have no idea about the guys now. Like I said, I haven't worked with Brian or his team in more than a year now. I don't know who he's got in place now.

That really isn't so much an improvement area, it's just recognizing that, as an individual, Brian has something special when it comes to the human side of software development, and when that wasn't there, we missed it. I guess that's saying that he just needs to clone himself to populate his company. Other than that, Brian was great to work with.

Please describe your organization.

We're an online e-commerce retailer that is focused specifically around manufacturing nutritional supplements. We sell everything directly to the consumer through online marketing and direct response marketing.

What is your position and responsibilities?

I am the chief technology officer, so I oversee all of the software and IT [information technology] aspects of the business.

What business challenge were you trying to address with OnPath Testing?

In general, as a small startup company, we use contractors in various forms for surge capacity, such as if I need additional hands for a project or even just to fill a staffing need for an unspecified duration. I need some flexibility with it, so I'll possibly be able to control that spend again. We partnered with OnPath to be able to add those contingent resources to the QA [quality assurance] process.

Please describe the scope of their involvement.

The work they did involved QA [quality assurance] for desktop and tablet and mobile manual testing for user experience and functional QA – general web-based testing. The team we were allocated ranged from one full-time person to three at the height. We have two major areas of development work internally. One focused around marketing initiatives, which is page presentation, conversion analysis, and split test work, and then another on more core back-end development work.

Mostly, I have focused OnPath on supporting the marketing end of things because the turnaround time needs to be much tighter with those releases. However, at the height, we had one person also associated with what I consider core development project effort. We didn't use a project manager from OnPath Testing. We managed the resources and assign their tasks directly. Not myself, but actually the QA manager on my team.

How did you come to work with OnPath Testing?

Brian [Borg], the founder of the company, was actually on staff as a direct, individual contributor and contract resource when I came on board. Through working with him, we hit a point where we needed additional resources and, at that point, he said, "I need to step away. The business is growing well, and I need to step away from the hands-on testing and really do more project and resource management pieces." At that point, we put somebody in place of his work, and when we were looking to broaden the work, it was a natural progression to talk to him about using OnPath across the board for my contingent QA needs.

Could you provide a sense of the size of this initiative in financial terms?

In the last 24 months, most of the time or the whole time, I've had at least one engineer, 40 hours a week. For probably half of that time, I had two engineers. You could call it two people for 24 months, and their billing rate is around $35 an hour.

What is the status of this engagement?

We've been using them for two years in various forms, with different numbers of people.

Could you share any statistics or metrics from this engagement?

I've been doing software engineering for close to 20 years, and in only two of the positions did I not use either contract QA or contract development work. Compared to other collaborations I've had, I actually rank OnPath highly. I would recommend them without hesitation. The nice thing in my experience with Brian is that he's very good at being flexible with how we wanted to manage the team. I've worked with companies where their only model was to have a project manager who we had to work through to assign tasks, and it's just been an inefficient way to manage the small numbers of resources we had.

Brian was very flexible about the model that we were using. He was good about taking feedback. At one point, we had a test engineer who initially was performing well, and then it seemed like there was a focus issue or something was going on. He was good about addressing that, and then eventually helping us find a different person to fill that role and getting a better resource in place. In general, openness to making a difference and moving on the fly has been great. I'm a huge fan.

We develop new products on about a six-month basis, so about two or three times a year, we're launching what we call a new video mashup. It's a new marketing site focused around whatever the new product line is. Historically, prior to putting a focus on it, ramping up QA and putting OnPath on, and creating the structured QA methodology that we have right now, we would put those launches out, and invariably, immediately we found issues with either page content, incorrect presentation for customers, or problems in completing an order. Frankly, I don't have those issues anymore, in part because of process changes, but also just around being able to add the staffing needed to make sure we have good coverage.

The second piece, from my standpoint, is just responsiveness and turnaround time for projects associated with the marketing team in particular. The ability to add contingent staff, and have that resource layer that I can grow as needed when projects come in, has helped me keep that responsiveness in turnaround time. When I came in the company, it was a significant talking point around how the software engineering team could support the marketing team better, and why we can't get stuff done. That conversation has pretty much ended. In most cases, now we really are beholden to the ability of the marketing team to create the copy that actually needs to go up online, versus being able to get the technical work done and test it.

At an execution level, they've been very focused on the test work. Honestly, that's partly me focusing them in that area, and really driving the strategy for QA process evolution internally. The flipside being, Brian and I had several very good, overarching conversations about where the relationship is going, what's our focus longer term, looking at broadening test automation, whether that's something that OnPath could support and help with. There's definitely strategic insight there, even though I haven't particularly leveraged it in the engagement with them.

How did OnPath Testing perform from a project management standpoint?

The project management tool we use is JIRA, the cloud-based version. We have a very organized  bug-tracking workflow. Brian actually helped establish that as the bug-tracking tool in the company before I came to the company. That was definitely, from my standpoint, one of the nice things that OnPath helped to drive, prior to me coming in.

What distinguishes OnPath Testing from other providers?

Like I said, it has to be their management approach and flexibility during project work. OnPath Testing has been very accommodating to us when things have come up unexpectedly. They're never afraid to give us crucial feedback, and they have made some key recommendations around structuring and improving our QA environment. I've been very grateful for that support.

Is there anything OnPath Testing could have improved or done differently?

The only thing is setting expectations with the contingent staff that they bring in and making sure those are aligned with my own expectations. When we do those product launches that I was talking about, we tend to move into what we call our launch schedule, where we essentially have development and QA staff available – if not online, working, but available – about 20 hours a day, seven days a week to be able to do changes.

We run a lot of A/B split testing at that point in time. At 11 o'clock at night, I need somebody available to rotate into the next test, to make sure that from a testing standpoint everything's functional. We had a couple of bumps with people who didn't understand that expectation upfront, and it was a little bit of a reset. That hasn't happened in the recent past. Brian and I have discussed this, and I think he has kind of taken it to heart.

Please describe your organization.

I am currently a freelance QA [quality assurance] manager. At the time, I was working with OnPath, I was working for an e-learning company.

What is your position?

I was the QA manager.

What business challenge were you trying to address with OnPath Testing?

We did a series of different projects with them. The biggest one for Evolve, which is a publishing and marketplace for medical books and research – a place where students can buy books for classes and teachers can administer quizzes to their students. It's a large platform with a lot of stuff going on. It would take a while to describe it all. That project was probably a period of more than two years or so.

We hired OnPath to provide us with engineers able to test the software's functional reliability and usability. At one time, we probably had as many as eight or nine testers from them, working every day, testing a product and whatnot. It was mostly usability, functional, and compatibility testing. We did hire them to start setting up automated testing as well, and that was a whole separate side thing, but it was for the same product. They provided someone to help get things more automated. The QA phase was integrated directly into the development process before we released the software.

Please describe the scope of their involvement in more detail.

The purpose of the software itself was transactional. It dealt with large volumes of data, and there were interactions between multiple third-party systems. It had e-commerce functionality to buy books, examination functionality to quiz students. Teachers could connect and contact students, it had grading - both the grades for various assignments and weighted grades for the whole class.

As far as the e-commerce side goes, that was connecting the teacher, who would send the book to the student, and the student would buy the book. The teacher would talk to the third-party people to set the book up into the marketplace as well. It was a large, multimillion-dollar product.

They were located in India, and we had a manager of the team that was in India, and I was managing the manager, so to speak. They were working at night, and we were working during the day, so I would talk to the manager and tell him what to do for the day, and he would manage the team there. I would say the manager was more of a senior level, and then all the testers were mid-level. There weren't any pure rookie testers.

How did you come to work with OnPath Testing?

Our vice president of engineering had worked with them in the past through a different company. He already had experience with them and recommended them for this particular project. I believe we were already working with them when we started that project, so they were the only ones we thought of because we were already happy with them.

Could you provide a sense of the size of this initiative in financial terms?

It was during a period of years, and we varied from two or three testers working every day up to 10 working every day. I would say it was somewhere around $60,000 to $100,000 a year for two years.

What is the status of this engagement?

When I left the company, which was about a year and a half ago, we had released the product, but we were still developing it and adding on to it, and still using OnPath to test the add-ons, fixes, and everything. As far as I know, they're still working on it. I'm not with that company anymore, so I don't know exactly, but I know the product is still live.

Could you share any statistics or metrics from this engagement?

We were happy with the way they were able to run tests efficiently, and they were able to manage their own team very well. We had them also writing test cases sometimes, and running test cases. Their test cases were written well. Their communication was good. Like I said, it was overseas and it was overnight, so we had to rely on a lot of emails and stuff. We couldn't talk directly very often. They handled that process well.

If we needed more people, they were quick to jump on and get us some more testers. We could ramp up and ramp down quickly, which is why we really liked using them a lot. Our testing needs would change quite often, and they were able to handle all the changes without any headaches or rigmarole. If we needed less, they'd take a couple people off. They were very flexible.

How did OnPath Testing perform from a project management standpoint?

They were able to deliver all their milestones on time and within budget. We were working with the developers, obviously, and the milestones would change quite a bit. With their ability to ramp up and down, we were able to meet milestones on a consistent basis with no problem. We did use JIRA sometimes for project management but, mostly, we interacted by Skype and email.

What distinguishes OnPath Testing from other providers?

I liked how independent OnPath Testing was. I've used overseas staff a lot, and usually I have to do a lot of hand-holding, a lot of phone calls. With them, I could just send an email. It had to be pretty detailed, but the email was enough. I would go for a couple or three weeks without a verbal conversation, and I was working with eight of them every day. I was impressed by their independence. It made them easy to work with, and my time could be put to better use than a lot of hand holding.

Is there anything OnPath Testing could have improved or done differently?

Some of the junior testers didn't have perfect English, but eight out of 10 of them had totally usable English. I was only really talking with the manager, and his English was perfect. The only time I would see language issues would be in the description of a bug by a junior tester, but those were few and far between. Every issue I can think of was just due to them being located overseas. We had a couple meetings with the CEO, he came to D.C. a couple times, but there wasn't a lot of personal interaction because they were overseas.

Please describe your company and your position there.

I'm the CTO of RAIR Technologies INC - an NFT infrastructure and streaming encryption platform.

For what projects/services did your company hire OnPath Testing, and what were your goals?

To review our codebases and ensure our security focused offerings were able to pass a rigorous QA process and integrate into a CI/CD pipeline.

How did you select this vendor and what were the deciding factors?

Brian is an expert in the QA field, very personable, and able to source us great QA talent.

Describe the scope of work in detail, including the project steps, key deliverables, and technologies used.

Full QA pipeline for our company. Sourcing qualified QA engineers. On our dailies frequently to stay on top of our project developments. Advice and expertise on CI/CD workflows, security auditing, and other aspects of the QA process we need help with.

How many people from the vendor's team worked with you, and what were their positions?

3 total. An original QA engineer, our current QA engineer, and a senoir security auditor.

Can you share any measurable outcomes of the project or general feedback about the deliverables?

Stood up the QA department to ensure our product is successful. Implemented testing plan, manual QA, QA automation, Atlassian structure for QA/QC of all developer tickets. Jenkins pipeline for testable deployment.

Describe their project management style, including communication tools and timeliness.

They're responsive, helpful, and proactive.

What did you find most impressive or unique about this company?

Getting to interface with the CEO Brian directly - he's taken a keen interest in blockchain and smart contracting which is really awesome as we can work together on a deeper level given a shared passion for the space.

Are there any areas for improvement or something they could have done differently?

Original QA engineer needed to be replaced which could have transitioned a little faster and smoother but also this was on our end being so early in the development cycle when we brought QA on.

Please describe your company and your position there.

I'm the director of technology at Emerge. Emerge is a digital product agency in Portland, OR. We specialize in strategy, UX and interface design, mobile and web app development.

For what projects/services did your company hire OnPath Testing, and what were your goals?

We brought in OnPath to lead QA for a custom software development project with a tight timeline and complex technical ecosystem.

How did you select this vendor and what were the deciding factors?

OnPath was our choice due to their reputation for exceptional work, excellent collaboration, and adaptability.

Describe the scope of work in detail, including the project steps, key deliverables, and technologies used.

OnPath created a customized testing plan suitable for the project budget, software type, and complexities of the technical ecosystem. OnPath also created test cases appropriate based of user stories and features, and executed quality assurance practices. They provided detailed feedback on bugs or other found issues, drastically improving the final software product.

How many people from the vendor's team worked with you, and what were their positions?

There were three people on the core team, including the owner, the senior QA manager / solutions architect, and a QA analyst.

Can you share any measurable outcomes of the project or general feedback about the deliverables?

OnPath was able to provide excellent QA on budget and on time. The quality of QA feedback was exceptional.

Describe their project management style, including communication tools and timeliness.

OnPath's project management was great. They felt like fully integrated members of our product team, which was key for this project as it featured many shifting variables. Communication was primarily done via video calls, chat tools, Jira, and email. Communication was always timely and clear.

What did you find most impressive or unique about this company?

OnPath Testing's adaptability and collaboration skills were truly impressive. The project we worked on together featured a very tight timeline, shifting requirements, and a complicated technical ecosystem. OnPath was able to bring critical thinking and focus on intent and outcome over output, yet still provide exceptional quality assurance while following best practice.

Are there any areas for improvement or something they could have done differently?

Nope, they were great.

Please describe your company and your position there.

I'm the CTO of a technology company that provides a nationwide platform for vehicle customization.

For what projects/services did your company hire OnPath Testing?

We hired OnPath to close an immediate gap in quality assurance testing for our online product which was (at that point) 1-2 months out from launch.

How did you select this vendor and what were the deciding factors?

I had a personal working relationship with the CEO, Brian Borg, dating back to 1999, when he was a quality assurance engineer and I was a software engineer. Fast forward to today, both being leaders in our respective domains, it was the right time & place to reach back out to him to see what new accomplishments we could achieve together.

Describe the project in detail and walk through the stages of the project.

OnPath helped focus in on the biggest, most immediate needs, in order to bring value to our in-house development team as quickly as possible. That meant providing ad hoc testing against a codebase in development, as well as the initial stages of laying a foundation for automated testing in the months to come.

How many resources from the vendor's team worked with you, and what were their positions?

OnPath provided two resources, a QA engineer, and a QA manager to help coordinate & align the deliverables.

Can you share any outcomes from the project that demonstrate progress or success?

The ad hoc testing provided an immediate gap closure. They put a fresh set of external eyes on our product, providing valuable feedback to identify and eliminate edge conditions we would have missed otherwise. Additionally, the framework for automated testing immediately provided the developers with scripts they could work with and expand upon to improve the quality of their own software.

How effective was the workflow between your team and theirs?

The workflow was extremely flexible; we initially had OnPath's QA engineer integrated directly into the team, participating in stand-ups, processing items ready for testing as they flowed through our JIRA board, adapting as necessary.

What did you find most impressive or unique about this company?

The most impressive part of OnPath is their ability to quickly assess the situation and culture, and consider for both when adapting their solutions. Ours was very informal and conversational, and they read that quickly, and became flexible to meet those same expectations.

Are there any areas for improvement or something they could have done differently?

Nothing that I can think of the moment!