OnPath Testing

Introduce your business and what you do there.

I’m the senior product manager for Leesa Sleep, a mattress company.

What challenge were you trying to address with OnPath Testing?

The main challenge we were looking to tackle was building our QA and testing department. When we started, we didn’t have a formalized QA and testing process or automation, which was challenging because we’d sometimes have bugs or have to do a rollback in our releases. In summary, we didn’t have a lot of confidence in the code we were building with the technology team.

What was the scope of their involvement?

OnPath Testing developed the testing automation platform and the different critical scenarios we needed for verification. They also executed those tests for us and gave us the results. They started by identifying different scenarios to test. Then, we launched different phases of testing automation, introduced the developers to it, and started sending out reports with our release notes. We had several stages throughout the project, including building the platform, creating dozens of critical and non-critical testing scenarios, and setting up reporting.

Initially, we gave them documentation on our systems, platforms, software development process, and business information so they’d know which scenarios were the most critical. They first used JavaScript and then shifted to C#. In the end, we utilized a platform called Cypress for testing.

The platform’s main function was to test our shopping journey on our website to ensure customers could add products to the cart, check out, and engage with any critical tasks during the main shopping journey. OnPath Testing also ensured that the images and pages loaded quickly and appropriately. We also did some scenarios around integrations to ensure none of them could break and the code was stable whenever we did a release.

Additionally, OnPath Testing offered to maintain the platform. Before we finished our engagement, they trained our developers on how to maintain the platform and use it effectively. They showed us how to do everything ourselves.

What is the team composition?

We worked with three teammates for the project, including a QA specialist. We only had up to two resources at the time.

How did you come to work with OnPath Testing?

My boss came across their company. Lon (Director of Operations) and Brian (President) were the initial contacts.

How much have you invested with them?

We spent roughly $100,000. We didn’t invest much in the platform; most of the cost was for the resources.

What is the status of this engagement?

We worked with them from January–November 2022. We paused our service with them due to budgetary issues.

What evidence can you share that demonstrates the impact of the engagement?

When we started, we had bugs and rollbacks, and we didn’t have any at all by the end of the project. OnPath Testing created around 35 scenarios for our testing automation. In the last few releases, we had 100% success on all the critical and non-critical testing scenarios, so we got to a really good place thanks to them.

How did OnPath Testing perform from a project management standpoint?

They performed really well. We often had sync-ups to check on the process; they’d demonstrate the progress they’d made. Their developer was very involved with our team, and we’d also discuss how we could iterate on the project every month or so.

We used monday.com as our software development platform. We mainly communicated through Microsoft Teams, but we sometimes had Zoom calls for the monthly sync-ups.

What did you find most impressive about them?

OnPath Testing was great at customizing their services based on our individual needs. As a small company, they were very in tune with our processes. They looked at our entire digital ecosystem and what we were doing to recommend the most effective processes and strategies to take us from crawling to riding a bike. We’re still not a race car, but we’re doing so much better than before.

Are there any areas they could improve?

Some of the work had to be done a second time because they didn’t act upon all the information that our developers gave them. However, it was a minor bump in the road; we didn’t have many challenges.

Do you have any advice for potential customers?

I suggest involving them in your company’s processes as much as possible. We struggled internally with it for a bit because we usually keep our contracted resources at an arm’s length. However, if you embrace their resources and treat them as internal employees, they’ll understand things better.

Introduce your business and what you do there.

I'm a manager at an NFT marketplace.

What challenge were you trying to address with OnPath Testing?

We needed help with platform testing but didn't have the internal resources.

What was the scope of their involvement?

OnPath Testing has been doing staff augmentation for us to test our platforms.

What is the team composition?

We work with three members of OnPath's team, including two testers and an account manager.

How much have you invested with them?

So far, we've spent about $70,000 with them.

What is the status of this engagement?

The project started in March 2022, and it's still ongoing.

What evidence can you share that demonstrates the impact of the engagement?

They've found bugs diligently and documented them well, so we're impressed with their work.

How did OnPath Testing perform from a project management standpoint?

Project management is great, and we have no complaints; OnPath Testing's team communicates well and delivers everything on time. Regarding tools, we use Slack and Google Meet.

What did you find most impressive about them?

OnPath is very responsive to feedback and questions.

Are there areas they could improve?

I feel that their approach is very corporate, and it could be better to look at it more from a startup perspective that is less caring about minor bugs and more caring about usability.

Any advice for potential customers?

Provide them with all the information needed for the project, and trust them to deliver the best.

Please describe your company and your position there.

CodeLogic is building the worlds first comprehensive system of record for tracking and storing relationships and dependencies between Applications, Data Stores, Infrastructure and Platforms. This includes discovering, in fine-grained detail, application structure, dynamic relationships and other complex systemic elements, and surfacing them in easy to understand ways that deliver value and drive action within development and IT organizations. As CTO, I'm responsible for the technical strategy, development and delivery of the product. As such, comprehensive, continuous test, as part of our DevOps pipeline, is exceptionally important.

For what projects/services did your company hire OnPath Testing?

As CodeLogic is a significantly complex technical undertaking, it was important for our development organization to have a well defined, structured approach to Quality Assurance and Testing. OnPath, on an ongoing basis, provides our core managed test team, including the development of automated test processes, test definitions and project management for these activities.

How did you select this vendor and what were the deciding factors?

Having worked successfully with OnPath in the past, and knowing the depth of their expertise, both in technical testing and in managing test infrastructure development, OnPath was an obvious choice for us, given our complex demands.

Describe the project in detail and walk through the stages of the project.

The project, defined across multiple phases, extends current unit and integration testing, with automated functional test harness development across our core product and agent frameworks. In latter phases we are extending comprehensive API based testing, to address simulated user acceptance testing. Additionally, this is being done in conjunction with DevOps CI/CB/CT/CD (CT being continuous test) integration. This includes eventual continuous comprehensive test automation across feature branches, in conjunction with in-line unit and automation tests that are already in place.

How many resources from the vendor's team worked with you, and what were their positions?

We have worked with 2 test development engineers, 2 test engineers and 1 team project manager.

Can you share any outcomes from the project that demonstrate progress or success?

In conjunction with unit and integration testing, the automated continuous functional test platform, implemented by OnPath, has provided critical insight and significant QA improvements. This is exceptionally important as CodeLogic delivers releases every two weeks, using a disciplined Continuous Delivery model. OnPath is one of the few Test Development, Automation and Outsourcing providers I'd trust to work in such a demanding environment.

How effective was the workflow between your team and theirs?

Given our tight, continuous delivery pattern, hard deadlines and highly technical product, OnPath has provided excellent, tightly coupled workflow and execution with our development engineering teams.

What did you find most impressive or unique about this company?

The quality of the resources, sourced from both North America and Asia, has been consistently impressive. Additionally, with team/project management located in North America, OnPath has managed to effectively bridge issues with the (usually) very challenging time-zone issues exhibited by other service providers. Finally, OnPath (in both my experiences working with them) requires remarkably little corrective oversight. Provided structured touchpoints and lines of communications are established, they get the job done with a minimum of fuss.

Are there any areas for improvement or something they could have done differently?

My only recommendation would be to broaden their offerings into core DevOps process development, specifically with an eye toward earlier integration of comprehensive testing as part of Continuous Integration and Build. That said, this isn't a current deficiency, but a recommendation due to the evolving nature of development processes.

Please describe your company and your position there.

I'm the CTO of RAIR Technologies INC - an NFT infrastructure and streaming encryption platform.

For what projects/services did your company hire OnPath Testing, and what were your goals?

To review our codebases and ensure our security focused offerings were able to pass a rigorous QA process and integrate into a CI/CD pipeline.

How did you select this vendor and what were the deciding factors?

Brian is an expert in the QA field, very personable, and able to source us great QA talent.

Describe the scope of work in detail, including the project steps, key deliverables, and technologies used.

Full QA pipeline for our company. Sourcing qualified QA engineers. On our dailies frequently to stay on top of our project developments. Advice and expertise on CI/CD workflows, security auditing, and other aspects of the QA process we need help with.

How many people from the vendor's team worked with you, and what were their positions?

3 total. An original QA engineer, our current QA engineer, and a senoir security auditor.

Can you share any measurable outcomes of the project or general feedback about the deliverables?

Stood up the QA department to ensure our product is successful. Implemented testing plan, manual QA, QA automation, Atlassian structure for QA/QC of all developer tickets. Jenkins pipeline for testable deployment.

Describe their project management style, including communication tools and timeliness.

They're responsive, helpful, and proactive.

What did you find most impressive or unique about this company?

Getting to interface with the CEO Brian directly - he's taken a keen interest in blockchain and smart contracting which is really awesome as we can work together on a deeper level given a shared passion for the space.

Are there any areas for improvement or something they could have done differently?

Original QA engineer needed to be replaced which could have transitioned a little faster and smoother but also this was on our end being so early in the development cycle when we brought QA on.

Please describe your company and your position there.

I'm the director of technology at Emerge. Emerge is a digital product agency in Portland, OR. We specialize in strategy, UX and interface design, mobile and web app development.

For what projects/services did your company hire OnPath Testing, and what were your goals?

We brought in OnPath to lead QA for a custom software development project with a tight timeline and complex technical ecosystem.

How did you select this vendor and what were the deciding factors?

OnPath was our choice due to their reputation for exceptional work, excellent collaboration, and adaptability.

Describe the scope of work in detail, including the project steps, key deliverables, and technologies used.

OnPath created a customized testing plan suitable for the project budget, software type, and complexities of the technical ecosystem. OnPath also created test cases appropriate based of user stories and features, and executed quality assurance practices. They provided detailed feedback on bugs or other found issues, drastically improving the final software product.

How many people from the vendor's team worked with you, and what were their positions?

There were three people on the core team, including the owner, the senior QA manager / solutions architect, and a QA analyst.

Can you share any measurable outcomes of the project or general feedback about the deliverables?

OnPath was able to provide excellent QA on budget and on time. The quality of QA feedback was exceptional.

Describe their project management style, including communication tools and timeliness.

OnPath's project management was great. They felt like fully integrated members of our product team, which was key for this project as it featured many shifting variables. Communication was primarily done via video calls, chat tools, Jira, and email. Communication was always timely and clear.

What did you find most impressive or unique about this company?

OnPath Testing's adaptability and collaboration skills were truly impressive. The project we worked on together featured a very tight timeline, shifting requirements, and a complicated technical ecosystem. OnPath was able to bring critical thinking and focus on intent and outcome over output, yet still provide exceptional quality assurance while following best practice.

Are there any areas for improvement or something they could have done differently?

Nope, they were great.

Please describe your company and your position there.

I'm the CTO of a technology company that provides a nationwide platform for vehicle customization.

For what projects/services did your company hire OnPath Testing?

We hired OnPath to close an immediate gap in quality assurance testing for our online product which was (at that point) 1-2 months out from launch.

How did you select this vendor and what were the deciding factors?

I had a personal working relationship with the CEO, Brian Borg, dating back to 1999, when he was a quality assurance engineer and I was a software engineer. Fast forward to today, both being leaders in our respective domains, it was the right time & place to reach back out to him to see what new accomplishments we could achieve together.

Describe the project in detail and walk through the stages of the project.

OnPath helped focus in on the biggest, most immediate needs, in order to bring value to our in-house development team as quickly as possible. That meant providing ad hoc testing against a codebase in development, as well as the initial stages of laying a foundation for automated testing in the months to come.

How many resources from the vendor's team worked with you, and what were their positions?

OnPath provided two resources, a QA engineer, and a QA manager to help coordinate & align the deliverables.

Can you share any outcomes from the project that demonstrate progress or success?

The ad hoc testing provided an immediate gap closure. They put a fresh set of external eyes on our product, providing valuable feedback to identify and eliminate edge conditions we would have missed otherwise. Additionally, the framework for automated testing immediately provided the developers with scripts they could work with and expand upon to improve the quality of their own software.

How effective was the workflow between your team and theirs?

The workflow was extremely flexible; we initially had OnPath's QA engineer integrated directly into the team, participating in stand-ups, processing items ready for testing as they flowed through our JIRA board, adapting as necessary.

What did you find most impressive or unique about this company?

The most impressive part of OnPath is their ability to quickly assess the situation and culture, and consider for both when adapting their solutions. Ours was very informal and conversational, and they read that quickly, and became flexible to meet those same expectations.

Are there any areas for improvement or something they could have done differently?

Nothing that I can think of the moment!

Introduce your business and what you do there.

I am the director of product management at ClearCaptions, a transcription service for people that are hard of hearing.

What challenge were you trying to address with OnPath Testing?

We needed OnPath Testing to test our software applications and hardware to confirm that our phones were meeting the product requirements, both from feature presentation and from a nonfunctional perspective as well. At the time, we were a part of another company called Purple Communications. We were in a transition period and developing our own solution.

What was the scope of their involvement?

It has really been a test, fix, and recast process where everything is done in a ticketing system where we document various cases and acceptance criteria.  Every week we typically have a build that is coming from our development team and then heads to the test team. They validate it based on the issues and acceptance criteria that are included in that sprint.

Depending on where we are in the development cycle, if it’s a released candidate, for instance, then we will go through regression testing and they will document everything within that same Jira ticket and then the process repeats. We have modified this process overtime from trying to communicate across externals versus using an internal resource, which has been very different.

What is the team composition?

I have worked a lot with Brian (President & Chief QA Engineer, OnPath Testing) and 3–4 testers. Over the last few years, we have had anywhere from 2–8 testers. 

How did you come to work with OnPath Testing?

I originally found them online. We reached out to various companies and did interviews to understand their process, how we integrate cost quotes, and sent out an RFE (Request for Evidence). Ultimately, OnPath Testing was the one that we gelled with most and in terms of both understanding the scope as well as culturally. This process took a few months.

How much have you invested with them?

We spend roughly $500,000–$1,000,000 annually. So, we have spent $1 million–$2 million in total. 

What is the status of this engagement?

We have been working together since January 2017 and the relationship is currently ongoing.

What evidence can you share that demonstrates the impact of the engagement? 

They’ve enabled us to get the product out the door and to catch issues and make assessments on whether we want to fix something now or later. We are also able to pinpoint, address, and resolve issues that we are seeing in the field very quickly.

How did OnPath Testing perform from a project management standpoint

They are able to focus on this particular product, which is really good. There are never distractions in terms of deprioritizing us based on other responsibilities they may have. They immediately take care of what we need and get it turned around. They’ve exceeded my expectations in regard to timing; they are always on time. With testers that are overseas, we communicate via Jira issue tracking, Confluence, or email. We had a little bit of a rocky start because we got a new project manager, but the relationship has integrated.

What did you find most impressive about them?

Their flexibility has been really beneficial. Brian and I have sat down and gone through product requirements, how various features and functionalities work and finding the acceptance criteria around those, and then how we will test them. It has been really great to reach out and talk through things. It’s a lot more interaction and collaboration. My impression with other companies was they wanted everything predefined, so we really appreciated the collaboration with OnPath Testing and being integrated with our teams.

Are there any areas they could improve?

There are some individuals in the team that haven’t kept up with the changes from our product requirements, so there’s some work that needs to be done there. There’s been some turnover here and there, but it’s been managed fairly well. 

Do you have any advice for potential customers?

Spending a lot of time working on the frontend, defining the process such as test cases, and understanding what test suite you’re going to use is important. Communicate the metrics you capture in order to understand how those cadences are going to work. Examine the frontend in the beginning so that you can just tweak the relationship as it continues.

Please describe your organization.

We are a Scrum Team working on developing software products for online registration of athletic events. This can include, for example, the New York City Marathon. Registration for the event is typically done online currently. We provide the necessary software for it. We also provide scoring systems for these races.

What is your position?

I am the product owner.

What business challenge were you trying to address with OnPath Testing?

The business goal was making sure that we had sufficient regression testing for the code we had developed. We initially used Selenium for the process, trying to automate the testing for our registration and scoring systems.

Please describe the scope of their involvement.

OnPath Testing wrote test scripts in Selenium and Capybara for us. We moved to the latter because we needed Ruby code support. They became familiar enough with our systems to be able to write tests scripts themselves, without the need of doing step-by-step scenarios.

OnPath built our entire test suite for Selenium in 2013, over the span of three to six months. They continued to write automated tests as we developed new features, in order to make sure that everything was covered. In late 2014, our automated test suite became too cumbersome, so we decided to switch from Selenium to Cucumber. This involved rewriting all of our application tests. OnPath were the ones who undertook the task in its entirety.

They had to learn the new technology, as they didn't have past Cucumber experience. They not only did this, but they also learned to write for Capybara, the Ruby code that runs Cucumber. It was a long project, but they stuck to it. The total time was about six months for complete code coverage. They continued to write new parts as we released new features.

OnPath was willing to learn new technologies, which we seriously appreciated. We had a remote relationship. The dev team we worked with is based in India.

How did you come to work with OnPath Testing?

They were referred to us by a collaborator who had worked with them in the past.

Could you provide a sense of the size of this initiative in financial terms?

The total expense for our collaboration was around $50,000.

What is the status of this engagement?

The collaboration ended in December 2015. Our company is moving to a different direction, and the Scrum developers are now responsible for writing tests. Ending our relationship had nothing to do with OnPath Testing’s performance.

We only gave them two weeks’ notice of the termination, and OnPath Testing continued the work up until the last day. It was impressive to see this attitude.

Could you share any statistics or metrics from this engagement?

We consider the relationship with OnPath to have been a very good investment. 

How did OnPath Testing perform from a project management standpoint?

We found them very reliable and great collaborators. They don’t need to be micromanaged and they’ve always been able to deliver the specs requested of them.

What distinguishes OnPath Testing from other providers?

We appreciate their desire to learn new technologies and their ability to put in long hours. OnPath Testing worked harder than most of our employees, which was impressive.

Is there anything OnPath Testing could have improved or done differently?

They were very willing to learn new technologies, but it took them a while to obtain the domain knowledge necessary to actually write out the test scenarios themselves. I had to take care of this for longer than I would have wanted.

Please describe your organization.

We work in the endurance event space, and we were building an online registration tool to facilitate transactions for participants in the events.

What is your position?

I was a project manager when we brought OnPath Testing onboard. 

What business challenge were you trying to address with OnPath Testing?

We were building out our integration tests, trying to improve them. We needed more manpower to help us ramp up the creation of specific quality assurance tests. Then, in time, we changed the technologies that we were using to write the tests. We knew this would add significantly to our time. We leveraged OnPath to help guide us to new systems and help build the foundation of those systems. They restructured our testing environment and kept us on pace with our growth. 

Please describe the scope of their involvement.

We moved to Cucumber, which is what they're using right now. Before that, we had a couple changes. We were always using Jenkins server, then we migrated, and now we're using Cucumber tests. OnPath helped us with that migration and took all of our old tests and rewrote them so they could be utilized again with the new system.

Our software developers also write tests. But, we needed a starting point and we wanted our software developers to focus on writing tests for new code in features recently developed. OnPath helped us catch up with the existing code that was not covered at the moment. They became an extension of our team.

We've used just two of their resources, continuously. They were like full-time members of our team. Since we started our relationship, it's always been full-time with those two. We never used anyone else.

We didn't have a choice regarding who joined our project team. That was up to OnPath. At one point, they brought in a project manager who was going to help and add to that and after working with him for a couple weeks it wasn't a good fit so we just kept with the two testing engineers. It was easier for them to work as members of our team than be managed through an intermediary. 

How did you come to work with OnPath Testing?

One of our senior software engineers had worked with Brian [Borg], the founder of OnPath Testing. He vouched for Brian. We spoke to Brian and decided we would give OnPath a shot starting out with just some basics, nothing too major. I'll say we were in our infancy in terms of our understanding as to what we truly needed. We didn't have a well-defined project scope when we first engaged with them. Since then, they have grown with us and gone through some growing pains with us.

Could you provide a sense of the size of this initiative in financial terms?

They were being billed out the two engineers each for $35,000 to $40,000 a year, which was higher than we would've expected. That's why we relied on them as full-time team members.

Could you share any statistics or metrics from this engagement?

We went from weekly to every other week releases. We were working on a live web application. We handled about $300,000 in transactions a day. We wanted to go to a continuous employment setup, but we didn't have the testing infrastructure to do so. As a result, we went from very large releases every two weeks that we littered with regressions to a rapid release to anywhere between three and seven deployments to production a day with a regression rate of less than 3 percent. That was all based on the work they helped facilitate. They also led the charge in training our software developers and bringing them up to speed. Our reliability increased a great deal.

Our own developers are more than impressed with the engineers we have from OnPath Testing. I don't know if we would switch those engineers if we would be as confident. Those two engineers became part of the team. They worked more than your 40 hours a week just as if they were members of the team. That was fantastic. I got the sense that we were the first company that brought them in as part of the team. The energy and enthusiasm they brought was awesome. It was a very good relationship. I know this was a higher price point than most people would expect to pay for offshore testing. We brought in a new vice president of technology about a year ago. That was one of the first things they said. IT [information technology] seemed expensive. Our software developers love working with them, though. There's a little bit of a learning curve to start, but once they're familiar with the process and standards, things to very smoothly. Our project managers love working with them, too. They're very consistent. 

How did OnPath Testing perform from a project management standpoint?

We managed everything through JIRA. OnPath Testing is well versed in agile development. I don't know if the whole organization is or not, but it's worked very well from our perspective. We never had any issues with delayed work or anything like that.

Is there anything OnPath Testing could have improved or done differently?

I don't have great judgment as to the value of the entire organization. I know these two test engineers are fantastic. One time they brought in a new person, and it wasn't a good fit. That project manager didn't work out. I can't really attest to whether or not other members would have produced the exact same results as the two we have. There were several times in which Brian offered to bring other people on board to help. But, we were satisfied with the people we had and the amount of output we were getting, so we felt like it would have been unnecessary. 

Do you have any suggestions for future clients of theirs?

We use offshore resources a lot and for us it's always been about making them feel like an integral part of the team. Have them partake in the wins and losses of the team. We've gotten more out of our contractors than I've seen other organizations just because they were part of our culture. Just get them as involved as possible so they understand the context around the tasks and can take some degree of ownership of the project success.

Please describe your organization.

We have a web platform that helps language teachers use the web for what's called blended learning, or hybrid learning. This is where part of the traditional language teaching class, part of it is the normal face to face in the classroom, and then part of it is online, where the student works on their own. It's not a virtual classroom where you're connecting with a teacher online, rather it's what they call a learning management system, where the student does different tasks to work with the language that they're learning.

The platform is a learning management system and a content repository. We license content from publishers in the English language teaching field, and then we do some fancy stuff with the content, including disaggregating it so that teacher can take bits and pieces from different courses and recombine them into a learning sequence that is personalized for each student. There's a learning management aspect to the platform, there is a content repository, there are authoring tools, and then there are some social features, where students can collaborate together on projects, and that type of thing.

We sell student subscriptions to language schools and universities. We don't sell directly to students. This is not a "learn on your own product" like Rosetta Stone would be. Rather, it's for language schools, universities, and even corporate training departments. There is a tremendous amount of corporate language training that goes on as well. It's technology geared to help teachers do a better job as opposed to technology that's supposed to replace teachers. It's a typical software as a service business model. We sell student subscriptions. It's about $12 per month, per student, and most of our customers are in Europe and Latin America. We're just getting started in Asia, and we have a fair amount in the Middle East as well.

What is your position?

I am the founder and CEO.

What business challenge were you trying to address with OnPath Testing?

We went through quite a dramatic evolution in terms of the coding talent that we were hiring, which might be typical in situations where you have a non-coder as the CEO because I'm not a tech guy. As a result, it was difficult for me to judge talent and manage the project. We went through some ups and downs with different firms that were building our software and then we finally decided to go ahead and just build our own.

We were working with our lead engineer at the time, a gentleman by the name of Aaron Zakowski, a very qualified and competent engineer. He recognized that we needed some good, hardcore quality assurance and brought testing in as an integral part of the processes we use to build our product, and we ended up hiring Brian [Borg, Founder at OnPath Testing] and his firm.

Please describe the scope of their involvement.

The development of our actual system was done in-house as well as with the help of other people we hired. We didn't get any really decent results until we hired our own. That's not to say it's the only way you can do it. It just worked out that way. I did hire a couple of firms and, in both cases, it did not work out in a satisfactory way. It was our in-house team that ended up doing it, although they inherited some buggy legacy code from the firms that had started on the project.

In a way, a lot of the development ended up being an undoing of what people had done before them. It was through that that we ended up working with Brian. Although he headed up the OnPath firm, at that point, he only had maybe five or six guys. But, Brian was able to fit in as a part of the internal team somehow. I think that was one of the reasons that it worked out so well with him. He was very good at the communication side of things, understanding the dynamic of the software team, and how to fulfill his responsibilities in a way that kept the whole thing moving along nicely.

Reworking some of the buggy legacy code was a nightmare. We used a rapid development framework, and had mixed results with a couple developers that we hired, and finally decided to hire the firm that invented this rapid development framework, thinking they would be great. All they wanted to do was deliver software that was so abstracted that they could then reuse it for other projects, and it was just nightmarish. There were just layers and layers on things. This meant that you had a core element that you could reuse for any project in the future, regardless of context or specific use. Once we cut ties with them and hired our in-house team, they had to pull some of those layers of abstraction out, so that everything ran a lot more efficiently.

OnPath was integrated directly into our development cycle. They would participate in the sprints we would run. We might also take some time and have a period with more comprehensive testing on a specific new piece that we were releasing. Brian was on all the stand-ups we would have each day, and he worked not only with our lead engineer but also with all the other coders on our team.

We did primarily performance and functional testing. We didn't do a lot of load testing. We really didn't do a lot of usability testing, so it was mostly just making sure that things were working the way they should. We had some complicated stuff happening on these web pages with the authoring tools. The authoring tools produce a range of almost 20 different question templates, including some drag-and-drop stuff, picture labeling, and stuff like that. Those are aspects characteristic of language teaching, but tough to actually implement in HTML and JavaScript.

Brian would work directly with our lead engineer, looking at some of the basic architectural questions as well. That's probably less traditional testing, but more of just his feedback on anticipating directions we didn't want to take. He could say, "Hey, I can see that causing problems down the road." That's an example of him being involved directly in the team in a way that was very much not how you would traditionally view a vendor, especially one running their firm.

Anyway, we didn't do a lot of load testing. It wasn't automated. A couple of times we were going to go ahead and put in some automated testing tools, and then, for a variety of reasons, and usually just the immense time pressure we were under, we decided not to actually do that. We were talking about whether we should go with Selenium, but it ended up being manual. Brian was actually the one pushing for that because he said, "Guys, let's do this right." I think I kept overruling him, to his displeasure, but he worked well with everybody, even though sometimes as part of the team he wasn't running the show.

How did you come to work with OnPath Testing?

I'm not sure. My guess is that we put something in Stack Overflow. Or, maybe it wasn't. I don't even know if Stack Overflow was around back then. This would have been 2009, so it was something like that. This was mostly run by our own lead engineer, and I wasn't really involved with the decision myself. Aaron probably interviewed two or three different guys. He's a very tough and demanding interviewer. I remember him being enthusiastic about hiring OnPath.

Could you provide a sense of the size of this initiative in financial terms?

We probably spent around a quarter of a million dollars annually.

What is the status of this engagement?

The last milestone completed was about 18 months ago. At the time, we were half-owned by Cambridge University in the United Kingdom. Just after I launched and got a beta version of my product, they came along and liked the technology, so we set up a joint venture that was 50/50. For a variety of reasons, the university publisher, Cambridge University Press, decided to go in a different direction in terms of technology. We had a very amicable separation about a year and a half ago. It was at that point that we dramatically downsized our team, and we don't have a dedicated testing unit right now. When we get going again, we will absolutely be calling Brian right away.

Could you share any statistics or metrics from this engagement?

I don't have anything specific to offer you. There wasn't really a before and after because he was there building it from scratch. I can't tell you anything on page load times, uptime, or anything like that. In terms of specific numbers, I can't really help you out there.

In terms of internal stakeholders, I do know that occasionally we would pull in people from the university, subject matter experts on something and have them be involved in certain projects where they would have actually been in meetings with Brian there. They were external to the company, but Brian has a way of getting to the crux of an issue very calmly in a way that doesn't step on anyone's toes. He is very diplomatic. He comes in very calm, gets right to the essence of the matter, and unravels the knots, so to speak. I do remember our Cambridge partners occasionally saying, "Wow, that guy really contributed to that meeting." These were sometimes technical people at the university as well because we had a couple joint projects with them.

In terms of customers, we have a reputation for doing some complicated stuff in a way that works. The platform is very stable, and we do some fancy stuff that really no other platform does. If you look at other learning management systems and the authoring tools that they come with, ours is really second to none in terms of the complexity of the types of tasks that we let the end-users create by themselves. Many times, if you go in and custom build something, you can make some fantastic learning tasks on the web, but we went beyond that and created a template that allowed any user to do it. We had a bottom-up, user-generated content element that's critical and a big differentiator for us as well.

That stuff just works, so in terms of customer reputation I think you can see Brian's work in that recognition of just the fact that it does work. People really trust English360 a lot because their whole work processes run through English360. If you're a small language school or a university, and you're delivering blended learning to these customers, years' worth of your work is inside the English360 platform in terms of the content that you've created. There's a trustworthiness that's important there and, if we didn't have that, we wouldn't be successful as a business. The fact that we do have it is a testament to the work that Brian did.

What distinguishes OnPath Testing from other providers?

One of the reasons that Brian is so good is he has a very high level of emotional intelligence. There are many good QA people out there, but what sets Brian apart is just fantastic communication skills. He's a great teammate; he's always very steady, and he calms everybody down. That was hugely beneficial for us because we were under a lot of pressure to release our partners at the time. We needed something out there in the market very quickly, and Brian played a big role in getting that done.

Is there anything OnPath Testing could have improved or done differently?

The only possible thing I could say on that is not anything that Brian needed to do better, that OnPath would do better, it's just that we got used to working with Brian personally, so that when we would have one of his guys jump in, I would email Brian and say, "Dude, we need you." Sometimes, Brian and I would go back and forth on this, he'd say, "I've got other stuff going on as well. I'm trying to do as much as I can to be dedicated to English360, but I do have other customers." At the end of the day, he has a business to run, too. It's understandable, but it's also a bummer because he was such a huge asset to us.

As it turned out, his guys are fantastic. I forget the other guy we used to work with a lot, who reported to Brian inside OnPath. He was a very high-quality tester. What Brian brought to everything was that emotional intelligence aspect of it. That dedication and expertise that he brings became essential to the whole build, so if he couldn't be in a meeting, we felt the absence. I have no idea about the guys now. Like I said, I haven't worked with Brian or his team in more than a year now. I don't know who he's got in place now.

That really isn't so much an improvement area, it's just recognizing that, as an individual, Brian has something special when it comes to the human side of software development, and when that wasn't there, we missed it. I guess that's saying that he just needs to clone himself to populate his company. Other than that, Brian was great to work with.