OnPath Testing

Could you share any statistics or metrics from this engagement?

We went from weekly to every other week releases. We were working on a live web application. We handled about $300,000 in transactions a day. We wanted to go to a continuous employment setup, but we didn't have the testing infrastructure to do so. As a result, we went from very large releases every two weeks that we littered with regressions to a rapid release to anywhere between three and seven deployments to production a day with a regression rate of less than 3 percent. That was all based on the work they helped facilitate. They also led the charge in training our software developers and bringing them up to speed. Our reliability increased a great deal.

Our own developers are more than impressed with the engineers we have from OnPath Testing. I don't know if we would switch those engineers if we would be as confident. Those two engineers became part of the team. They worked more than your 40 hours a week just as if they were members of the team. That was fantastic. I got the sense that we were the first company that brought them in as part of the team. The energy and enthusiasm they brought was awesome. It was a very good relationship. I know this was a higher price point than most people would expect to pay for offshore testing. We brought in a new vice president of technology about a year ago. That was one of the first things they said. IT [information technology] seemed expensive. Our software developers love working with them, though. There's a little bit of a learning curve to start, but once they're familiar with the process and standards, things to very smoothly. Our project managers love working with them, too. They're very consistent. 

How did OnPath Testing perform from a project management standpoint?

We managed everything through JIRA. OnPath Testing is well versed in agile development. I don't know if the whole organization is or not, but it's worked very well from our perspective. We never had any issues with delayed work or anything like that.

Is there anything OnPath Testing could have improved or done differently?

I don't have great judgment as to the value of the entire organization. I know these two test engineers are fantastic. One time they brought in a new person, and it wasn't a good fit. That project manager didn't work out. I can't really attest to whether or not other members would have produced the exact same results as the two we have. There were several times in which Brian offered to bring other people on board to help. But, we were satisfied with the people we had and the amount of output we were getting, so we felt like it would have been unnecessary. 

Do you have any suggestions for future clients of theirs?

We use offshore resources a lot and for us it's always been about making them feel like an integral part of the team. Have them partake in the wins and losses of the team. We've gotten more out of our contractors than I've seen other organizations just because they were part of our culture. Just get them as involved as possible so they understand the context around the tasks and can take some degree of ownership of the project success.

Could you share any statistics or metrics from this engagement?

I've been doing software engineering for close to 20 years, and in only two of the positions did I not use either contract QA or contract development work. Compared to other collaborations I've had, I actually rank OnPath highly. I would recommend them without hesitation. The nice thing in my experience with Brian is that he's very good at being flexible with how we wanted to manage the team. I've worked with companies where their only model was to have a project manager who we had to work through to assign tasks, and it's just been an inefficient way to manage the small numbers of resources we had.

Brian was very flexible about the model that we were using. He was good about taking feedback. At one point, we had a test engineer who initially was performing well, and then it seemed like there was a focus issue or something was going on. He was good about addressing that, and then eventually helping us find a different person to fill that role and getting a better resource in place. In general, openness to making a difference and moving on the fly has been great. I'm a huge fan.

We develop new products on about a six-month basis, so about two or three times a year, we're launching what we call a new video mashup. It's a new marketing site focused around whatever the new product line is. Historically, prior to putting a focus on it, ramping up QA and putting OnPath on, and creating the structured QA methodology that we have right now, we would put those launches out, and invariably, immediately we found issues with either page content, incorrect presentation for customers, or problems in completing an order. Frankly, I don't have those issues anymore, in part because of process changes, but also just around being able to add the staffing needed to make sure we have good coverage.

The second piece, from my standpoint, is just responsiveness and turnaround time for projects associated with the marketing team in particular. The ability to add contingent staff, and have that resource layer that I can grow as needed when projects come in, has helped me keep that responsiveness in turnaround time. When I came in the company, it was a significant talking point around how the software engineering team could support the marketing team better, and why we can't get stuff done. That conversation has pretty much ended. In most cases, now we really are beholden to the ability of the marketing team to create the copy that actually needs to go up online, versus being able to get the technical work done and test it.

At an execution level, they've been very focused on the test work. Honestly, that's partly me focusing them in that area, and really driving the strategy for QA process evolution internally. The flipside being, Brian and I had several very good, overarching conversations about where the relationship is going, what's our focus longer term, looking at broadening test automation, whether that's something that OnPath could support and help with. There's definitely strategic insight there, even though I haven't particularly leveraged it in the engagement with them.

How did OnPath Testing perform from a project management standpoint?

The project management tool we use is JIRA, the cloud-based version. We have a very organized  bug-tracking workflow. Brian actually helped establish that as the bug-tracking tool in the company before I came to the company. That was definitely, from my standpoint, one of the nice things that OnPath helped to drive, prior to me coming in.

What distinguishes OnPath Testing from other providers?

Like I said, it has to be their management approach and flexibility during project work. OnPath Testing has been very accommodating to us when things have come up unexpectedly. They're never afraid to give us crucial feedback, and they have made some key recommendations around structuring and improving our QA environment. I've been very grateful for that support.

Is there anything OnPath Testing could have improved or done differently?

The only thing is setting expectations with the contingent staff that they bring in and making sure those are aligned with my own expectations. When we do those product launches that I was talking about, we tend to move into what we call our launch schedule, where we essentially have development and QA staff available – if not online, working, but available – about 20 hours a day, seven days a week to be able to do changes.

We run a lot of A/B split testing at that point in time. At 11 o'clock at night, I need somebody available to rotate into the next test, to make sure that from a testing standpoint everything's functional. We had a couple of bumps with people who didn't understand that expectation upfront, and it was a little bit of a reset. That hasn't happened in the recent past. Brian and I have discussed this, and I think he has kind of taken it to heart.