Could you share any evidence that would demonstrate the productivity, quality of work, or the impact of the engagement?
For us, everything boils down to shipping on time. The quality of our engagement with vTEST has helped with the low-end of the bug count, as well as with the bugs carried over to release. As we continued to work on communication, better spec-writing, and questions, we ended up catching more QA issues prior to shipping the product. Much of this has played into the successful ongoing implementation of our partnership. There are hard metrics to dig up when it comes to testing quality, but it’s hard to separate that from development quality. I boil it down to the number of bugs shipped per release, the number of bugs caught, and how the quality of the product evolved over time.
It would be difficult to say how much we’ve improved, but the key thing is that we’ve brought vTEST in as an equal member into our processes and conversations. Many companies bring testing in at the end of the process, which is not the best approach in terms of quality. Overall, we’ve evolved as a team. The testing has helped the development get better, and the development has helped the spec-writing get better — we’re working as a very strong unit.
How did vTEST perform from a project management standpoint?
The collaboration was definitely not a one-way street. We have a development model which incorporates vTEST into our team, and there’s a lot of back and forth which has led to a higher-quality product and better engagement.
What did you find most impressive about vTEST?
The quality of communication has been very high. I’ve worked with vTEST’s founder himself and found his level of client focus to be great. I never found that I couldn’t get a hold of vTEST or that the communication was poor. This has been one of the main reasons for maintaining the relationship.
Are there any areas vTEST could improve?
We have brought them in as a team member for retrospectives after each release and have been able to point out things which could be improved. Having the flexibility to consistently improve is important to us. We ship every three weeks or so, and we’re all making corrections to our performance during those periods, from quality of communication to availability and how often we have international calls. To point out one thing would be to say that it hasn’t been worked on. We’ve been very aware of our shortcomings, and have worked together to take care of them.