Could you share any statistics, benchmarks or user feedback from this engagement?
The way they do it is they'll have each person test it. Let's say they need two testers per build. Each tester will typically spend at least three or four hours, sometimes going up to eight. They definitely spend a lot of hours. I also have them be very specific in the way they log their work, like having to capture video. They could spend eight hours and discover 12 bugs, but they'll also test and resolve a number of other bugs.
I can't say that they're the most efficient, but they are incredibly thorough, which is important when you're working with a team that's spread out throughout the world. I don't have any benchmarks that I hope they hit. I don't tell them they have to work a certain number of hours because then you get people trying to create issues, like what happens if you tap the screen with your entire palm 20 times in a row. Things like that just to produce a bug. I don't have any benchmarks like that, but they definitely do their work, and do it well.
How did QATestLab perform from a project management standpoint?
Yuliya is the one who runs my group of testers. She is excellent, and they're all excellent. I would really give them my highest recommendation. The team is awesome. In regards to project management, she handles it very well. She scales effectively, and she makes sure that they're good communicators. The scaling part is very difficult to do as you're scaling personnel. She pretty much does it on demand, and that's great.
The people at QATestLab are all really good communicators. I was a bug tester myself for almost five years, so I know that it's not the easiest thing in the world, and they're good and they really enjoy it. I also have an interesting product; they like to go out for walks with the activity tracker. Project management is an A+.
What distinguishes QATestLab from other providers?
The things that just make it enjoyable to work with someone. They have a great personality, they're pretty much always available, and they get back to you very quickly. Anyone who joins the team is always nice and professional. They don't fight back often. When we first started working with them, they didn't use JIRA. I think they were using a popular one that I hate – it's really messy. I forget the name, but it's a really popular bug-tracking tool. Anyway, they were using that, which is so complicated.
I wanted to move to JIRA, and they did it. I mean, I paid them for the hours, but they didn't complain, and they moved everything to JIRA, which is an incredibly tedious task. I can put a lot of strain on them, and they're always very flexible. They have good ideas. Yuliya really strives for her team to be the best, and also Yuliya was a tester herself. It's not like you just have a project manager jumping in just because she's good at communicating and speaking English. They're like a team of testers run by testers, which I think goes a long way.
Is there anything QATestLab could have improved or done differently?
The only thing I sometimes have to reprimand Yuliya on is sometimes her bug testers will log ridiculous bugs. Like, if you tap on the screen 30 times in 20 seconds, the app crashes, but the apps are not meant to work like that. I tell her to stop doing things like that because it's a waste of time, but it doesn't happen that often.
Every now and then, I'll get a silly bug like that, but that's alright. You want testers that are trained to do stuff like that, but the average person doesn't think like that. It's good that they have that mindset. That's one of the areas where she could improve, but that doesn't need to be on her list of priorities because it has no detrimental effect.
When we first started working together, they spending time testing on a lot of older devices. I didn't monitor that much initially, but now I do. I let them know what devices and what combinations to test on; otherwise, they'll test everything under the sun.