Ubertesters

BACKGROUND

Introduce your business and what you do there.

We make mobile apps for chatting and communication. Our engineering team is in Finland. We fit in the 11-50 employee category.

I’m am an engineering manager. I’m the guy who gets in touch with the testers and coordinates the product schedules and all that. I’m the middle man between the team and Ubertesters.

OPPORTUNITY / CHALLENGE

What challenge were you trying to address with Ubertesters?

They’re helping currently with a messaging app that we are building. It has specific requirements such as testing. Testing includes having multiple people having a conversation at the same time in order to emulate real-world situations. So we test both multiple people in a group chat and lots of simultaneous messages.

SOLUTION

What was the scope of their involvement?

The way they work is they assemble a group of testers for each testing session with a couple of days advance notice from us. You’ll see the same names and sometimes new names. I’ve seen a lot of improvement in the quality of testing reports over the years. I think they’re getting better and better in the quality of testing results. They use people from all over the world for any given test session. It’s five people per session. We do sessions once a month or more or less often depending on our current project situation. We do a test with them every time we release something to the app store for both iOS and Android. I work with the same project manager each time.

How did you come to work with Ubertesters?

A couple of years ago we were looking for a testing partner to test our app. I wanted to find a company that would provide this kind of service. There are many different models but we didn’t want to have an off-site full-time tester, and this dynamically sourced model seems to be better. I found them just using Google. A couple of companies that we spoke to were similar, but Ubertesters seemed to fit us the best at the time.

How much have you invested with Ubertesters?

We had more than 10 sessions. An average session is 10-13 man hours, so it’s probably around $5,000.

What is the status of this engagement?

We started working with them about a year and a half ago.

RESULTS & FEEDBACK

Could you share any evidence that would demonstrate the productivity, quality of work, or the impact of the engagement?

I think it gives us an extra layer of certainty whether our build is ready for release. It allows us to improve the quality of our product. It’s an important mechanism for validating the things we haven’t thought in our tests. They’re tested by a reasonable number of people with the intention of specifically finding those issues. It saves us time and money, but I can’t quantify it.

How did Ubertesters perform from a project management standpoint?

They have an issue tracking tool which integrates with the app. So, when you make an application build that you want to be tested with them, you add this component to your app. It enables you to take screenshots and executes specific test cases and records bugs and issues. That tool then integrates into their bug tracking system which then integrates into third-party systems that people use. So, if our company uses a tool ‘X’ to track bugs, then Ubertesters’ tool allows us to easily export bugs to our system so we see the bugs in the same monitoring tool that we use internally.

What did you find most impressive about Ubertesters?

I think their offering is pretty good and I’m hoping they keep improving their web tools and mobile add-ons. I hope they manage to make those better and better all the time. As long as they keep doing that, it’s all great.

Are there any areas Ubertesters could improve?

Just keep improving the web tools in general. There are a lot of small improvements that can be made to any user interface. I’m sure they have a lot of ideas. They need to keep improving those tools and not keep them as they are because there are new ideas that can be added. It’s hard to say specific things. It’s nice to see things keep developing and improving. When things get slightly better every time I open their tool, that’s what interests me and they should keep doing that. Every time I open it, I’m surprised that there are new improvements and features and it's slightly easier and slightly faster to use.

BACKGROUND

Introduce your business and what you do there.

Geshu.com is a company that basically develops websites and applications for clients and also deals with online marketing and social media marketing. At the same time, we produce our own products; either standalone CMS [content management system] products, content management products or individual products that solve a specific problem for clients around the world.

Our most recent products are dealing with growth hacking, social media marketing, automation of new user generation for Twitter/Facebook, automatic posting of new content to different channels, lead generation through social media, etc.

I am the Chief Executive Officer.

OPPORTUNITY / CHALLENGE

What challenge were you trying to address with Ubertesters?

We've developed an app called Inspire. It was basically an inspiration app. There are tons of inspiration apps that deal with quotes from a bunch of dead people, but there are none that exist, outside of ours, that is essentially an inspiration-generation tool from friends. For example, I could receive a message from a friend wishing me a great day or saying that I'm a great person, what have you. As we know, when we receive information from our friends, family, or community, that has a much deeper impact on our daily lives. That was the crux of the main idea behind the app.

We used Ubertesters to get a bunch of people to test most of the functionality and the use cases. We wrote up a bunch of use cases for them and they actually helped us write a bunch of more advanced use cases, based on their experience with other apps, to help us test the functionality, test the flow of the app and the overall idea.

SOLUTION

What was the scope of their involvement?

We had a number of different phases. They did jump in on beta version 1; that's when we commissioned them to help us. After the users did their test, they posted different things they didn't necessarily like. They also made suggestions of how they felt the app could be made better.

In a sense, it wasn't that Ubertesters themselves, the organization, or the company, was a part of the development, but the feedback we received from their users was very helpful in refining a number of the functions, and honing in on the core features that development loses track of because you want all these new great features. You lose track of the main focus of your project. It was good to have them as a sounding board, so we oftentimes got back to what the core product was.

We did break out Android and iOS and we did make sure we did different tests for both. Then, we did an integration test between the two. We had them do some tests as if they had no internet connection, to see how the app responded. There were tests for different versions of iOS and Android, to make sure that our base case for whatever the Android version was, was met. They also did a test for different internet speeds, to see how the app responded.

How much have you invested with Ubertesters?

Under $10,000.

What is the status of this engagement?

For that app, it was around six months in total. We started either Oct. 2014 or Nov. 2014. It might have even been earlier. I remember my first calls were around June or July 2014. The feedback and the information that we got from them and their users were invaluable.

It was our goal to get it launched and get it in the app store by end of Jan. 2015. We got it in the app store in late Jan. 2015. We still went to another round with them after that. We launched with a reduced spec version and then had them do more tests for us a few weeks after that. We're still working with them.

RESULTS & FEEDBACK

Could you share any evidence that would demonstrate the productivity, quality of work, or the impact of the engagement?

Based on a lot of the feedback we got from the testers, our second version had another feature and had user experience enhancements. There, we saw a significant increase in regular downloads. I don't have the exact statistics but there was an absolute increase.

What did you find most impressive about Ubertesters?

The organization of the launch of the test was very impressive. There's a lot involved and a lot that I hadn't done before with an external testing firm. We do tests, but at lot of our testers are, essentially, either people in a room with a single-window glass with cameras outside, and they sit there and either do it on paper or do it actually on an app, or user tests with a bunch of people that come over to fiddle around with something that we've developed.

In this sense, writing the use cases and going so deep into the different devices and different operating systems on those devices because of their experience, they were able to really manage all of that. Get it to the point where we said we wanted to start our test on November 1st and we'd like to have our results by the 5th. Everything ran smoothly. It wasn't like all of a sudden people backed out and they didn't have anybody to replace them. They had enough testers to be able to very quickly replace testers.

In terms of the whole management side of making sure that it ran smoothly, it ran on time and we got detailed results within the time period that we needed them and wanted them. It was fantastic.

Are there any areas Ubertesters could improve?

It's hard to say. I don't really think so. In my dealings with them, I didn't have any problems. They were very accommodating. There was nothing that they said they couldn't do or that they didn't do in accordance with my expectations.

In terms of their service, and this isn't a negative for their service, it just shows sometimes how daunting the task of actually routing through all of the information is. Their platform provides a ton of information. The first time you go through it, or the first time that you log in, it seems a little daunting because there's so much information that you can actually see. We could see crash results for instance. There's just so much information that at first glance it may seem overly complicated.

It could be an opportunity for them to make a few small changes to where a novice can get the basic information that they need and not see everything. Then they could have an advanced view, where a real professional can really dig down into the data, to the nuts and bolts.

The first time I was in there, I was like, “oh wow that's a lot of information.” The first thing that popped into my mind is that there was so much in there and that it was going to take me a lot to go through it. I wasn't expecting that much information. It's actually a great thing, not necessarily a negative thing.

There was a couple of people that didn't like the app at all and will never use it, which is also good. It's great feedback. Then there were people who saw that the in certain functions, there were too many clicks, which helped us to rethink that flow to reduce the amount of clicks or taps necessary for someone to get from Point A to Point B.

BACKGROUND

Introduce your business and what you do there.

FairFly at the time was a mobile application that travelers would use. After purchasing their tickets, they would submit their ticket information into our app, and we would track that flight taking into consideration cancellation and change fees, and looking for cheaper flights or better flights in the sense that it might be the same price but with fewer stops and so on. It’s in the general realm of post-booking automation.

I was the Vice President of Research & Development.

OPPORTUNITY / CHALLENGE

What challenge were you trying to address with Ubertesters?

It started with one very simple engineering problem. We were looking for a cloud platform to do our standard test planning and standard test design. Rather than deploying an in-house product, we were looking for something in the cloud that we could use.

SOLUTION

What was the scope of their involvement?

It was very simple. It was everything it claims to be. It was a cloud platform for test planning, and it integrated very nicely into our application so that we could run test plans from our mobile devices. Once there were bugs and issues, we could immediately take screenshots, which would be submitted to the application. Then we could take all the issues and automatically open JIRA tickets for them so that our developers could solve them. On that front, there were no resources. It was just the self-service there, and it worked out of the box. That’s one part of their offering. 

It’s important to add that the test planning and the cloud are only one aspect of their offering. Their main offering is their qualified testers for ad-hoc projects. They have this system that I described, which is test planning in the cloud. So if an organization wants concentrated QA [quality assurance] and software testing efforts; rather than hire someone, Ubertesters has a qualified workforce that could be used for specific testing projects.

How did you come to work with Ubertesters?

Israel is a small country, so word gets around. Our CEO [Chief Executive Officer] met or knew Ran [Rachlin], who is Ubertesters’ CEO, and he described the product. Then he came and described it to me and I said, “Great, that’s exactly what we are looking for.” That’s how we ran into them.

How much have you invested with Ubertesters?

I’d rather not say.

What is the status of this engagement?

We started working with them in the first or second quarter of 2015. FairFly has changed its model a bit, and it’s not developing a mobile app any longer. The engagement was wrapped up just about the time that we switched our business model, sometime in the second quarter of 2016.

RESULTS & FEEDBACK

Could you share any evidence that would demonstrate the productivity, quality of work, or the impact of the engagement?

First of all, their platform works. Their platform really gave a boost to our testing efforts because we used it internally so we could streamline the entire bug finding and bug fixing process. I would say that it gave us probably a 25% acceleration over what we would have done without their platform.

As far as the project, they built a four-person strong workforce for us in location. One of the ideas was that we wanted a very focused workforce because we wanted to see how our app would perform in the United States. We could not do it from where we were in Israel, but they built a workforce for us with testers residing in the United States. Moreover, we wanted to test it on specific devices. They even qualified the workforce even more by saying, “we want these testers to be in the United States on these and these devices.” They found them and it worked very well.

It gave us a lot of confidence that we would have a successful launch in the US. They really came at the tail end of the testing process, and hence they did not find that many bugs because there weren’t many bugs to be found by that stage. They ran the entire suite of tests, which we had already defined on their platform, and it gave us the boost and the confidence that we needed to know that we can release the product in the US.

How did Ubertesters perform from a project management standpoint?

They performed very well. They were very responsive on all accounts. Sometimes we needed some technical support for integrating their software development kit into our application, and we would get very timely responses. When we did the testing project, we actually got a project manager assigned to us, and that project manager worked with us. I’m sure that project manager was also managing other projects, but we would not know it. Every time we wanted that project manager, she was available to us and had great responsiveness, be it by email or phone call or whenever, any time we needed. The overall handling of the project was very good.

What did you find most impressive about Ubertesters?

The ramp-up is really as fast as advertised. They say that they need two days to get the project manager on top of the project and then another two days to get a team working. Within less than a week from the time we decided to kick the project off, we were up and running and we completed it in less than a week.

Are there any areas Ubertesters could improve?

No, actually not. Considering all the crowdsourcing testing solutions out there, I think that they really stand head and shoulders above anything else out there because of the fact that they qualify their testers. Of course, you need to pay premium dollar for that, but if I want someone in Virginia with a 64GB iPhone 6 then I can get it. That’s one hell of a focus, and they can do that. I don’t have anything negative to say.

BACKGROUND

Introduce your business and what you do there.

Our company provides software development services to other companies. Our clients are mostly from Western Europe and the United States. I’m a product manager. I deal with development projects, our development team, and our clients.

OPPORTUNITY / CHALLENGE

What challenge were you trying to address with Ubertesters?

Our main issue was that clients were not able to report bugs efficiently. In many cases, clients wanted to test mobile apps themselves, even though we provided our internal QA [quality assurance]. When they noticed bugs or saw that something did not work properly, they sent us descriptions. Our product managers would ask clarification questions about their device, battery usage, Wi-Fi connection, and so on. Obviously, that was something that our clients would not have submitted initially. Generally, the communication process and overall QA process were too long. This was not satisfactory for us nor our clients.

SOLUTION

What was the scope of their involvement?

Ubertesters have testers all over the world, including Australia and New Zealand. The company let us hire those testers. We just contacted their project managers and told them, “We need 1 or 2 testers in New Zealand tomorrow.” We’d also give a specific number of hours and they would respond quickly. We didn’t need to spend time searching for people and managing them because Ubertesters was managing the people on their side.

How did you come to work with Ubertesters?

I think our founder recommended Ubertesters. However, before deciding to work with them, we also considered a few other testing companies and platforms. Finally, we chose to work with Ubertesters.

How much have you invested with Ubertesters?

We have probably spent $10,000 to $25,000 on their services.

What is the status of this engagement?

We started working with Ubertesters a couple of years ago, in 2014. We are still working with them.

RESULTS & FEEDBACK

Could you share any evidence that would demonstrate the productivity, quality of work, or the impact of the engagement?

Ubertesters enabled our customers to submit bugs quickly. With one tap of a button, they could send us information about a bug or change request that automatically included information about the client’s device, battery usage, Wi-Fi connection, and other important details.

What I can say for sure is they helped improve customer satisfaction. For us, this  the most important contribution we can expect from any external vendor. Our customers no longer need to submit bug reports manually, which helped us improve the quality of our QA services a lot.

How did Ubertesters perform from a project management standpoint?

The project management is excellent. The thing that I like is that they have this large community of testers and somehow they manage them in a way that we don’t need to get involved at all. We have just one main point of contact, a dedicated project manager who works with our company. For example, one customer wanted to launch their mobile app in Australia. We don’t have testers in Australia. They obviously wanted their app tested there in the intended location. In many cases, this is necessary. Ubertesters allowed us to do that.They did it really well.

What did you find most impressive about Ubertesters?

I am impressed by their project management and their technical support. It’s rare to find a partner who will respond to you within an hour or 2. With Ubertesters, I can’t really remember a case where they didn’t respond within a few hours, even on holidays. In many cases, it is important to reply during holidays when you are developing for other clients.

Are there any areas Ubertesters could improve?

We sent requests to their technical support team about the new features on their platform that our clients want to have. We think that these will be very useful in our testing process. They are already implementing those, so that is excellent.