DeviQA

BACKGROUND

Introduce your business and what you do there.

I’m a QA manager for an advertising technology company; I manage a team of 18. We provide advertising automation tools to customers.

OPPORTUNITY / CHALLENGE

What challenge were you trying to address with DeviQA? 

We engaged DeviQA to help resolve our QA process issues. Our goal was to create a simpler QA process to test new features for our web applications.

SOLUTION

What was the scope of their involvement?

We’ve added many new features to our web applications since engaging DeviQA. They’ve been responsible for helping us test the new features, which we’ve built using React, Python, JavaScript, and Scala for machine learning (ML) tools. 

What is the team composition?

We’ve worked with three team members from DeviQA and 10 QA testers. 

How did you come to work with DeviQA?

Our previous CTO found DeviQA through a referral. I wasn’t involved in the process, but they had great reviews on Clutch, so our team initiated the partnership. 

How much have you invested with them?

We’ve invested over $1 million in the partnership with DeviQA. 

What is the status of this engagement?

We contracted DeviQA in June 2019, and the partnership is ongoing. 

RESULTS & FEEDBACK

What evidence can you share that demonstrates the impact of the engagement? 

Our products are complex and ever-changing, so we don’t typically create use cases or track metrics. However, since working with DeviQA, we’ve gradually expanded their responsibilities, and they’ve never missed a deadline. It’s unrealistic to guarantee our application’s success through testing, but the team has done a great job working alongside our customer team to address issues promptly. They’ve helped drive down our average customer response time to less than three hours. 

How did DeviQA perform from a project management standpoint?

We’ve been pleased with how DeviQA has handled our project; they’ve been willing to work late to address various client issues and questions. We’ve used Jira to keep track of tasks and communicated via Slack. We’ve worked with 5–8 QA testers since the beginning of the project, and the rest of the team has worked with us for over two years. 

What did you find most impressive about them?

I’ve been most impressed by DeviQA’s management team. They’ve resolved issues within 24 hours and supported us steadfastly throughout our partnership. During the Russian Invasion of Ukraine, we asked them to supply us with global QA resources, and they immediately identified skilled, qualified candidates. 

Are there any areas they could improve?

I have nothing DeviQA could improve on. 

Do you have any advice for potential customers?

I advise you to communicate your needs with DeviQA and share your feedback. Be as specific as possible when detailing your requirements, and they’ll deliver.

BACKGROUND

Please describe your company and your position there.

I am the VP of Product at CYDEF where we are delivering affordable managed cyber security endpoint detection and response to small and medium businesses.

OPPORTUNITY / CHALLENGE

For what projects/services did your company hire DeviQA?

We needed some help and expertise to develop full integration and regression test coverage for our platform and then to have these test plans integrated into our automated DevOps pipelines.

SOLUTION

How did you select this vendor and what were the deciding factors?

I hired this vendor based on their reviews and proven track record of having achieved success on similar projects with our software companies.

Describe the project in detail and walk through the stages of the project.

The project began by breaking down our system into modular components and writing manual test cases to cover all the use cases. Next we began automating the finished test cases. At present we are still developing and creating new content. We are also in the process of integrating finished automated regression and integration test cases into our DevOps Pipelines.

How many resources from the vendor's team worked with you, and what were their positions?

We currently are up to 3 resources from the vendor. All of them work in Quality Assurance either writing and editing test plans, automating test plans and executing test plans.

RESULTS & FEEDBACK

Can you share any outcomes from the project that demonstrate progress or success?

To date, DeviQA has helped us create hundreds of test cases and are fully integrated into our build and release process. We are using them as members of our development team and not as fixed project start-stop resources.

How effective was the workflow between your team and theirs?

The DeviQA resources came up to speed very quickly and work very efficiently with our team. We have virtually seamless integration.

What did you find most impressive or unique about this company?

The company is efficient and understands its core business value. They delivered what they promised and have been excellent to work with.

Are there any areas for improvement or something they could have done differently?

I can say that there is nothing I would change.

BACKGROUND

Introduce your business and what you do there.

We’re a diabetes care and solutions team that’s responsible for developing and operating a global medical platform that connects medical devices and allows patients to share data with healthcare providers and family members. I’m the director of program management, making sure that we implement updates throughout the year on time and at a high level of quality.

OPPORTUNITY / CHALLENGE

What challenge were you trying to address with DeviQA?

We brought them on to accelerate our automated testing efforts.

SOLUTION

What was the scope of their involvement?

They helped build something to accelerate the documentation of test results so we could package them for federal compliance documents. 

Our team started with essentially zero automated testing capabilities, intending to ramp up automated testing to about 80% of our product. They worked with us to achieve those benchmarks.  

Since scaling our automated testing, they’ve also come onto our manual testing team. There, they make updates to the framework and scale our capacity to run manual tests.

What is the team composition?

We have five automated testing specialists and six manual testers.

How did you come to work with DeviQA?

Our general manager conducted a web search, and we interviewed 5–6 companies. They asked good questions upfront and could forecast where difficult portions of the scenario would be, able to demonstrate their acumen from the start.

How much have you invested with them?

We’ve spent north of $500,000 to this point. 

What is the status of this engagement?

We started working with them in January 2020 and ramped up the team in October 2020. Our engagement is ongoing.

RESULTS & FEEDBACK

What evidence can you share that demonstrates the impact of the engagement?

Their work has been great. The automated testing framework they developed allows them to respond to changes rapidly and get test results in a timely fashion.

They’ve met our requirements as we’ve requested and overviewed potential process improvements. That’s helped speed up the test execution and provide more data on the output, so we don’t need as much documentation on our side.  

How did DeviQA perform from a project management standpoint?

We’re delegating tasks to their project managers, who are managing their individual teams. Their project management team effectively manages feature builds to ensure that our team isn’t being peppered with questions.

They aggregate updates and information to report back to us as well. Despite being in a different time zone, they’ve always been available when we needed them.  

What did you find most impressive about them?

They treat our automated testing project as a development effort rather than a testing effort, always anticipating next steps and improvements. Whenever we need changes, they’re able to respond quickly. They take on current problems and address future issues as well. 

Are there any areas they could improve?

No, they’re doing great. 

Do you have any advice for potential customers?

Empower them to be in the feedback loop rather than just using them for the test at hand. They can provide feedback and suggestions to benefit the development process.

BACKGROUND

Please describe your company and your position there.

Audio recording, editing, and transcription app QA Manager

OPPORTUNITY / CHALLENGE

For what projects/services did your company hire DeviQA?

DeviQA was brought on board to develop an automation framework and write and maintain tests for our Electron/JS app. Features included STT/TTS, audio file processing, media file compatibility, export formats, text and audio editing, and general UI functions.

What were your goals for this project?

To develop automated test framework for complete feature coverage of UI/functional elements of our application, with the goals of identifying regressions as quickly as possible and keeping pace with new feature development.

SOLUTION

How did you select this vendor?

Deciding factors for selecting DevIQA were their specific skill set and expertise in our tech stack, years of proven experience in leading QA automation efforts, and general knowledge of test automation frameworks and CI integration.

Describe the project in detail.

The scope of our project was to implement and maintain an automated testing framework for our desktop application on an ongoing basis. Technical requirements were CI integration, UI functional testing across media types, formats, upload/download, text and audio editing, authentication workflows. I joined the company after DeviQA stated working with us, and we eventually made the decision to adapt our QA model to in-house.

Regardless, I worked with them for 3 months in a management capacity, depending on their team for nightly test builds and results, bug reports, and fast turnaround on regression testing. The DeviQA team was responsible for following new features and code changes made in our repos and quickly automating test cases and ensuring test coverage for new and existing features. They were in charge of test case organization, designing and implementing the testing framework, and integrating it with our CI system (Jenkins).

What was the team composition?

The team was 2 members, who were both test automation engineers.

RESULTS & FEEDBACK

Can you share any outcomes from the project that demonstrate progress or success?

DevIQA engineers were experts in Javascript test libraries and tools, and designed a framework that was easy to maintain and integrated well with the application services.

How effective was the workflow between your team and theirs?

DevIQA was consistently available via slack and video chat when needed with flexible hours that worked for our engineering team. We held weekly meetings on goals and progress, and they provided daily status updates. Their communication was clear and straight forward, and they could be relied on to respond swiftly to urgent or overnight requests.

Administrative overhead was minimal, as they were able to actively review and engage with ongoing feature development in our repos to add or update new tests. The DeviQA team was self-structured and self-driven, submitting updates and new code on a daily basis. If changes were needed to the existing plan or workflow, they were able and willing to adapt immediately.

What did you find most impressive about this company?

DeviQA had deep experience in javascript and test frameworks, and were able to develop a custom framework that included a number of tools selected carefully for reliable performance, CI integration, depth of reporting capabilities, and lightweight integration into an existing app repo.

The test suite they created was exhaustive and wide-spanning, and their ability to continue updating it with the latest delivered features was impressive. It was a pleasure to work with them and we would recommend them as test automation experts.

Are there any areas for improvement?

No

BACKGROUND

Introduce your business and what you do there.

I am the CTO of a financial firm. We are a private equity investor that has been in business since 1976. We have an extensive portfolio of businesses and assets spanning real estate, high tech, healthcare, retail, and other areas.

OPPORTUNITY / CHALLENGE

What challenge were you trying to address with DeviQA?

We had multifold problems finding and retaining good-quality QA and development resources. It is a pretty commonplace problem, particularly in Western Europe, and particularly magnified in the cities where there’s a large financial services sector. We are headquartered in London, which is possibly still the largest financial center in Europe.

SOLUTION

What was the scope of their involvement?

They’ve helped us on two, soon to be three, projects, in addition to helping us with raw development capabilities. We’ve used their test architects, test engineers, and some of the manual testing resources to bring our products to life. They have a fairly large team of on-demand developers, and some have been with us for over 12 months at this point. They also have wide-ranging skillsets in front- and backend technologies.

On the QA side, they’ve been responsible for scoping out, building, and running a fairly sophisticated automation platform for two of the companies in our portfolio. Effectively taking the project from end to end, they’ve helped with everything from the architecture phases to the operation of the automated QA testing.

What is the team composition?

We’ve had between 7–10 resources at the most extreme time. Currently, we have four or five people.

How did you come to work with DeviQA?

Finding people who are highly skilled, motivated, and affordable is incredibly difficult, and we had to cast our net further and further east. The owner of DeviQA had been in touch with me and was very respectful. He would approach me asking if I had any requirements, and even though I didn’t have anything at that time, I asked him to send me a message in three months. That process went on for about nine months. One day, I found that I needed him quite urgently, and he was there for me.

How much have you invested in them?

To date, we’ve spent around $500,000, and that number is going up every month as we continue to draw on their resources.

What is the status of this engagement?

We began working with them in September 2018, and the work is still ongoing.

RESULTS & FEEDBACK

What evidence can you share that demonstrates the impact of the engagement?

Because it has all been greenfield development, the best way to measure deliverables is how quickly you are getting product out of the door and delivered to the market. With what they have been doing, particularly on the full-cycle test and automated testing, they have done everything we have asked them to do on time and within budget. I think that is a good success marker. If you were to look at financial KPIs, we’ve never blown our budget with them, and they have always delivered when they said they were going to.

Sometimes in an outsourcing relationship, things go wrong. People can get sick, go on holiday— all of the usual stuff. Even making allowances for all of that, they have always done what they said they were going to do. At times when something looked like it was going to go off the rails, they threw their own resources at it for no cost to get back on track.

How did DeviQA perform from a project management standpoint?

They have been super reliable, responsive, and professional through our entire engagement. I keep giving them more work, which I think is always a pretty good indicator of how successful your relationship with the company is.

They were very good, but we never found ourselves having to stress any of their capabilities in that area. Their resources were working under our product and project leads.

What did you find most impressive about them?

They’re focused on delivering incredibly high quality. They don’t complain, and they do exactly what they know needs to be done. In a time-pressed environment, you can’t ask for more than that.

They’re also good at challenging us. They don’t do stuff because you tell them to do it; they will challenge it, acknowledge we want something done in a certain way, but will still suggest other ways to do it. That level of challenge is good.

Are there any areas they could improve?

I’ll say the same thing I would say about any outsourced relationship: communication. You always have to be on the front foot on that, but I know for a fact that it isn’t unique to them because I run several outsourced relationships. Given that they are a remote resource, they really try hard to make it feel like you are in the same room as them.

Do you have any advice for potential customers?

Make sure you have a really clear idea of what you are trying to do. Articulate that upfront rather than making it up as you go along. I think these guys thrive on having structure. Equally, if you can’t do that because of project timing, you can still work with them, but to get the most out of them, give them some structure.

BACKGROUND

Please describe your company and your position there.

I am a CTO of a company that provides software to companies in different industries that help them manage their talent acquisition and existing workforce. I'm responsible for managing the development process for our software.

OPPORTUNITY / CHALLENGE

For what projects/services did your company hire DeviQA?

We hired DeviQA early when we started, to have someone capable of making sure that our product has no issues during the development. Their role was to manually test all-new features and to cover them with an automated suite. The market did not provide us with good candidates to hire them in-house, so we have decided to outsource QA process.

What were your goals for this project?

There weren't any specific goals but in the end, we just wanted a good quality product.

SOLUTION

How did you select this vendor?

We have interviewed a number of potential vendors, but they stood out. They had the multi-disciplinary knowledge we required and were able to quickly suggest the solutions to our problems.

Describe the project in detail.

We needed QA to supplement our development process. Their responsibility was to test every new feature, to cover it with automated tests, and to be a part of Scrum teams. We were confident in their skills enough to give them the freedom to implement the QA process based on their experiences. They would make sure that our releases had everything fixed before the deployment, and once they were satisfied, we could proceed.

What was the team composition?

We have a number of teams of developers and SM/PO/BA people. We have also hired additional in-house QA people, but we still depend on DeviQA to make sure everything goes smooth.

RESULTS & FEEDBACK

Can you share any outcomes from the project that demonstrate progress or success?

Since this was a long-term and still ongoing relationship, I can only say that our ever-growing client base is evidence of the success of that relationship. 

How effective was the workflow between your team and theirs?

Very effective. They are quick to adapt to any changes we needed to make.

What did you find most impressive about this company?

Knowledge and skills aside, I think the most impressive was an ability to understand the industry-specific needs without a lot of input on our end. 

Are there any areas for improvement?

We're completely satisfied with DeviQA so far. 

BACKGROUND

Introduce your business and what you do there.

I’m a QA manager at a Swiss IT company that focuses on security solutions. We’ve developed a secure messenger app with video calling and file sharing features. Our main clients are governments, hospitals, and financial institutions. We’re a relatively small company, with about 30 people more or less, and I’m responsible for the entire process of testing and delivery.

OPPORTUNITY / CHALLENGE

What challenge were you trying to address with DeviQA?

There were too many things for us to test ourselves, and we couldn’t find enough good specialists in the region, so we outsourced some of the work to DeviQA. 

SOLUTION

What was the scope of their involvement?

DeviQA has helped us do several things. First of all, they worked on automation. We have a lot of test cases to automate—around 1,000 on Android and iOS each. We were doing this manually before hiring DeviQA, which amounted to a lot of time—around three weeks for three engineers. Their team has helped us automate everything, and they keep going, which means that we can basically run tests overnight.

They also assist with manual testing, as there are a lot of things to test before going to automation and to production. We’ve hired more testers from DeviQA in order to cover all product parts. Furthermore, they’re helping us lead a development project that we’ve outsourced somewhere else. One of DeviQA’s people is leading testing within that other company as well. 

Overall, they’re helping us a lot and I’m really happy we’ve hired DeviQA. I’d really love to hire more people from them, but we’re having some budget constraints. 

What is the team composition?

We’re working with three people at the moment: two automation testers and one manual tester. They work directly with me as a dedicated team, alongside the three testers on our side.  

How did you come to work with DeviQA?

I did a lot of research online and found DeviQA on Clutch. They had good feedback, so we started a small proof-of-concept project to see if they were as good as described. We liked them so much that we formally started working together, and we’ve been happy so far.

How much have you invested with them?

The cost so far has been between $300,000–$400,000 in total.

What is the status of this engagement?

We started working with DeviQA around two years ago. 

RESULTS & FEEDBACK

What evidence can you share that demonstrates the impact of the engagement?

I’m 100% happy with them. We’ve worked with lots of other companies, but DeviQA is really responsible and has independent staff. Most of the time, they perform really well and I have no issues with them at all.

I’ve included them in our development environment, which has two-week milestones. I can see what they’re doing every day and whether they’ve achieved those milestones on a weekly basis. This is mainly how we measure their performance.

How did DeviQA perform from a project management standpoint?

If they have a problem, they will contact me directly and we’ll resolve it immediately. Everything is running smoothly and I can contact their principal whenever I have an issue, which he’ll likely know the solution to. DeviQA is pretty responsible, though, so I don’t normally need to call their boss.

We’re using several tools, including our own internal chat tool. It’s encrypted, which makes it more secure for exchanging important information. We’re also using Skype and Slack. I’m a Russian speaker as well, which has helped me work with them, but DeviQA has also worked with some of our English-speaking clients and the experience was positive. The communication part is easy-going, with no problems or misunderstandings.

What did you find most impressive about them?

They’re a really competent and solid team. We need to be careful when working with offshore providers, as some of them overpromise, but DeviQA’s people do their jobs and are professionals in what they do. I can leave them on their own and go on vacation, and trust that everything will be done and documented properly. I haven’t had any problems concerning their comprehension or ability to execute a task.

Are there any areas they could improve?

At this stage, they could only help us improve our own processes. DeviQA has helped us understand how we can improve internally. 

Do you have any advice for future clients of theirs?

It’s important to run a demo project and see if they can do what’s being asked of them. This is true for any provider, though. DeviQA will tell us when they lack the expertise to do something. 

BACKGROUND

Please describe your company and your position there.

I am the VP of business development, and I have a project management background. 

WiserBrand is a business process outsourcing company that helps clients complement their business with a team of dedicated professionals. Our company has about 60 employees, and we serve clients all over the world.

OPPORTUNITY / CHALLENGE

For what projects/services did your company hire DeviQA?

We needed a reliable, skilled partner to be responsible for the quality of all of our applications. They would build a structural testing approach, design automated test suite architecture, and develop automated tests. They would also conduct performance and manual testing.

What were your goals for this project?

We wanted our applications to be high quality.

SOLUTION

How did you select this vendor?

We contacted many different agencies and asked them to conduct a test task. DeviQA did it excellently.

Describe the project in detail.

We have mobile and web applications. We didn’t have a formal structural process. We had worked with other agencies before, but their processes were not as good. 

Once we started with DeviQA, our in-house team felt that their process differed much from what we had before. DeviQA’s services included automated testing.

What was the team composition?

The team consisted of four senior manual QA engineers, six senior automated testing engineers, and one project manager.

RESULTS & FEEDBACK

Can you share any outcomes from the project that demonstrate progress or success?

The quality of our applications increased significantly. People started to recommend our applications thanks to DeviQA. We started getting perfect reviews in the Appstore and Google Play. Also, we needed to invest in automated testing to reduce manual work, and DeviQA did this perfectly and saved us money.

How effective was the workflow between your team and theirs?

Communication between our in-house team and the specialists from DeviQA was excellent. They are really fast. All of our team was happy with the workflow that they proposed.

What did you find most impressive about this company?

DeviQA is a high-class QA testing company. They are very experienced, skilled engineers. Their team is reliable and technical. Before DeviQA, I’d never met a partner I could trust with the quality of our apps.

Are there any areas for improvement?

No, they are perfect.

BACKGROUND

Please describe your company and your position there.

I’m the mobile applications manager at a mobile location data company.

OPPORTUNITY / CHALLENGE

For what projects/services did your company hire DeviQA?

We needed a partner to manage and do quality testing for our iOS and Android applications. They would also have to fix bugs and improve features reported by our users.

What were your goals for this project?

We wanted to maintain our apps’ full functionalities and ensure they complied with Twitter, Apple, and Google’s terms of use.

SOLUTION

How did you select this vendor?

I was not directly involved in the decision-making process. DeviQA was already working with us prior to me starting this role.

Describe the project in detail.

Devi QA works with three of our applications, two of which are for iOS and Android while the third is only for Android. They're responsible for quality testing release candidates for all of the applications and working alongside our developers to troubleshoot and resolve issues. The team also manages the App and Play Store reviews and assists in responding to user complaints.

What was the team composition?

We work with a team of QA testers, one of whom focuses on iOS while the other focuses on Android.

RESULTS & FEEDBACK

Can you share any outcomes from the project that demonstrate progress or success?

All of our applications are functioning at the best level I’ve seen since starting my role. We continue to hear positive feedback from users about the apps’ stability and long-time usability. The apps have strong ratings in the App and Play stores thanks to DeviQA’s team.

How effective was the workflow between your team and theirs?

Despite the time difference, there hasn’t been any shortage of communication. Our developers and their QA team are in sync and able to schedule time easily to discuss and plan things. 

What did you find most impressive about this company?

Their work ethic and attention to detail are outstanding. The two resources on our team have great knowledge of our applications and processes and have significantly improved our workflow and productivity. Though one of them joined the team recently, they’ve stepped up quickly and displayed an ability to thoroughly integrate themselves.

Are there any areas for improvement?

No specific improvements are needed. Everything has been top notch and exceeded all expectations.

BACKGROUND

Please describe your company and your position there.

I’m the senior quality manager responsible for our flagship product, a video-based sales learning and readiness platform. I manage the quality of our web and mobile clients as well as the backend services.

OPPORTUNITY / CHALLENGE

For what projects/services did your company hire DeviQA?

Our company is growing out of its startup phase. We’re signing larger clients and need to add features quickly to accommodate those client needs while maintaining the base level of quality that our initial clients have come to expect. So, we hired DeviQA to assist with testing.

What were your goals for this project?

Automation is the key factor in ensuring quality while delivering product features quickly and consistently. Our goal is to build a reliable set of automation regression tests across our mobile and web applications.

SOLUTION

How did you select this vendor?

I inherited this team when I came on board with the company several months ago and have been pleased with their performance since the beginning.

Describe the project in detail.

The project included the development of a test framework based on open source tools and open standards. I supplied all guidelines, but the team was responsible for all of the coding and maintenance of the automated tests. They currently maintain the framework, including integration into other tools, writing new tests, monitoring test executions, analyzing test failures, and updating outdated tests.

What was the team composition?

The team consists of three full-time automation engineers. I provide guidance and set priorities on tasks, which are divided among the team based on who has the best experience for each task. This makes managing the team very easy for me, and I’m able to deliver a concise report of accomplishments to our executive management.

RESULTS & FEEDBACK

Can you share any outcomes from the project that demonstrate progress or success?

The number of defects we find through the automation process before releasing to customers has been growing and we’re also able to spend more time writing comprehensive test plans and reducing the amount of manual testing required before each release.

How effective was the workflow between your team and theirs?

The workflow is very efficient and easy. We meet once a week for a status update and to discuss priorities. We communicate as needed throughout the week to discuss any issues and resolve problems blocking the team. There is a project manager available with whom I can discuss any problems with team members, but there haven’t been any. I’ve only needed to talk with him to add members to the team.

What did you find most impressive about this company?

Their talent pool is great. Each individual has a lot of experience and exceeds my expectations. They’re also able to support each other to stay engaged and to keep learning. Also, ramping up the team as needed is very easy.

Are there any areas for improvement?

I can't think of any.