QualityLogic

BACKGROUND

Please describe your company and your position there.

I am the Director of Engineering for a medium sized e-commerce company. We provide services to clients that want to sell their products on-line and other clients that want to market those products through an affiliate relationship.

OPPORTUNITY / CHALLENGE

For what projects/services did your company hire QualityLogic?

We utilized Quality Logic for staff augmentation to provide quality assurance testing on our application changes prior to deploying those changes to production.

SOLUTION

How did you select this agency and what were the deciding factors?

We found this agency through a local referral to help with staff augmentation while we worked on our automation testing and capabilities. They had the skills and approach needed to perform the manual testing functions that we needed.

Describe the project in detail and walk through the stages of the project.

They provided up to 4 quality assurance testers, a technical lead and a project manager to work on the manual testing for multiple major projects and application support efforts. At least one tester was assigned to each team and in some cases certain team members covered multiple projects simultaneously.

Describe the recruitment process in detail.

We discussed our projects with the account manager and project manager and they recommended the team members and approach.

RESULTS & FEEDBACK

Can you share any outcomes from the engagement that demonstrate progress or success?

The team worked well in performing the manual quality assurance work that we needed.

How effective was the workflow between your team and theirs?

Our workflow requires a high level of business process and systems knowledge. It took some time to gain that knowledge and once team members understood both the business process and systems the workflow began to look seamless.

What did you find most impressive or unique about this company?

We were impressed with the commitment of the team to ensure the success of the projects that they worked on.

Are there any areas for improvement or something they could have done differently?

Resource turnover had caused some issues early on.

BACKGROUND

Introduce your business and what you do there.

I’m at Tubi, a video streaming and entertainment service. Within the organization, I’m the senior director of product for core systems and foundation, the department responsible for our core video experience, international product experience, privacy compliance, and internal tools.

OPPORTUNITY / CHALLENGE

What challenge were you trying to address with QualityLogic?

Our goal was to increase our throughput of linear channels on Tubi. At the time, we were averaging about 15 channel launches per month, and our goal was to hit 200 linear channels by September 30th, 2022.

In order to hit that goal, we needed to increase the number of linear channels we were launching per month. However, our bottleneck was that we lacked QA support to QA our outputs before launching them into our platforms. Thus, we decided to hire QualityLogic to increase our QA capacity and hit that goal.

SOLUTION

What was the scope of their involvement?

QualityLogic has provided us with QA services for our platforms’ linear channels, which are similar to live TV channels. To launch those channels, we first get the stream of assets from partners such as ABC, NBC, and FOX, and our engineering team uploads those assets. Then, QualityLogic provides QA to ensure videos come through our 25 different platforms correctly.

Their QA process includes testing on all those platforms. Whenever they find an issue with things like subtitles, sound, or video quality, they create tickets and work with our engineering team to ensure that those issues are resolved.

Our goal was to launch 200 channels, but halfway into the project, we realized we had extra bandwidth, so we’ve also tasked QualityLogic with other things. Since hitting our goal, their team has been helping us support our existing channels, and they provide ongoing QA to ensure we don’t have any issues with our platforms.

What is the team composition?

We worked with two full-time QA agents, a part-time project manager, and a part-time program manager.

How did you come to work with QualityLogic?

Our director of QA at Tubi found them, contacted them, and handed them over to me to handle the contract negotiations. One of the things that stood out to me was that they were in the US, so we would be able to work the same hours.

How much have you invested with them?

So far, we’ve invested about $60,000, but we’ve budgeted around $250,000 for the year.

What is the status of this engagement?

We started working with them in July 2022, and our engagement is ongoing. We plan to use them for the rest of our fiscal year and beyond.

RESULTS & FEEDBACK

What evidence can you share that demonstrates the impact of the engagement? 

QualityLogic has exceeded our expectations. In order to hit our goals, we needed to increase our channel launch capacity from 12–15 channels per month to over 20 channels. However, they achieved a much higher throughput, reaching 30 channel launches per month and hitting our goal in half the time we had budgeted. By late September 2022, we already had 202 channels. They were a pivotal part of helping us achieve our goal; we wouldn’t have achieved it without them.

How did QualityLogic perform from a project management standpoint?

QualityLogic moves very fast. We’ve spent half the money we had originally budgeted because they’ve done more than we thought they would do, demonstrating that they’re a hyper-focused team. We’re very pleased with how fast they get the job done, which ultimately saves us money and allows us to have extra bandwidth to work on other projects.

Due to security issues, we can’t provide QualityLogic with direct access to Shortcut, our internal project management system. Instead, their team uses Excel to create tickets and pass them over to us to then create the tickets in our internal system. However, we’re working on creating a third-party integration that will enable them to pass those tickets directly to Shortcut so that we don’t have to enter each one manually.

As for communication, we have a shared Slack channel through which QualityLogic communicates directly with our engineering team. Additionally, their team joins relevant meetings via Zoom, and we have monthly and quarterly business review meetings with them.

What did you find most impressive about them?

QualityLogic has been very flexible and professional; they’ve done a fantastic job all around. They also have really talented people who have added to our culture in a positive way and become valued members of our team, ultimately improving our productivity.

Additionally, I really like that they’re able to scale their team up and down very quickly, and they don’t try to secure a long contract. Instead, we’re able to work with them month by month based on our needs.

Are there any areas they could improve?

No.

Do you have any advice for potential customers?

If QualityLogic joins your team, allow them to join completely. To get the most out of them, you’ll need to treat them as part of your team and give them access to the resources they need to do their job. When people are happy, they’re more engaged with their work, and they’re ultimately more productive.

BACKGROUND

Introduce your business and what you do there.

I’m a director at a robotics company.

OPPORTUNITY / CHALLENGE

What challenge were you trying to address with QualityLogic?

We engaged QualityLogic to help us with QA testing for our application.

SOLUTION

What was the scope of their involvement?

QualityLogic helped us with manual QA testing. The team performed regression and production testing and documented their findings in test cases. Once they identified bugs, they took screenshots and videos so our development team knew what to look for. Then, our product managers reviewed the test cases, and our development team implemented changes to address the bugs. When QualityLogic identified new bugs that affected the application’s functionality, they advised our product team to reopen the story and triage the bug. 

What is the team composition?

We worked with two QA engineers and a project manager from QualityLogic. 

How did you come to work with QualityLogic?

Our company engaged QualityLogic before I joined. Our team had heard positive feedback about QualityLogic through word of mouth, so we approached them to help us with the project. 

How much have you invested with them?

We invested between $10,000–$15,000 in the partnership with QualityLogic. 

What is the status of this engagement?

We worked with QualityLogic between December 2021–August 2022.

RESULTS & FEEDBACK

What evidence can you share that demonstrates the impact of the engagement? 

We measured QA success by looking at the success of each release. They reported all the bugs efficiently and caught most of them before each release. The QualityLogic team delivered thorough test cases and performed immaculate work. We had some problem cases, but those were exceptions.  

How did QualityLogic perform from a project management standpoint?

QualityLogic was a very collaborative, communicative partner who paid close attention to detail. They were kind and dependable, which made the process easier for us. We had daily stand-ups via Google Meet and used Jira to manage the project. We also communicated via Slack.

What did you find most impressive about them?

I was most impressed by their proactive communication style, reflected in the precision of their test cases. QualityLogic consistently followed up with us and took the initiative to loop in our developers as they saw fit. When they identified particularly problematic issues, they’d advise us to block the next release, which was helpful. Throughout the partnership, they kept our customers in mind, which helped us in the long run. 

Are there any areas they could improve?

I have nothing QualityLogic could improve on.

Do you have any advice for potential customers?

I’d recommend engaging with QualityLogic to experience the great outcomes for yourself.

BACKGROUND

Introduce your business and what you do there.

I’m a manager at P.volve, a company that offers solutions to women looking for low-impact exercises that can work with their lifestyle, health, and pace of life.

OPPORTUNITY / CHALLENGE

What challenge were you trying to address with QualityLogic?

We hired QualityLogic to provide app testing services for the app that we were creating. 

SOLUTION

What was the scope of their involvement?

In our current setup, we work with a manager from QualityLogic. Their resources do automated and manual testing for us. They serve as a part of our project team that develops our app. To do the tests, they use Cypress tools. We have web and mobile offerings, and they do tests for both. 

What is the team composition?

We work with a total of nine people from Quality Logic, including the manager, two QA engineers, and five manual testers. 

How did you come to work with QualityLogic?

I’ve only recently started at P.volve, and QualityLogic was already there when I joined.

How much have you invested with them?

We’ve invested between $1 million–$5 million. 

What is the status of this engagement?

We started working with QualityLogic in June 2020, and the engagement is ongoing. 

RESULTS & FEEDBACK

What evidence can you share that demonstrates the impact of the engagement? 

QualityLogic is highly efficient in their testing work. They put a lot of effort, and compared to the other quality testing vendors we’ve had, QualityLogic provides 10x the amount of work as other teams.  

Given that, my internal team loves them. They have the best people to work with; it’s been amazing working with them.

How did QualityLogic perform from a project management standpoint?

Their project management is excellent. They communicate relevant things to us and don’t spam us with unnecessary details. The team also gives us ample notice if they have to do more than 40 hours per person in a week so that we can adjust our budget and expectations accordingly.

They’re great at telling us what they’re trying to deliver. In a lot of ways, their work is our rock as we undergo massive shifts as a company over the last few months. Their output is the most solid piece that helps direct everything else around it.

In terms of tools, we primarily use Jira to manage the project and GitHub for source control. To communicate, we use Zoom, Slack, Outlook, and Confluence. 

What did you find most impressive about them?

QualityLogic is the best QA team I’ve ever worked with — I’ve been doing QA work for 25 years, and I’m stunned at how good they are at their job. They have everything that we want in a QA team; they’re flexible and always maintain good quality. 

Are there any areas they could improve?

No, there aren’t any. 

Do you have any advice for potential customers?

Understand that QualityLogic brings a huge amount of value to the table. If you go for a team that’s less expensive, you’re going to get better rates but won’t get value. Meanwhile, the QualityLogic team does many things above and beyond a standard organization. 

For instance, they supply their own mobile devices so that they can test the app on various types of phones. If somebody from their team has to go out for a week, they automatically have a backup that can step into the position. Overall, you’re going to get great value out of the money you invest in their services. 

BACKGROUND

Introduce your business and what you do there.

I’m a director with Inspirant Group, a consulting company.

OPPORTUNITY / CHALLENGE

What challenge were you trying to address with QualityLogic?

We were tasked with building a very large and complex application that required extensive testing. We didn’t have enough resources internally to accomplish this, so we decided to hire QualityLogic to help fill this gap.

SOLUTION

What was the scope of their involvement?

We hired QualityLogic to help test the application from all aspects. This involves testing the different types of web browsers that the app can be used with and testing all of its components to come up with a way to break the system so we can improve it. They work in an agile format, so they do individual and regression testing every two weeks.

What is the team composition?

We work with an automation tester, an automation engineer, two system testers, and a project manager from QualityLogic

How did you come to work with QualityLogic?

We found QualityLogic through an online search. We decided to go with QualityLogic for this project based on their responses in our interview.

How much have you invested with them?

We’ve spent somewhere around $100,000.

What is the status of this engagement?

We started working together in August 2021, and the engagement is ongoing.

RESULTS & FEEDBACK

What evidence can you share that demonstrates the impact of the engagement?

QualityLogic has done a tremendous amount of individual and regression testings, and they’ve successfully identified several defects or bugs within the application that we are actively working to fix.

How did QualityLogic perform from a project management standpoint?

QualityLogic has been great in terms of project management. They’re self-sufficient and incredibly flexible in their ability to maneuver people to accommodate our needs. Their communication has also been fantastic, and they’re very comprehensive in their work. They join all of our meetings and provide detailed updates on their test cases and all of the bugs they’ve identified.

What did you find most impressive about them?

I’m most impressed with QualityLogic’s expertise in pure testing. That, combined with their excellent communication and the breadth of what they can test, really stood out to us.

Are there any areas they could improve?

I don’t think there’s anything they could have done better, and we’re looking forward to doing additional work with them in the future.

Do you have any advice for potential customers?

Make sure to have a good sense of what your needs are before engaging QualityLogic. They have a breadth of expertise in different areas that will help them identify the right people to step on your project if you let them know what you're ultimately going to need.

BACKGROUND

Introduce your business and what you do there.

I’m working for Spin Electric Scooter Company as the acting quality manager of the company, handling all our QA efforts.    

OPPORTUNITY / CHALLENGE

What challenge were you trying to address with QualityLogic?

They provide QA resources for our two mobile apps.

SOLUTION

What was the scope of their involvement?

They do regression testing for new releases, which happen every two weeks, and they’re also writing test cases for all new features. They handle most of our QA and regression testing work.

We coordinate the work with releases coming up, looking at features to test and adding in additional test cases if needed. They’re fully involved in all scenarios.

What is the team composition?

We had to reduce the team last year, but we started with 6 QA engineers. Now, we have two QA resources on our team. Our budget shrank; they didn’t do anything wrong.

We work with a QA lead and a full-time QA engineer. When we were working with six people, there were five full-time engineers. If our situation changes, I will certainly ask for more people.

How did you come to work with QualityLogic?

They were already onboard when I joined the team, but they were brought on by our previous quality manager who’d worked with them at a previous company.

What is the status of this engagement?

We’ve been working together since November 2019.

RESULTS & FEEDBACK

What evidence can you share that demonstrates the impact of the engagement? 

Without their help, we wouldn’t be able to achieve our needs at the moment. They have full-time resources running our test cases, and we couldn’t complete this work without them. They’re hard-working people who are dedicated.

How did QualityLogic perform from a project management standpoint?

They work remotely from a different state. We communicate easily, through daily Zoom calls and Slack messages every day. We have a very good collaboration working remotely.

What did you find most impressive about them?

I haven’t worked with other contractors, so I don’t have anyone to compare them to, but I don’t treat them as some third-party I’m working with. I treat them like part of our team. Our entire team treats them as partners.

Are there any areas they could improve?

I don’t have any improvements. They do a great job on all their assignments.

Do you have any advice for potential customers?

I would recommend them if anyone needed help testing their applications. I don’t have any hesitations.

BACKGROUND

Introduce your business and what you do there.

I’m the VP of engineering at BrightLine, an ad tech company within the OTT space.

OPPORTUNITY / CHALLENGE

What challenge were you trying to address with QualityLogic?

We were lacking internal QA support, so we hired QualityLogic.

SOLUTION

What was the scope of their involvement?

QualityLogic tests different software development kit (SDK) products for different platforms, including Apple TV, Android, Roku. They test them on different devices as well. We’ll usually have a kickoff session to gain an understanding of the product and what it should do and how things should flow. 

The team then creates test cases and proceeds to do tests on a per-platform basis. Throughout that process, if there are any questions or ambiguities, they’ll stay engaged to clarify them.

What is the team composition?

We typically work with one QA lead, but it could be a combination of two at any given point.

How did you come to work with QualityLogic?

Our previous CTO onboarded them, and I picked up where he left off.

How much have you invested with them?

The cost ranges depending on the level of support we need. Typically, we engage them for at least 20 hours per week, but it can go up to 40 or more hours. We spend $5,000–$10,000 per month with them, and the total cost so far is in the $50,000–$200,000 range.

What is the status of this engagement?

We started working with QualityLogic around October 2018, and our engagement is ongoing.

RESULTS & FEEDBACK

What evidence can you share that demonstrates the impact of the engagement?

It’s been tremendous. Before working with QualityLogic, we had a single QA teammate internally, and we were experiencing a lot of bugs and complaints from media partners. 

We saw a huge decrease in complaints after we brought QualityLogic onboard, and we saw a big increase in the quality of our releases. It’s been a tremendous impact overall because they’ve helped us decrease bugs, processes, and things of that nature.

How did QualityLogic perform from a project management standpoint?

We discuss what should be tested during the week, and they give us daily updates on any issues they find and report them in Jira. QualityLogic also gives detailed descriptions of how they found the issue and where, along with how to reproduce it. They have a very good process for that.

We have weekly sessions every Monday, and we have a lot of communication over email, which is pretty clear. We also hop on phone calls if we can’t get the understanding through email. 

What did you find most impressive about them?

This is the first QA team I’ve worked with while at the company, but they’ve definitely been better in terms of process and tracking compared to our internal resources.

Are there any areas they could improve?

It’s not necessarily a bad thing, but, in terms of the collaboration between our companies, having more scheduled sessions would help.

Although we had weekly meetings, more frequent sessions throughout the week would be good as well. It can take me longer to get to an email if I’m swamped with work, so having calls and screen shares would make it easier to describe elements of the project.

Do you have any advice for future clients of theirs?

QualityLogic is very in-depth with the products they QA, but we may have an urgent release, where identifying blockers becomes most important. It’s important to understand the product in terms of what’s critical and what isn’t and how that can impact other aspects.

BACKGROUND

Introduce your business and what you do there.

Our organization provides online invitations to events, and I am a mobile QA manager for our apps on both iOS and Android.

OPPORTUNITY / CHALLENGE

What challenge were you trying to address with QualityLogic?

We were a little bit shorthanded and were looking for some help with our regression testing and regular tickets, so we reached out to QualityLogic for that.

SOLUTION

What was the scope of their involvement?

Our biggest pain point was that our regression testing was taking a long time, and we were essentially able to just hand that off to QualityLogic for their team to run with. Our product had developed a lot of new features in a fairly short amount of time, but they were able to really build out nuanced test cases to deal with the complexity of the application and then run manual testing from there.

What is the team composition?

There are three individuals from their team on our project, two of whom work full time on QA and one who is the project manager.

How did you come to work with QualityLogic?

I scheduled phone calls with several companies that I found on the internet, and we really liked their business model what their team had to say so we decided to work with them.

What is the status of this engagement?

We started in October 2019 and the engagement is ongoing.

RESULTS & FEEDBACK

What evidence can you share that demonstrates the impact of the engagement?

With the help of QualityLogic, we’ve been able to accelerate our update schedule from around monthly releases to every two weeks. That’s a really significant jump for us, so we are highly satisfied with the quality of the work so far.

How did QualityLogic perform from a project management standpoint?

For the most part, communication has been very straight to the point and direct, which we really appreciate. Not having to filter through a lot of business talk saves us a lot of time and makes sure we are able to address each problem or improvement that they raise. We use a JIRA board to track progress as we go.

In addition, their tickets are incredibly detailed, which makes it easy to read through them and understand exactly what is going on. That allows for minimized back and forth over specific issues so that we can use Slack and email to discuss larger changes and strategies without distraction.

What did you find most impressive about them?

I’ve worked with other remote companies that handle QA before, and I think that QualityLogic has been much better prepared than almost any others. They fundamentally understand QA, so I don’t have to double-check their work at all.

Are there any areas they could improve?

I’ve been really happy with them, so I don’t have anything I think they need to improve.

Do you have any advice for potential customers?

Provide as much feedback as you can early on if there’s anything you want from them that isn’t getting done. As long as you give clear directions and communicate throughout the project, you’re going to be satisfied with the work that they do.

BACKGROUND

Introduce your business and what you do there.

We are a growth assessment provider for school districts, helping to measure areas of opportunity for students that teachers can focus on. We deliver assessments through a hosted solution, and much of our software is web application based. I'm a QA manager.

OPPORTUNITY / CHALLENGE

What challenge were you trying to address with QualityLogic?

We hired them to provide more coverage in our operating system and browser compatibility testing. We have multiple development teams who focus on functional and regression testing, but we don't always have time to test operating systems, browsers, and devices.

SOLUTION

What was the scope of their involvement?

QualityLogic runs compatibility tests on a monthly basis to make sure we have no regression in any of the operating systems or browsers we support. As the partnership has gone on, we've added more coverage across our product suite: roles testing, new feature testing, and testing to resolve production issues. They're also extending our accessibility testing to ensure good keyboard navigation and screen reader coverage for student assessments. We hope to expand that as we grow our offerings. We continue to find uses for them, and they plug holes in our coverage.

What is the team composition?

We work with a project leader. She coordinates a team of about three testers, and they allow some flexibility when we need to scale.

How did you come to work with QualityLogic?

I found them via a web search. We identified vendors who could meet our need, and we completed an RFP process. QualityLogic stood out from the beginning, and we were very impressed by their services and abilities.

How much have you invested with them?

We've spent approximately $70,000 on their services.

What is the status of this engagement?

The first official engagement started in January 2019, and they've done testing with us in every month since.

RESULTS & FEEDBACK

What evidence can you share that demonstrates the impact of the engagement?

We track the number of test runs they complete, as well as the defects they find. We've improved both sides of the process, and are pleased with the progress.

How did QualityLogic perform from a project management standpoint?

They're very communicative, and their status reports have the perfect amount of information. We have a daily standup with them, and we also use RingCentral to send messages throughout the day. They're very mindful of resources, and they only make changes when we give them the go-ahead. They continue to improve their processes and are very efficient.

What did you find most impressive about them?

This is a unique collaboration for us, but their flexibility, experience, attention to detail, and communication all stand out.

Are there any areas they could improve?

We haven't run into anything. There has been a learning curve on our side, but I can't think of anything they should improve or change.

Do you have any advice for potential customers?

Make sure that you are clear about your own processes. QualityLogic is more than willing to help you as you solidify those processes, and it is useful to maintain an open communication mechanism for feedback.

BACKGROUND

Introduce your business and what you do there.

I work for a product consulting company called Sidebench. We do product design and development for our clients.

OPPORTUNITY / CHALLENGE

What challenge were you trying to address with QualityLogic?

We’re a fairly nimble company, so we typically bring in outside resources to staff a project. We were building a mobile and web application for a large Hollywood production company. Due to a three-month deadline, we needed to ramp up resources with an outside team very quickly. This meant we needed a team with consistent, high-quality work to which we could delegate testing responsibilities.  

SOLUTION

What was the scope of their involvement?

QualityLogic’s team performed rigorous, manual regression testing. They drew up their QA testing plans and wrote test scripts in a silo. As we got closer to launch, we brought them closer to our team.

What is the team composition?

We had one point of contact. He worked with two engineers who did a large portion of the manual work. We also had direct communication with Joe (General Manager, QualityLogic), a senior member of the team who worked with resourcing, customer service, and account management.

How did you come to work with QualityLogic?

We’ve worked with them in the past, and they’ve become our go-to team with these types of projects. It was my first time working with them, but they were highly recommended by another member of my team. Picking QualityLogic was a no-brainer, it was just a matter of how quickly we could get started.

How much have you invested with them?

We’ve spent around $25,000 with them.

What is the status of this engagement?

The project began in early April 2019 and is still ongoing. This is the most recent extension of a contract we have with them. We’ve extended the contract 3–4 times.

RESULTS & FEEDBACK

What evidence can you share that demonstrates the impact of the engagement?

They helped us achieve a huge goal in launching the Hollywood production app in a short period of time. Due to that success, we’ve negotiated another $300,000 contract from the client. Without being able to pass that initial launch, we wouldn’t have gotten any additional contracts. They've put in a ridiculous amount of time for us.

At first, our development team thought QualityLogic was overly picky or going too far into detail, but when the first engagement ended, our team asked for them to come back immediately because they knew how much value they added. I think that showed that the level of detail they provide makes things easier for all of us. They do such a robust job of testing different devices and browsers. Every ticket has a video or a screenshot associated with it. They’re saving us mental energy, effort, and time.

How did QualityLogic perform from a project management standpoint?

I was in regular communication with the main contact at QualityLogic every single day. We mostly used Jira and Slack to communicate.

They were always flexible with my hours and what I was working on. From a project planning perspective, they’re so flexible. I think that’s something they pride themselves on. There’ve been several instances where I’ve set an ambitious goal, and they were quickly able to adjust their resources, build a new test plan, and adapt.

What did you find most impressive about them?

The ability to ramp up and ramp down has been very important for us. During regression testing phases, we’re able to easily add testers. When testing is done, we can ramp back down to a single person. The resourcing plans are very accommodating to our needs. Along with this, it feels like they’re truly bought in and care about these projects. It’s not just another contract to them, and that’s very impressive given the number of projects they have.

Are there any areas they could improve?

No, it’s been a really easy process. Sometimes from a project burn and hours perspective, it was a bit high, but they always did a great job communicating that. There was never any time poorly spent; if they had no more tasks, they would tell me they were doing some other task. I’ve honestly not had any complaints.

Do you have any advice for potential customers?

You should be as detailed in your requests as possible. Ask what your best-case scenario would be. Sometimes I felt like we were asking for too much, but they were able to easily meet our demands. If you ask what you’d ideally like to happen, they’d always be able to accommodate.

When you’re forming the partnership, set out the ground rules and your own expectations. I’d advise having an open conversation about how closely they can meet those expectations, which is a testament to how flexible, knowledgeable, and caring they are in this line of work.