QualityLogic

Introduce your business and what you do there.

I’m the VP of engineering at BrightLine, an ad tech company within the OTT space.

What challenge were you trying to address with QualityLogic?

We were lacking internal QA support, so we hired QualityLogic.

What was the scope of their involvement?

QualityLogic tests different software development kit (SDK) products for different platforms, including Apple TV, Android, Roku. They test them on different devices as well. We’ll usually have a kickoff session to gain an understanding of the product and what it should do and how things should flow. 

The team then creates test cases and proceeds to do tests on a per-platform basis. Throughout that process, if there are any questions or ambiguities, they’ll stay engaged to clarify them.

What is the team composition?

We typically work with one QA lead, but it could be a combination of two at any given point.

How did you come to work with QualityLogic?

Our previous CTO onboarded them, and I picked up where he left off.

How much have you invested with them?

The cost ranges depending on the level of support we need. Typically, we engage them for at least 20 hours per week, but it can go up to 40 or more hours. We spend $5,000–$10,000 per month with them, and the total cost so far is in the $50,000–$200,000 range.

What is the status of this engagement?

We started working with QualityLogic around October 2018, and our engagement is ongoing.

What evidence can you share that demonstrates the impact of the engagement?

It’s been tremendous. Before working with QualityLogic, we had a single QA teammate internally, and we were experiencing a lot of bugs and complaints from media partners. 

We saw a huge decrease in complaints after we brought QualityLogic onboard, and we saw a big increase in the quality of our releases. It’s been a tremendous impact overall because they’ve helped us decrease bugs, processes, and things of that nature.

How did QualityLogic perform from a project management standpoint?

We discuss what should be tested during the week, and they give us daily updates on any issues they find and report them in Jira. QualityLogic also gives detailed descriptions of how they found the issue and where, along with how to reproduce it. They have a very good process for that.

We have weekly sessions every Monday, and we have a lot of communication over email, which is pretty clear. We also hop on phone calls if we can’t get the understanding through email. 

What did you find most impressive about them?

This is the first QA team I’ve worked with while at the company, but they’ve definitely been better in terms of process and tracking compared to our internal resources.

Are there any areas they could improve?

It’s not necessarily a bad thing, but, in terms of the collaboration between our companies, having more scheduled sessions would help.

Although we had weekly meetings, more frequent sessions throughout the week would be good as well. It can take me longer to get to an email if I’m swamped with work, so having calls and screen shares would make it easier to describe elements of the project.

Do you have any advice for future clients of theirs?

QualityLogic is very in-depth with the products they QA, but we may have an urgent release, where identifying blockers becomes most important. It’s important to understand the product in terms of what’s critical and what isn’t and how that can impact other aspects.

Introduce your business and what you do there.

Our organization provides online invitations to events, and I am a mobile QA manager for our apps on both iOS and Android.

What challenge were you trying to address with QualityLogic?

We were a little bit shorthanded and were looking for some help with our regression testing and regular tickets, so we reached out to QualityLogic for that.

What was the scope of their involvement?

Our biggest pain point was that our regression testing was taking a long time, and we were essentially able to just hand that off to QualityLogic for their team to run with. Our product had developed a lot of new features in a fairly short amount of time, but they were able to really build out nuanced test cases to deal with the complexity of the application and then run manual testing from there.

What is the team composition?

There are three individuals from their team on our project, two of whom work full time on QA and one who is the project manager.

How did you come to work with QualityLogic?

I scheduled phone calls with several companies that I found on the internet, and we really liked their business model what their team had to say so we decided to work with them.

What is the status of this engagement?

We started in October 2019 and the engagement is ongoing.

What evidence can you share that demonstrates the impact of the engagement?

With the help of QualityLogic, we’ve been able to accelerate our update schedule from around monthly releases to every two weeks. That’s a really significant jump for us, so we are highly satisfied with the quality of the work so far.

How did QualityLogic perform from a project management standpoint?

For the most part, communication has been very straight to the point and direct, which we really appreciate. Not having to filter through a lot of business talk saves us a lot of time and makes sure we are able to address each problem or improvement that they raise. We use a JIRA board to track progress as we go.

In addition, their tickets are incredibly detailed, which makes it easy to read through them and understand exactly what is going on. That allows for minimized back and forth over specific issues so that we can use Slack and email to discuss larger changes and strategies without distraction.

What did you find most impressive about them?

I’ve worked with other remote companies that handle QA before, and I think that QualityLogic has been much better prepared than almost any others. They fundamentally understand QA, so I don’t have to double-check their work at all.

Are there any areas they could improve?

I’ve been really happy with them, so I don’t have anything I think they need to improve.

Do you have any advice for potential customers?

Provide as much feedback as you can early on if there’s anything you want from them that isn’t getting done. As long as you give clear directions and communicate throughout the project, you’re going to be satisfied with the work that they do.

Introduce your business and what you do there.

We are a growth assessment provider for school districts, helping to measure areas of opportunity for students that teachers can focus on. We deliver assessments through a hosted solution, and much of our software is web application based. I'm a QA manager.

What challenge were you trying to address with QualityLogic?

We hired them to provide more coverage in our operating system and browser compatibility testing. We have multiple development teams who focus on functional and regression testing, but we don't always have time to test operating systems, browsers, and devices.

What was the scope of their involvement?

QualityLogic runs compatibility tests on a monthly basis to make sure we have no regression in any of the operating systems or browsers we support. As the partnership has gone on, we've added more coverage across our product suite: roles testing, new feature testing, and testing to resolve production issues. They're also extending our accessibility testing to ensure good keyboard navigation and screen reader coverage for student assessments. We hope to expand that as we grow our offerings. We continue to find uses for them, and they plug holes in our coverage.

What is the team composition?

We work with a project leader. She coordinates a team of about three testers, and they allow some flexibility when we need to scale.

How did you come to work with QualityLogic?

I found them via a web search. We identified vendors who could meet our need, and we completed an RFP process. QualityLogic stood out from the beginning, and we were very impressed by their services and abilities.

How much have you invested with them?

We've spent approximately $70,000 on their services.

What is the status of this engagement?

The first official engagement started in January 2019, and they've done testing with us in every month since.

What evidence can you share that demonstrates the impact of the engagement?

We track the number of test runs they complete, as well as the defects they find. We've improved both sides of the process, and are pleased with the progress.

How did QualityLogic perform from a project management standpoint?

They're very communicative, and their status reports have the perfect amount of information. We have a daily standup with them, and we also use RingCentral to send messages throughout the day. They're very mindful of resources, and they only make changes when we give them the go-ahead. They continue to improve their processes and are very efficient.

What did you find most impressive about them?

This is a unique collaboration for us, but their flexibility, experience, attention to detail, and communication all stand out.

Are there any areas they could improve?

We haven't run into anything. There has been a learning curve on our side, but I can't think of anything they should improve or change.

Do you have any advice for potential customers?

Make sure that you are clear about your own processes. QualityLogic is more than willing to help you as you solidify those processes, and it is useful to maintain an open communication mechanism for feedback.

Introduce your business and what you do there.

I work for a product consulting company called Sidebench. We do product design and development for our clients.

What challenge were you trying to address with QualityLogic?

We’re a fairly nimble company, so we typically bring in outside resources to staff a project. We were building a mobile and web application for a large Hollywood production company. Due to a three-month deadline, we needed to ramp up resources with an outside team very quickly. This meant we needed a team with consistent, high-quality work to which we could delegate testing responsibilities.  

What was the scope of their involvement?

QualityLogic’s team performed rigorous, manual regression testing. They drew up their QA testing plans and wrote test scripts in a silo. As we got closer to launch, we brought them closer to our team.

What is the team composition?

We had one point of contact. He worked with two engineers who did a large portion of the manual work. We also had direct communication with Joe (General Manager, QualityLogic), a senior member of the team who worked with resourcing, customer service, and account management.

How did you come to work with QualityLogic?

We’ve worked with them in the past, and they’ve become our go-to team with these types of projects. It was my first time working with them, but they were highly recommended by another member of my team. Picking QualityLogic was a no-brainer, it was just a matter of how quickly we could get started.

How much have you invested with them?

We’ve spent around $25,000 with them.

What is the status of this engagement?

The project began in early April 2019 and is still ongoing. This is the most recent extension of a contract we have with them. We’ve extended the contract 3–4 times.

What evidence can you share that demonstrates the impact of the engagement?

They helped us achieve a huge goal in launching the Hollywood production app in a short period of time. Due to that success, we’ve negotiated another $300,000 contract from the client. Without being able to pass that initial launch, we wouldn’t have gotten any additional contracts. They've put in a ridiculous amount of time for us.

At first, our development team thought QualityLogic was overly picky or going too far into detail, but when the first engagement ended, our team asked for them to come back immediately because they knew how much value they added. I think that showed that the level of detail they provide makes things easier for all of us. They do such a robust job of testing different devices and browsers. Every ticket has a video or a screenshot associated with it. They’re saving us mental energy, effort, and time.

How did QualityLogic perform from a project management standpoint?

I was in regular communication with the main contact at QualityLogic every single day. We mostly used Jira and Slack to communicate.

They were always flexible with my hours and what I was working on. From a project planning perspective, they’re so flexible. I think that’s something they pride themselves on. There’ve been several instances where I’ve set an ambitious goal, and they were quickly able to adjust their resources, build a new test plan, and adapt.

What did you find most impressive about them?

The ability to ramp up and ramp down has been very important for us. During regression testing phases, we’re able to easily add testers. When testing is done, we can ramp back down to a single person. The resourcing plans are very accommodating to our needs. Along with this, it feels like they’re truly bought in and care about these projects. It’s not just another contract to them, and that’s very impressive given the number of projects they have.

Are there any areas they could improve?

No, it’s been a really easy process. Sometimes from a project burn and hours perspective, it was a bit high, but they always did a great job communicating that. There was never any time poorly spent; if they had no more tasks, they would tell me they were doing some other task. I’ve honestly not had any complaints.

Do you have any advice for potential customers?

You should be as detailed in your requests as possible. Ask what your best-case scenario would be. Sometimes I felt like we were asking for too much, but they were able to easily meet our demands. If you ask what you’d ideally like to happen, they’d always be able to accommodate.

When you’re forming the partnership, set out the ground rules and your own expectations. I’d advise having an open conversation about how closely they can meet those expectations, which is a testament to how flexible, knowledgeable, and caring they are in this line of work.

Introduce your business and what you do there.

I’m the technical and managing director of the OpenADR Alliance. We’re a membership-based industry standards group for the smart goods space which includes devices like smart thermostats and electric-vehicle charger stations.

What challenge were you trying to address with QualityLogic?

One of our core organizational functions is to organize tests and certify products according to our standards. Almost nine years ago, we made the decision to develop and own a specific certification test tool that we would use to certify products. We needed a partner to work with us to develop it.

What was the scope of their involvement?

We’ve been working with QualityLogic for almost nine years now. They initially built the test tool for us including all of the testing, the test plan, and some test cases. We now co-own the test tool with them and QualityLogic serves as an endorsed partner that can test products for our members based on our standards and certify them. They’ve delivered that tool to a number of our member companies. They provide consulting services and additional development support to our members upon request. We also maintain a working relationship to update the software every couple of years and do other, smaller projects as needed.

What is the team composition?

We work directly with a lead engineer who serves as a project manager and with the general manager. There are other programmers who work behind the scenes.

How did you come to work with QualityLogic?

We put out an RFP to a number of test equipment companies as well as test houses. We didn’t want to just hire a programmer to build the test tool; we wanted to establish a partnership. We understood that we would need support and modifications over the years and that a long-term arrangement would be more effective. We eventually selected QualityLogic based on a combination of their experience, their pricing, and the partnership arrangement they offered.

How much have you invested with them?

We’ve spent about $400,000.

What is the status of this engagement?

We started working together in July or August 2011 and it’s ongoing.

What evidence can you share that demonstrates the impact of the engagement?

We haven’t specifically tracked performance but the test tool works very well. I get very few questions about it, if at all, and we’ve had a positive response.

How did QualityLogic perform from a project management standpoint?

They’ve performed well. Working directly with one of their lead developers has worked well for us. He’s always been on top of things. They’ve worked well according to timelines and plans.

What did you find most impressive about them?

I would say two things. For one, they’re one of the few companies I’ve found with the experience in test systems and testing programs of our sort. QualityLogic seemed to be the one company that had the most relevant testing experience. They also wanted to dive into the smart goods space, which was great for us.

Also, they have the right people on their team. Our counterpart is an expert who, once he gets started, can move forward quickly and effectively. He’s incredibly knowledgeable and easy to communicate with.

Are there any areas they could improve?

I can’t think of anything in particular. We’ve been working together for such a long time and everything moves the way we’d like it to. They could potentially be a bit better at tracking their hours. It would be nice to have advance warning if we’re going to exceed our estimates. But I know it’s sometimes hard to quantify work when you’re building something from scratch. The standards were changing as they built the test tool.

Do you have any advice for potential customers?

The challenge is always to scope a project well, which can be tricky given so many unknowns. Make sure your project is fully outlined and most aspects are clear. That way there won’t be any surprises.

Introduce your business and what you do there.

I’m the director of software systems quality for a transportation company.

What challenge were you trying to address with QualityLogic?

We wanted a nearshore vendor as an affordable alternative for testing resources.

What was the scope of their involvement?

QualityLogic tested our web and mobile applications, specifically the latter’s flow. Tangentially, the team also looked at our automation infrastructure.

What is the team composition?

We worked with a project manager and an account executive.

How did you come to work with QualityLogic?

I had worked with QualityLogic at another company and their mobile testing expertise stood out. We selected the team for this project because they clearly understood our business domain.

How much have you invested with them?

We spent between $50,000–$199,999.

What is the status of this engagement?

We started working with them in March 2018 and the partnership ended in January 2019.

What evidence can you share that demonstrates the impact of the engagement?

We tracked results via daily touch points to ensure proper oversight and clarification. Our side actively monitored the testing and execution.

How did QualityLogic perform from a project management standpoint?

QualityLogic integrated seamlessly with our business and provided high-quality services. Their attention to detail was notable.

What did you find most impressive about them?

The high-energy team was engaged throughout the partnership. They asked probing questions and proactively communicated any obstacles or challenges.

Are there any areas they could improve?

The partnership ended before we wanted, which limited QualityLogic’s involvement in the automation initiative. We wanted them to take a more active role with that.

Do you have any advice for potential customers?

Constant engagement with QualityLogic keeps the partnership on track. Clients that treat them more like an integrated team than an outsourcing vendor will get better results.

Introduce your business and what you do there.

I’m the CTO of Direct Care Innovations. We’re a healthcare technology company offering SaaS-based software to service providers in the Medicaid and Medicare markets.

What challenge were you trying to address with QualityLogic?

We were in the testing phase for multiple projects and didn’t have enough internal resources to perform quality tests. We wanted to outsource to a vendor who had the skills we lacked.

What was the scope of their involvement?

QualityLogic test all of our new developments and bug solutions. They conduct tests on our products’ functionalities, performance, and security. Their team uses their own testing tools and show us analytics on the success of our products. The reports show if bugs have been solved and if a system functions correctly. When they find an issue, they screencast with our developers to make the locating and reproducing process simpler.

What is the team composition?

They provide us with five full-time testers and one part-time tester, who also functions as the project lead. Their team is on constant retainer, so there is always a member available to help us.

How did you come to work with QualityLogic?

I searched for vendors online after performing marketing research. One of our system analysts found them, and we preferred their smaller composition compared to the other behemoth-like companies. We also liked their flexibility toward an open-ended arrangement with an undefined scope of work.

How much have you invested with them?

We’ve spent around $150,000, so far.

What is the status of this engagement?

We started in March 2017, and we keep them on retainer for our testing needs.

What evidence can you share that demonstrates the impact of the engagement?

We’ve been able to increase the number of test cases we get through drastically. Along with quantity increasing, our KPIs show that the thoroughness of the tests is much better than we were doing previously. Our customers have given us positive feedback on the enhanced usability of our system’s overall performance and security.

How did QualityLogic perform from a project management standpoint?

They provide us with detailed documentation about the issues they find and give us quality suggestions for solutions. We receive daily updates from the project lead, which is one of the best parts of working with them. They deliver everything on time and within the budget we set.

What did you find most impressive about them?

Their entire team is professional, knowledgeable, and willing to go the extra mile—we consider them a partner. We rely on them heavily, and they’ve never let us down, so we find the project to be money well-spent.

Are there any areas they could improve?

Although there have been inevitable bugs that went undetected, it’s infrequent, and they give us a reasonable explanation. Our work is complex, and there will always be some issues that slip through—we can’t blame them for that.

Introduce your business and what you do there.

I'm the chief operating officer for High Fidelity, a San Francisco-based startup in the virtual reality space that's building a platform for free servers running virtual reality environments. We have about 50 employees, 24 of whom are engineers.

What challenge were you trying to address with QualityLogic?

We wanted to scale up our QA testing for the development work we're doing without expanding our in-house team too quickly. We wanted to keep a good balance of both engineering and QA staff.

What was the scope of their involvement?

QualityLogic provided additional QA support for already developed software in C++. They execute test plans that we develop to verify that the code our engineers produce works as it's intended to. These tests require due diligence to ensure we're meeting specific outcomes of each feature within our platform. They report back to us, alerting us to any bugs or broken code.

What is the team composition?

Paul (Client Program Manager, QualityLogic) and Joe (Director of Program Management, QualityLogic) are our main point of contacts, but there are an additional six people from their team working on QA for us full-time.

How did you come to work with QualityLogic?

We had worked with another provider previously but were unhappy with their results. The rate at which they would go through tests and the consistency of their responses on the tests wasn't what we wanted, so we searched online and interviewed a few other vendors. We chose to work with QualityLogic in parallel with our previous vendor at first as a comparison, but ultimately decided to solely work with them based on their performance.

How much have you invested with them?

We work with them on a month to month agreement under a budget of $500,000 per year.

What is the status of this engagement?

We've been working with them since November 2017.

What evidence can you share that demonstrates the impact of the engagement?

QualityLogic improved our approach to QA testing, helping us rewrite hundreds of previously deficient tests as a result. The number of issues they've identified runs into the hundreds, which has helped us significantly improve the quality of every release in the process. They also process tests at an impressive rate.

How did QualityLogic perform from a project management standpoint?

They're very practical in how they deal with unexpected situations and have two people on their staff trained on our technology should they need to replace or add a couple QA staff to our projects. That approach is very sensible and shows that they're resilient to change.

When we've needed overtime from their current staff, they've been very responsive. Everything seems very well managed.

What did you find most impressive about them?

Their ability to deploy resources to help us build the test suite has been invaluable—they're a great partner in that way. Rather than just providing staff to run through each step of the test, they're able to add value by helping us build the system we're using for testing.

Are there any areas they could improve?

They're very sensible in the way they have a backup plan for replacing any staff on our project. They could, perhaps, be more transparent and alert us as those changes occur just so we're all on the same page and don't have any questions as to why a particular person is no longer on our project.

Introduce your business and what you do there.

We are a large associate development company, with locations all around the world. I work in software quality.

What challenge were you trying to address with QualityLogic?

We needed to support a new file format in our machines. There were some new specifications that were being developed at the same time we were developing our machines. We needed QualityLogic to help develop a suite of files that covered the specifications. We also needed help developing the tools to test the files with our machines.

What was the scope of their involvement?

We had two main goals. We handed QualityLogic the specifications that were in development for the new file format. First goal: QualityLogic needed to develop a set of files, a plot suite, that covered all of the specifications, including the possible variants, the positive and negative clauses, and everything that the specification says. The specifications were very complex with a core module and several extensions. In the end, there were 300 different plots that covered all the specifications. Our proposal was to make sure that our machines were compliant with the specifications. If a machine is going to consume this kind of file format and be considered "compliant with the spec," it needs to accept certain kinds of files and reject other kinds of files. If all the positive cases pass and all the negatives cases fail, we could be sure that we were compliant with the specification.

The second goal was to develop a set of tools so that our team didn’t need to test our machines manually. Every week, there are 300 untested files. QualityLogic developed a set of tools that were automatically able to connect with our machines, send the files, and report the results. The evaluation needed to be formed according to both our quality standards and those of our development team. Every time our development team made a change to the code, they had to make sure that it was compliant with the specifications.

What is the team dynamic?

I had one person that I interfaced with, and I also spoke to a senior engineer. We discussed requirements and technical matters and also budget, scope, etc. I know they had a team of developers and testers behind them, but I only interacted with these two people.

How did you come to work with QualityLogic?

They had worked with my company 15 years ago for another project. We recalled that their work was very good, so we decided to contact them. It was an internal recommendation from a colleague.

How much have you invested with them?

Our initial budget was around $200,000. We expanded the budget a couple of times because we wanted to perfect some tools and add more capabilities with them, to a final of $250,000.

What is the status of this engagement?

We started working with QualityLogic in the spring of 2016; the first project is finished and the second project is ongoing. The first set of standards was completed in seven or eight months.

What evidence can you share that demonstrates the impact of the engagement?

The most valuable metric is that we ended up with 300 valid test cases developed by QualityLogic. We also encountered 20-25 issues in our code that we found thanks to this plot suite.

We didn’t intend to qualify the specifications, but QualityLogic found several issues in the specifications themselves. They were in contact with the authors so they could correct them. There was a lot of success in the requirements. That was very valuable because if we had built the code based on wrong specifications, it would have cost us a lot of effort to re-build later on. That was not expected, but QualityLogic reviewed all the specification and asked questions and proposed improvements which was very helpful.

How did QualityLogic perform from a project management standpoint?

We had a meeting every week with a call. In every call, they told us the progress in the development and the new plots. We discussed requirements and priorities. If they needed to choose between different options for a tool, for example, we discussed the best approach. They would inform us on the budget and the cost of doing a certain functionality. They were very transparent. We didn’t have any problem with priorities or time estimations. If they said they were going to do something in a certain time, they made sure they did. We used GoToMeeting for our communication.

What did you find most impressive about them?

Their commitment is impressive. They were very involved in the project. They cared about the success. They understood our problem and wanted to know how we were going to use the product in order to adequate their job. They went above and beyond our expectations. They were good at avoiding failures and giving suggestions. In the beginning, I was the only interface with my company, but in the end, we saw that it was better if they spoke directly to our engineers. Now there’s very frequent communication between their team and ours. Their work is going well, so there will be other opportunities working with us.

Are there any areas they could improve?

No. If I had any suggestions of improvement for them, I told them and they fixed it right away.

Introduce your business and what you do there.

Stand Up To Cancer is a division of the Entertainment Industry Foundation, a 501(c)(3) charitable organization that raises funds to support collaborative cancer research.

What challenge were you trying to address with QualityLogic?

We had a very large-scale event that usually sees a very high level of transactional traffic. We needed QualityLogic to help make sure that all of our forms were functioning correctly and that there were no surprise issues that would be encountered by any of our users.

What was the scope of their involvement?

We engaged QualityLogic to run transactional testing as well as cross browser and device testing. We began by outlining what we were concerned about, and then QualityLogic looked over the plans, suggested what we should test, and then ran a full suite of functional and cross browser tests for us to make sure pages displayed correctly and that there were no bugs or submission issues.

We've worked with QualityLogic for three years and the team that we work with has always stayed consistent. It's made it very easy to engage with them. They're pretty familiar with our forms so it's easy to go back to them time and time again. We worked heavily with Don Walker, Paul Morris, Don Burke, and several other people on the team, and they've always been really great.

How did you come to work with QualityLogic?

Someone we had worked with knew of them and said that they might be worth considering. When we first engaged with them, we looked at several different testing companies, and we just liked the style of QualityLogic. They were very technical, but easy to work with. It was our first engagement with a firm to handle this kind of work as a neutral third party.

What is the status of this engagement?

We started working with them August 2014 and have done several projects together since that time.

Could you share any evidence that would demonstrate the productivity, quality of work, or the impact of the engagement?

For each engagement, they’ve found at least 20 reasonably significant bugs that could be addressed. On a recent project, we ended up with zero user complaints of any issues on the forms.

How did QualityLogic perform from a project management standpoint?

They’re very proactive about communicating what they’re working on and their schedule estimation is always very accurate. They are very flexible in timelines and the testing, and they really understand digital work cycles. When we had delays or things that came up, the team was really professional and easy to work with. Our correspondence has been mostly over the phone and email, and then also working with tracking through shared documents.

What did you find most impressive about QualityLogic?

What’s always impressed me is the accuracy and thoroughness of their work and their flexibility. They are usually able to help us make sure that we can walk into any page launch feeling confident that there aren’t any issues that we don’t know about. They’re a good third party sanity check for us.

Are there any areas QualityLogic could improve? ​​​​​​

Not that I can think of.