QualityLogic

BACKGROUND

Introduce your business and what you do there.

I’m the technical and managing director of the OpenADR Alliance. We’re a membership-based industry standards group for the smart goods space which includes devices like smart thermostats and electric-vehicle charger stations.

OPPORTUNITY / CHALLENGE

What challenge were you trying to address with QualityLogic?

One of our core organizational functions is to organize tests and certify products according to our standards. Almost nine years ago, we made the decision to develop and own a specific certification test tool that we would use to certify products. We needed a partner to work with us to develop it.

SOLUTION

What was the scope of their involvement?

We’ve been working with QualityLogic for almost nine years now. They initially built the test tool for us including all of the testing, the test plan, and some test cases. We now co-own the test tool with them and QualityLogic serves as an endorsed partner that can test products for our members based on our standards and certify them. They’ve delivered that tool to a number of our member companies. They provide consulting services and additional development support to our members upon request. We also maintain a working relationship to update the software every couple of years and do other, smaller projects as needed.

What is the team composition?

We work directly with a lead engineer who serves as a project manager and with the general manager. There are other programmers who work behind the scenes.

How did you come to work with QualityLogic?

We put out an RFP to a number of test equipment companies as well as test houses. We didn’t want to just hire a programmer to build the test tool; we wanted to establish a partnership. We understood that we would need support and modifications over the years and that a long-term arrangement would be more effective. We eventually selected QualityLogic based on a combination of their experience, their pricing, and the partnership arrangement they offered.

How much have you invested with them?

We’ve spent about $400,000.

What is the status of this engagement?

We started working together in July or August 2011 and it’s ongoing.

RESULTS & FEEDBACK

What evidence can you share that demonstrates the impact of the engagement?

We haven’t specifically tracked performance but the test tool works very well. I get very few questions about it, if at all, and we’ve had a positive response.

How did QualityLogic perform from a project management standpoint?

They’ve performed well. Working directly with one of their lead developers has worked well for us. He’s always been on top of things. They’ve worked well according to timelines and plans.

What did you find most impressive about them?

I would say two things. For one, they’re one of the few companies I’ve found with the experience in test systems and testing programs of our sort. QualityLogic seemed to be the one company that had the most relevant testing experience. They also wanted to dive into the smart goods space, which was great for us.

Also, they have the right people on their team. Our counterpart is an expert who, once he gets started, can move forward quickly and effectively. He’s incredibly knowledgeable and easy to communicate with.

Are there any areas they could improve?

I can’t think of anything in particular. We’ve been working together for such a long time and everything moves the way we’d like it to. They could potentially be a bit better at tracking their hours. It would be nice to have advance warning if we’re going to exceed our estimates. But I know it’s sometimes hard to quantify work when you’re building something from scratch. The standards were changing as they built the test tool.

Do you have any advice for potential customers?

The challenge is always to scope a project well, which can be tricky given so many unknowns. Make sure your project is fully outlined and most aspects are clear. That way there won’t be any surprises.

BACKGROUND

Introduce your business and what you do there.

I’m the director of software systems quality for a transportation company.

OPPORTUNITY / CHALLENGE

What challenge were you trying to address with QualityLogic?

We wanted a nearshore vendor as an affordable alternative for testing resources.

SOLUTION

What was the scope of their involvement?

QualityLogic tested our web and mobile applications, specifically the latter’s flow. Tangentially, the team also looked at our automation infrastructure.

What is the team composition?

We worked with a project manager and an account executive.

How did you come to work with QualityLogic?

I had worked with QualityLogic at another company and their mobile testing expertise stood out. We selected the team for this project because they clearly understood our business domain.

How much have you invested with them?

We spent between $50,000–$199,999.

What is the status of this engagement?

We started working with them in March 2018 and the partnership ended in January 2019.

RESULTS & FEEDBACK

What evidence can you share that demonstrates the impact of the engagement?

We tracked results via daily touch points to ensure proper oversight and clarification. Our side actively monitored the testing and execution.

How did QualityLogic perform from a project management standpoint?

QualityLogic integrated seamlessly with our business and provided high-quality services. Their attention to detail was notable.

What did you find most impressive about them?

The high-energy team was engaged throughout the partnership. They asked probing questions and proactively communicated any obstacles or challenges.

Are there any areas they could improve?

The partnership ended before we wanted, which limited QualityLogic’s involvement in the automation initiative. We wanted them to take a more active role with that.

Do you have any advice for potential customers?

Constant engagement with QualityLogic keeps the partnership on track. Clients that treat them more like an integrated team than an outsourcing vendor will get better results.

BACKGROUND

Introduce your business and what you do there.

I’m the CTO of Direct Care Innovations. We’re a healthcare technology company offering SaaS-based software to service providers in the Medicaid and Medicare markets.

OPPORTUNITY / CHALLENGE

What challenge were you trying to address with QualityLogic?

We were in the testing phase for multiple projects and didn’t have enough internal resources to perform quality tests. We wanted to outsource to a vendor who had the skills we lacked.

SOLUTION

What was the scope of their involvement?

QualityLogic test all of our new developments and bug solutions. They conduct tests on our products’ functionalities, performance, and security. Their team uses their own testing tools and show us analytics on the success of our products. The reports show if bugs have been solved and if a system functions correctly. When they find an issue, they screencast with our developers to make the locating and reproducing process simpler.

What is the team composition?

They provide us with five full-time testers and one part-time tester, who also functions as the project lead. Their team is on constant retainer, so there is always a member available to help us.

How did you come to work with QualityLogic?

I searched for vendors online after performing marketing research. One of our system analysts found them, and we preferred their smaller composition compared to the other behemoth-like companies. We also liked their flexibility toward an open-ended arrangement with an undefined scope of work.

How much have you invested with them?

We’ve spent around $150,000, so far.

What is the status of this engagement?

We started in March 2017, and we keep them on retainer for our testing needs.

RESULTS & FEEDBACK

What evidence can you share that demonstrates the impact of the engagement?

We’ve been able to increase the number of test cases we get through drastically. Along with quantity increasing, our KPIs show that the thoroughness of the tests is much better than we were doing previously. Our customers have given us positive feedback on the enhanced usability of our system’s overall performance and security.

How did QualityLogic perform from a project management standpoint?

They provide us with detailed documentation about the issues they find and give us quality suggestions for solutions. We receive daily updates from the project lead, which is one of the best parts of working with them. They deliver everything on time and within the budget we set.

What did you find most impressive about them?

Their entire team is professional, knowledgeable, and willing to go the extra mile—we consider them a partner. We rely on them heavily, and they’ve never let us down, so we find the project to be money well-spent.

Are there any areas they could improve?

Although there have been inevitable bugs that went undetected, it’s infrequent, and they give us a reasonable explanation. Our work is complex, and there will always be some issues that slip through—we can’t blame them for that.

BACKGROUND

Introduce your business and what you do there.

I'm the chief operating officer for High Fidelity, a San Francisco-based startup in the virtual reality space that's building a platform for free servers running virtual reality environments. We have about 50 employees, 24 of whom are engineers.

OPPORTUNITY / CHALLENGE

What challenge were you trying to address with QualityLogic?

We wanted to scale up our QA testing for the development work we're doing without expanding our in-house team too quickly. We wanted to keep a good balance of both engineering and QA staff.

SOLUTION

What was the scope of their involvement?

QualityLogic provided additional QA support for already developed software in C++. They execute test plans that we develop to verify that the code our engineers produce works as it's intended to. These tests require due diligence to ensure we're meeting specific outcomes of each feature within our platform. They report back to us, alerting us to any bugs or broken code.

What is the team composition?

Paul (Client Program Manager, QualityLogic) and Joe (Director of Program Management, QualityLogic) are our main point of contacts, but there are an additional six people from their team working on QA for us full-time.

How did you come to work with QualityLogic?

We had worked with another provider previously but were unhappy with their results. The rate at which they would go through tests and the consistency of their responses on the tests wasn't what we wanted, so we searched online and interviewed a few other vendors. We chose to work with QualityLogic in parallel with our previous vendor at first as a comparison, but ultimately decided to solely work with them based on their performance.

How much have you invested with them?

We work with them on a month to month agreement under a budget of $500,000 per year.

What is the status of this engagement?

We've been working with them since November 2017.

RESULTS & FEEDBACK

What evidence can you share that demonstrates the impact of the engagement?

QualityLogic improved our approach to QA testing, helping us rewrite hundreds of previously deficient tests as a result. The number of issues they've identified runs into the hundreds, which has helped us significantly improve the quality of every release in the process. They also process tests at an impressive rate.

How did QualityLogic perform from a project management standpoint?

They're very practical in how they deal with unexpected situations and have two people on their staff trained on our technology should they need to replace or add a couple QA staff to our projects. That approach is very sensible and shows that they're resilient to change.

When we've needed overtime from their current staff, they've been very responsive. Everything seems very well managed.

What did you find most impressive about them?

Their ability to deploy resources to help us build the test suite has been invaluable—they're a great partner in that way. Rather than just providing staff to run through each step of the test, they're able to add value by helping us build the system we're using for testing.

Are there any areas they could improve?

They're very sensible in the way they have a backup plan for replacing any staff on our project. They could, perhaps, be more transparent and alert us as those changes occur just so we're all on the same page and don't have any questions as to why a particular person is no longer on our project.

BACKGROUND

Introduce your business and what you do there.

We are a large associate development company, with locations all around the world. I work in software quality.

OPPORTUNITY / CHALLENGE

What challenge were you trying to address with QualityLogic?

We needed to support a new file format in our machines. There were some new specifications that were being developed at the same time we were developing our machines. We needed QualityLogic to help develop a suite of files that covered the specifications. We also needed help developing the tools to test the files with our machines.

SOLUTION

What was the scope of their involvement?

We had two main goals. We handed QualityLogic the specifications that were in development for the new file format. First goal: QualityLogic needed to develop a set of files, a plot suite, that covered all of the specifications, including the possible variants, the positive and negative clauses, and everything that the specification says. The specifications were very complex with a core module and several extensions. In the end, there were 300 different plots that covered all the specifications. Our proposal was to make sure that our machines were compliant with the specifications. If a machine is going to consume this kind of file format and be considered "compliant with the spec," it needs to accept certain kinds of files and reject other kinds of files. If all the positive cases pass and all the negatives cases fail, we could be sure that we were compliant with the specification.

The second goal was to develop a set of tools so that our team didn’t need to test our machines manually. Every week, there are 300 untested files. QualityLogic developed a set of tools that were automatically able to connect with our machines, send the files, and report the results. The evaluation needed to be formed according to both our quality standards and those of our development team. Every time our development team made a change to the code, they had to make sure that it was compliant with the specifications.

What is the team dynamic?

I had one person that I interfaced with, and I also spoke to a senior engineer. We discussed requirements and technical matters and also budget, scope, etc. I know they had a team of developers and testers behind them, but I only interacted with these two people.

How did you come to work with QualityLogic?

They had worked with my company 15 years ago for another project. We recalled that their work was very good, so we decided to contact them. It was an internal recommendation from a colleague.

How much have you invested with them?

Our initial budget was around $200,000. We expanded the budget a couple of times because we wanted to perfect some tools and add more capabilities with them, to a final of $250,000.

What is the status of this engagement?

We started working with QualityLogic in the spring of 2016; the first project is finished and the second project is ongoing. The first set of standards was completed in seven or eight months.

RESULTS & FEEDBACK

What evidence can you share that demonstrates the impact of the engagement?

The most valuable metric is that we ended up with 300 valid test cases developed by QualityLogic. We also encountered 20-25 issues in our code that we found thanks to this plot suite.

We didn’t intend to qualify the specifications, but QualityLogic found several issues in the specifications themselves. They were in contact with the authors so they could correct them. There was a lot of success in the requirements. That was very valuable because if we had built the code based on wrong specifications, it would have cost us a lot of effort to re-build later on. That was not expected, but QualityLogic reviewed all the specification and asked questions and proposed improvements which was very helpful.

How did QualityLogic perform from a project management standpoint?

We had a meeting every week with a call. In every call, they told us the progress in the development and the new plots. We discussed requirements and priorities. If they needed to choose between different options for a tool, for example, we discussed the best approach. They would inform us on the budget and the cost of doing a certain functionality. They were very transparent. We didn’t have any problem with priorities or time estimations. If they said they were going to do something in a certain time, they made sure they did. We used GoToMeeting for our communication.

What did you find most impressive about them?

Their commitment is impressive. They were very involved in the project. They cared about the success. They understood our problem and wanted to know how we were going to use the product in order to adequate their job. They went above and beyond our expectations. They were good at avoiding failures and giving suggestions. In the beginning, I was the only interface with my company, but in the end, we saw that it was better if they spoke directly to our engineers. Now there’s very frequent communication between their team and ours. Their work is going well, so there will be other opportunities working with us.

Are there any areas they could improve?

No. If I had any suggestions of improvement for them, I told them and they fixed it right away.

BACKGROUND

Introduce your business and what you do there.

Stand Up To Cancer is a division of the Entertainment Industry Foundation, a 501(c)(3) charitable organization that raises funds to support collaborative cancer research.

OPPORTUNITY / CHALLENGE

What challenge were you trying to address with QualityLogic?

We had a very large-scale event that usually sees a very high level of transactional traffic. We needed QualityLogic to help make sure that all of our forms were functioning correctly and that there were no surprise issues that would be encountered by any of our users.

SOLUTION

What was the scope of their involvement?

We engaged QualityLogic to run transactional testing as well as cross browser and device testing. We began by outlining what we were concerned about, and then QualityLogic looked over the plans, suggested what we should test, and then ran a full suite of functional and cross browser tests for us to make sure pages displayed correctly and that there were no bugs or submission issues.

We've worked with QualityLogic for three years and the team that we work with has always stayed consistent. It's made it very easy to engage with them. They're pretty familiar with our forms so it's easy to go back to them time and time again. We worked heavily with Don Walker, Paul Morris, Don Burke, and several other people on the team, and they've always been really great.

How did you come to work with QualityLogic?

Someone we had worked with knew of them and said that they might be worth considering. When we first engaged with them, we looked at several different testing companies, and we just liked the style of QualityLogic. They were very technical, but easy to work with. It was our first engagement with a firm to handle this kind of work as a neutral third party.

What is the status of this engagement?

We started working with them August 2014 and have done several projects together since that time.

RESULTS & FEEDBACK

Could you share any evidence that would demonstrate the productivity, quality of work, or the impact of the engagement?

For each engagement, they’ve found at least 20 reasonably significant bugs that could be addressed. On a recent project, we ended up with zero user complaints of any issues on the forms.

How did QualityLogic perform from a project management standpoint?

They’re very proactive about communicating what they’re working on and their schedule estimation is always very accurate. They are very flexible in timelines and the testing, and they really understand digital work cycles. When we had delays or things that came up, the team was really professional and easy to work with. Our correspondence has been mostly over the phone and email, and then also working with tracking through shared documents.

What did you find most impressive about QualityLogic?

What’s always impressed me is the accuracy and thoroughness of their work and their flexibility. They are usually able to help us make sure that we can walk into any page launch feeling confident that there aren’t any issues that we don’t know about. They’re a good third party sanity check for us.

Are there any areas QualityLogic could improve? ​​​​​​

Not that I can think of.

BACKGROUND

Introduce your business and what you do there.

I am the QA manager of an app development group working on behalf of a large media company. We focus on multiple platforms, including mobile over-the-top content on iOS, tvOS, Android mobile and TV, Amazon Fire TV, and a few others, including the web.

OPPORTUNITY / CHALLENGE

What challenge were you trying to address with QualityLogic?

We were growing fast, and needed testing help. QualityLogic was one of a few vendors who were available to us.

SOLUTION

What was the scope of their involvement?

We assigned QualityLogic some projects and discussed how we wanted the testing to be done. They were open to suggestions and new testing methods. They learned the process from us and took ownership of the testing for a wide array of our apps. This has included all-manual testing, not just black-box (having users interact with the apps), but also with a more complex approach. Based on a user’s location, they will receive different streams for live viewing. For instance, if someone wanted to see a particular New York station, the app would recognize that they are in that location, and give them the right feed. This is challenging to do with black-box testing because we can’t manipulate locations, and need to go into a gray-box area, looking at the APIs and manually manipulating responses so that we receive the desired results.

Another example: our app can handle five shows on a page; we wanted to see what would happen if there was a sixth show. Rather than us programming a test for each use case, QualityLogic has looked at our API and its responses, manually added shows, and assessed the reactions.

The composition of the team has changed recently. We have recently expanded to 50 people, but my interactions are with the project manager and the leads. We have also decreased the team size when there were fewer projects in a fiscal year.

We have worked with other testing vendors, but not at the scale of QualityLogic. They hold around 75-80% of all our projects.

How did you come to work with QualityLogic?

We used to have two different teams in-house: a web team and a mobile player team, which I was a part of. QualityLogic was contacted by someone on the web part. The teams have since merged, and I took over the QA department. I contacted QualityLogic and started giving them business minimally, at first. After they were onboarded around processes and functionalities, we started giving them more projects.

What is the status of this engagement?

We started working with QualityLogic 2 ½ or 3 years ago for my department. They may have started working for the company a little earlier.

RESULTS & FEEDBACK

Could you share any evidence that would demonstrate the productivity, quality of work, or the impact of the engagement?

Over the last year, we haven’t had any major issues in production after the testing performed by QualityLogic. Software will always have bugs, but we haven’t had any major ones that would make us stop everything and rerelease the code. QualityLogic has full coverage of test cases, and the work is done to a level that makes us confident in whatever they provide signoff on.

How did QualityLogic perform from a project management standpoint?

I deal directly with Joe Walker, their program management director, who is extremely flexible and able to present a plan for any challenges I present. I’ve also talked with one of their QA managers.

What did you find most impressive about QualityLogic?

The type of testing we do is not standard, not something which could be done by low-cost resources in India, for example. With QualityLogic, we have access to testers working at engineer levels. They’re willing to adapt, and they’re quick learners. We can teach something to one person, who will then relay it to the whole team.

Are there any areas QualityLogic could improve?

There were things to fix in the beginning, and we provided QualityLogic with ideas on how to change things around, but we haven’t had any production issues in the last six months to a year. When we first started, if there was anything missed on their side, they owned the mistake, which was a positive for us. We still provide feedback to them, but we haven’t had any recent issues.

BACKGROUND

Introduce your business and what you do there.

I’m the CTO and VP of engineering for a company that makes mobile and SaaS [software-as-a-service] solutions for emergency and clerical communication management. 

OPPORTUNITY / CHALLENGE

What challenge were you trying to address with QualityLogic?

We needed testing for our mobile apps and web app in a way that could be repeated without staffing our own team. We also needed enough physical devices to test everything properly. We’ve worked with other quality companies that focus on test automation. Despite all the automated tests as we build and execute, we still need manual, ad hoc, and exploratory testing. 

SOLUTION

What was the scope of their involvement?

The relationship was already in place when I joined the company. Now, they provide testing services for our applications. As we add more features, they incorporate new tests and have built up a library of those to determine functionality. If new issues are found, they integrate with our defect-tracking system directly. They’re able to enter what type of issue they find, how to reproduce them, and which accounts and users they’re acting on behalf of when they found it. We have other communication channels between them, the development teams, and the support teams in order to refine and triage items and determine what actions we should be taking.

We had a single main point of contact. At the beginning of the year, he started joining us twice a week in our daily standup meetings. He comes to the office and sits in our scrum. That’s working really well. A few other people are involved too.

How did you come to work with QualityLogic?

They’re a local company, and I think a manager of ours at the time ran into them somewhere. I’m not exactly sure how they got connected because that predated me.

How much have you invested with QualityLogic?

We have spent between $150,000 and $250,000 so far.

What is the status of this engagement?

We started working with them in June 2014 and the work is ongoing. 

RESULTS & FEEDBACK

Could you share any evidence that would demonstrate the productivity, quality of work, or impact of the engagement?

We have a great relationship, and they do excellent work for us. We have urgent timing on some of the releases that we make to enterprise customers. These tasks have time-sensitive events where we’re releasing things for use around the world. On some occasions, QualityLogic has been asked to help us test things really quickly or over the weekend. They’ve always been responsive and worked with us on those urgent requests.

How did QualityLogic perform from a project management standpoint?

We use JIRA for tracking all of the R&D work. They interface directly with JIRA and our Atlassian accounts. We use HipChat for communication about testing and qualification. We also use email and phone to communicate, and our project manager attends our meetings twice a week.

What did you find most impressive about QualityLogic?

They’ve developed a keen understanding of our applications. They know how they work as well as we do, maybe even better than some people on our team because of their broad usage. They take ownership of what they do and are quite proactive.

Are there any areas QualityLogic could improve?

I can’t think of anything. They offer good service. They don’t get into test automation, but that’s not a downside really.  

BACKGROUND

Please describe your company and position.

I am the Director of QA of Tango Analytics

Describe what your company does in a single sentence.

Software provider.

OPPORTUNITY / CHALLENGE

What specific goals or objectives did you hire QualityLogic to accomplish?

• accessibility of our software

SOLUTION

How did you find QualityLogic?

Online Search

Why did you select QualityLogic over others?

• High ratings
• Pricing fit our budget
• Great culture fit

How many teammates from QualityLogic were assigned to this project?

6-10 Employees

Describe the scope of work in detail. Please include a summary of key deliverables.

508 tetsing of multiple products and feedback in resolving.

RESULTS & FEEDBACK

What were the measurable outcomes from the project that demonstrate progress or success?

The pace of 508 tetsing and the results and communictaion.

Describe their project management. Did they deliver items on time? How did they respond to your needs?

They are awesome at communication and their presentation of the results. 

What was your primary form of communication with QualityLogic?

• Virtual Meeting
• Email or Messaging App

What did you find most impressive or unique about this company?

The crew is wonderful they are very knowledgeable and great at their job.

Are there any areas for improvement or something QualityLogic could have done differently?

Nothing they are amazing.

BACKGROUND

Please describe your company and position.

I am the QA Manager of a real estate company

Describe what your company does in a single sentence.

Rent and build homes.

OPPORTUNITY / CHALLENGE

What specific goals or objectives did you hire QualityLogic to accomplish?

• Automate varuius workstreams

SOLUTION

How did you find QualityLogic?

Online Search

Why did you select QualityLogic over others?

Onshore testers

What was the size of QualityLogic’s team?

2-5 Employees

Describe the scope of work in detail. Please include a summary of key deliverables.

We had several workstreams that included the 4services and 4vendors. They delivered automated web testing pipelines written in cypress.io and running in Azure Dev ops Pipelines.

RESULTS & FEEDBACK

What were the measurable outcomes from the project that demonstrate progress or success?

• Our velocity was increased massively due to the web automation for regression and smoke tests that are in place. hundreds of manual testing hours saved.

Describe their project management. Did they deliver items on time? How did they respond to your needs?

very well met. great metrics too.

What was your primary form of communication with QualityLogic?

• In-Person Meeting
• Virtual Meeting
• Email or Messaging App

What did you find most impressive or unique about this company?

The onshore team and their ability to expand and contract on demand. Hired great people too!

Are there any areas for improvement or something QualityLogic could have done differently?

A Better understanding of what the engineering managers role is. Initially I was not clear why the person was on the engagement.