QA Consultants

Introduce your business and what you do there.

I’m the vice president at Allcal. We provide mobile and web-based software services around specific events.

What challenge were you trying to address with QA Consultants?

We were building a new feature as part of a project for one of our clients and wanted it to be vetted before deploying it. We’re a small team and didn’t have the bandwidth required to take on the QA process internally.

What was the scope of their involvement?

We provided QA Consultants with a URL link to the web-based software in question and they built test cases independently, without the need for extensive documentation.

What is the team composition?

Brian (Managing Director, QA Consultants) has served as our primary contact, and he passed our project to a team leader who oversaw the QA process. There was also one specialist on the team that did most of the grunt work.

How did you come to work with QA Consultants?

Brian introduced himself to me at a networking conference in Dallas, TX, where we’re both based.

How much have you invested with them?

We’ve invested $10,000-$50,000 on their services altogether.

What is the status of this engagement?

We’ve used their services several times since we first started working together in July 2016. We have an internal QA process, but we reach out to them on larger-scale projects that require more extensive testing.

What evidence can you share that demonstrates the impact of the engagement?

I don’t have the exact numbers, but they caught a very impressive number of bugs in a very short amount of time. They work efficiently and ensure that the product is ready for launch.

How did QA Consultants perform from a project management standpoint?

They were very flexible and worked hard to make the process as simple as possible for us. There was a lot of communication throughout the process. They were organized and they recorded any issues that came up in the format that we wanted, which simplified the process of making the necessary changes on our end. They also worked efficiently to meet strict deadlines, which allowed us to deliver the final product to the client on time. Our developers were able to go through the bugs they found on Jira and fix them in real time. In many ways, they functioned as an extension of our team without the hassle of onboarding.

What did you find most impressive about them?

They’re very proactive in exploring all areas of the product and identifying issues that we may not have thought to look for. They owned the project, rather than simply doing exactly what we said and nothing more.

Are there any areas they could improve?

No, everything they provide is very useful.

Do you have any advice for potential customers?

Before we worked with QA Consultants, we thought our organization was too small for extensive QA. I would encourage smaller companies to consider activating a group like QA Consultants. They produce solid results, and you get a lot of value in their services without spending multiple weeks on a project.

Introduce your business and what you do there.

We are a nationwide retailer of a wide range of consumer products. I am a project manager for the company. I started my role as a QA lead, went up to QA manager, and ultimately became responsible for multiple client projects as the lead manager. I have been involved in quality assurance work since 2010, occupying various roles, from manual to automated testing for databases and backend systems all the way to frontend and UI work.

What challenge were you trying to address with QA Consultants?

QA Consultants is just one of four consultants we've hired for various large projects that involved manual or automated testing services.

What was the scope of their involvement?

One of our projects is a complete rewrite of an AS/400 application, which helps stores get their systems upgraded. Our other projects are frontend website upgrades, backend system upgrades, as well as one related to big data. QA Consultants has provided manual and automated testing for these projects.

QA Consultants can setup testing environments if needed, but in general, we manage our own environments. We own and have built multiple ones using our in-house infrastructure team.

Our AS/400 database migration project has a 60-million-dollar cost and is using 14 teams in Ukraine, with a total of 200 members for heavy Agile development. We have 70 onshore members consisting of a combination of business analysts, developers, and testers. The offshore solution is being validated onshore, given that it requires different levels of testing on actual devices being used in stores. Indirectly, we have a resource allocation of an additional 10–20 people, depending on what departments we are working with. There are easily 20–30 departments in touch with the supply-chain infrastructure database and reporting.

The other major project is a 30 million dollar one for refreshing of the front-end UI layer and the backend of a content management system, which is being upgraded to Adobe Experience Manager 6.1.

Estimates are normally done by in-house project managers, based on past project histories. We have timeframe requirements for resources, and we extend those based on how the project is behaving, what defects are identified, and so forth. In general, our habit has been to request a 3-month estimate and extend resource involvement for up to 3 years. The advantage of using contractors is that we can keep going for as long as we need, and stop the tap at any time.

How did you come to work with QA Consultants?

We mostly deal with teams in Ukraine, so communication can become challenging. We need to provide as many details as we can so that the remote team can reproduce the issue. I was not sure if QA Consultants would be able to provide this; I reached out to 4–5 different vendors, ultimately hiring 2 members from QA Consultants.

How much have you invested with QA Consultants?

The cost of QA Consultants' testing services is between $1,500,000 and $2,000,000 per year. QA Consultants has provided close to 20 resources for our projects. We use an average of 10 testers per year, at a cost of $17,000 per resource, per month.

What is the status of this engagement?

When we started working with QA Consultants in 2010, we were using close to 70 testers from their team. Due to a strategic change in our organization in 2014, we've brought in more full-time in-house resources, so QA Consultants' involvement has been reduced.

Could you share any evidence that would demonstrate the productivity, quality of work, or the impact of the engagement?

When looking for resources, I will prefer to have a blend of developers and testers. Resources need to work with offshore teams in trying to understand what's being built, get the build deployed locally, execute it, find anomalies, and provide feedback to the offshore team, actually making them accept that there is an issue to be fixed. More than a tester, I need a leader, an automation specialist, and a developer with very good technical skills, who can come in and challenge the nature of the issues we're experiencing.

Testing in itself is not rocket science, but this type of skill set in understanding the nitty-gritty of our work was a challenge. QA Consultants presented a number of resumes, out of which we picked the right people.

We are building a solution for the Canadian government. Not many companies are familiar with the tools and the levels of acceptance testing required for such a project. QA Consultants stood out by providing experienced resources.

Whenever we have a need for acceptance testing, we will contact QA Consultants. They are solely specialized in quality management and have appropriate resources on standby.

Big data and Hadoop is another important area where we need technical specialists who can actually understand the technology in and out, and work with our chief architect. QA Consultants has supplied one tester for this.

How did QA Consultants perform from a project management standpoint?

QA Consultants has been okay with pulling the plug on consultants with little prior notice. As such, we know that we won't be charged for more capacity than we need. QA Consultants can supply this level of convenience.

We use JIRA and Confluence quite a lot. We have evolved from SharePoint to JIRA, as QA Consultants' team was brought on board. They're very good at absorbing, accepting, and learning.

A constant requirement is an understanding of Selenium, JMeter, and LoadRunner. We put an emphasis on CI/CD [Continuous Integration/Continuous Deployment] and want the developers to understand tools like Jenkins, GitHub, and others. Given this, JIRA is on the bottom of my requirements. After giving someone access to it, it will only take them 2 days to learn it.

What did you find most impressive about QA Consultants?

It's important that I am given access to resources based on changing dynamics and evolving market requirements. I have worked with vendors who came to understand what we wanted as a client, and we have established this type of relationship with QA Consultants. When it comes to quality management resources, QA Consultants is well-known for providing strong candidates with a very fast turnaround.

Are there any areas QA Consultants could improve?

In 2012, we had a need for junior testers for pure frontend user-acceptance work, only wanting the resources to play around with the site for 3 months, without even giving them access to the quality center. At the end of the day, we would have a jam session and get their impressions of the site as users. We didn't want to pay as heavily as we would for a technical tester. It proved a bit challenging for QA Consultants to bring in lower-level testers, but they were able to procure some brand-new students from their IT courses. The one-week delay that QA Consultants had at the time was our only glitch, and it was not a big deal, given that we had other people for that testing.

I can't think of any significant improvement for QA Consultants. We're able to negotiate rates, and QA Consultants has been quite accommodating and reasonable.

Please describe your organization.

I work for a very large, multinational bank, and I work in the technology division. We hire and manage contractors and consultants from various technology contracting firms.

What is your position?

I'm a senior manager in the large testing organization.

What business challenge were you trying to address with QA Consultants?

It's been years now, so I don't really remember the original challenge. They probably helped us staff our automation and performance resources departments four years ago for the first time. We've since expanded their involvement both in terms of project size as well as complexity.

Please describe the scope of their involvement in detail.

I think the most complex project we had with them was a Web services and middleware project, where we were helping to define the process for Web services testing. At one point, we had up to 10 quality assurance specialists from their company assigned to our project. QA Consultants provided automation development, Web services, and performance testing resources. We had two different types of automation, as well as performance. We expanded to multiple areas of testing.

How did you come to work with QA Consultants?

We went through a rigorous RFP [request for proposal] process with a number of different providers. QA Consultants seemed to be the best fit for our given needs.

Could you provide a sense of the size of this initiative in financial terms?

I can't disclose that information.

What is the status of this engagement?

We started working together about four years ago, and the relationship is still ongoing.

Could you share any statistics or metrics from this engagement?

We did end up having probably five or six of their resources that stayed with my company for 20 months, which is a very long time to stay as contractors at our particular company. I think that alone shows that they had some talented people working for us. If any contractor stays that long with a company, and not only for a short couple months, then you know that they had some quality resources.

From a quality perspective, I think it fluctuated somewhat depending on the people they gave us. Maybe they could have improved their screening process, but problems have arisen even with people we've hired directly in the past. These things just happen. Sometimes, it's the person themselves and their attitude, not the lack of coding ability.

There were only a few situations where we had to request the replacement of a resource. Generally, they reallocated a different resource within a month. Some of these roles are not so easy to find. Overall, I would say we've been 75 percent to 80 percent successful with their resources. I'm pleased with that retention rate.

How did QA Consultants perform from a project management standpoint?

I'm a senior manager, so I didn't really manage them directly, but I managed the relationship. I have peers that have managers that would manage the QA Consultants resources directly. I rarely received reports of issues, though.

I've found that it's important to be very specific about what we need and what the overall goal of the program is that we're trying to roll out. Detailed preparation is the key to a beneficial relationship.

What distinguishes QA Consultants from other providers?

They perform software and application testing exclusively, so they really know testing. It's just in their blood. They are the only vendor we have on our approved vendor list that has such a niche service and skill set. All the other vendors staff all types of technology resources, but the QA Consultants guys just do testing, and that's why we chose them.

In hindsight, are there areas in which they could improve, or things you might do differently?

I think at times they bite off more than they can chew. Sometimes, growth is good, but knowing when to stop growing is also good. They need to learn how to say no at times, and understand that old business is just as important as new business.

Please describe your organization.

My company is Shoppers Drug Mart, and we are one of the largest Canadian retailers in the industry.

What is your position and responsibilities?

I am one of the executives. I’m responsible for development, testing, and business analysis.

What business challenge were you trying to address with QA Consultants?

The first collaboration with QA Consultants was not at Shoppers, it was at another company in Canada, where I used to be the assistant vice president. Our problem was that we didn't have a solid resource structure. We worked with multiple agencies and their resources would come and go and when people left, they took their knowledge of the project with them. We engaged with QA Consultants to support us in sourcing and then to support our automation testing.

Please describe the scope of their involvement in greater detail.

They demonstrated great aptitude in terms of innovative thinking, finding better models, and increasing flexibility. They worked on pretty much every single insurance domain project that we had.

They offer something called Test Factory. It’s a great model because we could communicate with them easily, even though their resources were working from their own facility. It gave us the flexibility to have resources working near our office, without having to pay for their desks, phones, or any other overhead costs that come with bringing in permanent resources.

In terms of testing, they did automation with the broker portal, which is like an insurance portal, as well as web services. We also put in place an HP ALM [Application Lifecycle Management], and they utilized performance center components to build our performance factor. I also utilize QA Consultants in a team augmentation capacity.

How did you come to work with QA Consultants?

We looked at multiple partners. We wanted a local presence because of the nature of our business and what we needed to test at the time. We also looked at the rates, capabilities, and quality of the resources that they were willing to provide.

Could you provide a sense of the size of this initiative in financial terms?

It probably cost around $16 million over three years. The number of resources they give us ranges between 60 and 90 people.

What is the status of this engagement?

We've been collaborating with them for eight years, and they still continue to service us. The last project before I changed roles, which was two years ago [2013], was the implementation of the Guidewire platform. Guidewire is a technology for insurance. We could not transfer our database to Guidewire without using an agile methodology, and this was a big organizational change for us. QA Consultants took the lead on some of the agile components and automation for that project.

Could you share any statistics or metrics from this engagement?

Without the automation they did for the Guidewire project, the delivery date would have been nine months later, and each month would have cost the company over $1 million. So the benefit was huge. They saved us millions in revenue.

Using Test Factory, we actually cut the cost down about half a million for a year. From my experience, you can be successful with any vendor if you know how to manage them, provide them with the right support, and have the ability to deal with change effectively.

How did QA Consultants perform from a project management standpoint?

I would say their flexibility and resource availability are excellent and extremely conducive to our overall business strategy. Testing is all they do, and quality is of utmost importance to them. When I’m working with vendors, they become part of my extended team, and I found I could bring issues to them for discussion, and they were very good at being able to lift my business up in certain areas.

What distinguishes QA Consultants from other providers?

The Test Factory component is definitely unique to them. They also have a specific model for automation. They go into the organization, they do the assessment of all the systems, and they start doing automation. They can build your test automation and you don't have to invest anything, so your budget doesn’t suffer. Utilization for execution is a very unique concept for them as well. Testing is fairly standard, but this tool really stands out to me.

In hindsight, are there areas in which they could improve, or things you might do differently?

It's difficult to say, because everyone has a different purpose. A lot of big companies are making their profits based on volume, but QA Consultants is more of a boutique company. They are more profitable from the quality of their work, I would say. I guess that isn’t an area for improvement, but if they could manage to maintain that level of quality while also increasing their talent pool, that would be fantastic.