QASource

BACKGROUND

Please describe your company and position.

I am the Senior Software Engineering Manager of Elasticsearch

Describe what your company does in a single sentence.

real-time search, analytics, and logging

OPPORTUNITY / CHALLENGE

What specific goals or objectives did you hire QASource to accomplish?

• Manual Testing
• e2e automation development

SOLUTION

How did you find QASource?

Online Search

Why did you select QASource over others?

• High ratings
• Great culture fit
• Good value for cost
• Company values aligned

What was the size of QASource’s team?

15

Describe the scope of work in detail. Please include a summary of key deliverables.

Testing: Design, develop and execute smoke, feature, regression tests for the Security Solutions product of Elastic. This includes building deploying systems on many different platforms, writing Test Plans, coordinating feature testing with developers, exploratory testing, running full regression tests for each release and managing test cases in our test case management system. e2e

Testing: Coordinate with developers to build out automated scripts in Go to support integration tests between our Agent and Fleet server

RESULTS & FEEDBACK

What were the measurable outcomes from the project that demonstrate progress or success?

• Reduced bugs in production
• Increased test coverage
• Faster development times

Describe their project management. Did they deliver items on time? How did they respond to your needs?

Projects were managed within GitHub tickets or through spreadsheets. QA Source delivered on time 99.99% of the time. Missed deliverables were due to blockers on our end that we needed to resolve. Any change requests or additional work was received gracefully and confirmed with an email.

What was your primary form of communication with QASource?

• Virtual Meeting
• Email or Messaging App

What did you find most impressive or unique about this company?

Team members stayed on the projects for a long time (years!) and we developed relationships with them. Each team member participated in our team slack channels, team meetings and we all looked forward to our quarterly reviews when we would review their progress and get to see the impact they were having.

Are there any areas for improvement or something QASource could have done differently?

Due to the COVID-19 pandemic, we were unable to do more in-person visits. I hope this changes soon as I think it would be a great way to further our working bonds.

BACKGROUND

Please describe your company and your position there.

I am a Quality Assurance Manager with Southern New Hampshire University.

OPPORTUNITY / CHALLENGE

For what projects/services did your company hire QASource?

Initially we hired QASource to help with our data transformation from our current student information system, to our new product. The project has been pushed out several times over the years, and the work is still ongoing.

Since we've had such great success with QASource, we added staff to help with our automation efforts as well. Over the past few years, there has been a need to ramp up and ramp down...and QASource has been totally flexible in working with our needs

SOLUTION

How did you select this vendor and what were the deciding factors?

We reviewed and contacted several vendors for the ETL work in 2019. QASource offered onshore, nearshore and offshore resource options. That was important to the project, even though we have not yet taken advantage of the nearshore option. We like the idea of having that as an option

Describe the project in detail and walk through the stages of the project.

There are way too many stages to get into here. But the ETL piece entails exporting the data, written in an old query language, and creating a .txt file.

The contents of the .txt needs to be compared against business rules/crosswalks and other validations, in order to "fit' into the new system. After it is "transformed" it is loaded into an Oracle database, where it is then compared to the actual data for accuracy.

With 100's of millions of rows, this is an enormous, time consuming effort. Each pass helps us clean the data, detect crosswalk errors, and detect load issues.

How many resources from the vendor's team worked with you, and what were their positions?

Initially it was 5...in the ETL space...we then added 8 automation engineers...so we maxed out at 13. Currently, we have 4 working int he ETL space and 4 in automation. I hope to add 3-5 more in the upcoming months.

RESULTS & FEEDBACK

Can you share any outcomes from the project that demonstrate progress or success?

The results of our ETL effort through the first 5 rounds have been excellent. We went from a 90% success rate with our date transformation to over 99%. The next round will be a huge effort since we haven't performed this testing in over a year now, and expect the data to be back in the low 90% range...

How effective was the workflow between your team and theirs?

The team works very well with our team. They all have established an excellent working relationship and are looked at as valued members of the team.

What did you find most impressive or unique about this company?

The ability to adapt. This project has been very trying on many fronts. Sometimes the ask or the need is not very clear. They are always willing to help us with our resource needs, even when we need to ramp down for a period of time.

Most recently, we needed to ramp down last summer, but worked with us to bring staff back on board to get to current level.

Are there any areas for improvement or something they could have done differently?

I don't think so. They have been able to accommodate everything we've asked for to date, and I don't anticipate that changing anytime soon.

BACKGROUND

Please describe your company and your position there.

I'm an executive of a company that manufactures CDs/DVDs for independent musicians and filmakers and prints, publishes and distributes books and ebooks for independent authors.

OPPORTUNITY / CHALLENGE

For what projects/services did your company hire QASource?

QASource has been our website functionality testing outsource partner for many years.

SOLUTION

How did you select this vendor and what were the deciding factors?

We looked at multiple testing vendors. QA Source had the best combination of services and pricing.

Describe the project in detail and walk through the stages of the project.

Their team has learned our website functionality and created automation tests for most of our processes. As we roll out new applications/functionality, they do a combination of automated and manual testing to find bugs and make sure we're delivering a quality product to our customers.

How many resources from the vendor's team worked with you, and what were their positions?

We work with a team of 4 engineers.

RESULTS & FEEDBACK

Can you share any outcomes from the project that demonstrate progress or success?

Their testing has been very effective, allowing us to find and fix bugs that we would not be able to find on our own.

How effective was the workflow between your team and theirs?

They do their work during our evenings, so we're able to come in in the morning and have the test results to know what bugs we need to fix. They communicate very well and the flow between our teams works well.

What did you find most impressive or unique about this company?

They've been consistently strong over many years, which is why we've continued working with them. They're very reliable and charge a fair price for their services.

Are there any areas for improvement or something they could have done differently?

i just hope they can hold the line on price increases given the current economic climate.

BACKGROUND

Introduce your business and what you do there.

I’m the CTO of MikMak. We offer marketing technology to the world’s biggest brands and allow them to show their shoppers where their products are available, depending on where the users are. Shoppers experience ads on social media, which drive them to our landing pages, where they can find where to buy those products based on their location — we call it Where-To-Buy technology.

In my position, I lead the entire engineering side of our company. I’ve been working here for about 6 years, building our engineering team from the ground up.

OPPORTUNITY / CHALLENGE

What challenge were you trying to address with QASource?

Our platform supports a lot of brands, and we have many different iterations of our products in the market. We realized that our platform had reached a scale and level of complexity that made it hard for us to manage and run its tests on our own. Thus, we needed extra resources to help us with QAing all the different features and functionalities we were launching each week to ensure we weren’t breaking anything for the brands using our platform.

SOLUTION

What was the scope of their involvement?

Initially, QASource helped us QA our products and feature launches. However, we later found out that they were able to help us with things beyond QA. Thus, they’ve helped us develop a lot of new features for our platform since then. As for the technology stack, we use a lot of JavaScript internally, so QASource has used a Java-based tool for their testing, as well as JavaScript for their development work.

Currently, we’re building a lot of features. Our main product is the tool that users around the world use to find where a specific product is available. Still, we do a lot of other things with QASource’s help as we expand to new clients and grow our presence.

Continuously, we need more features, whether it is because we’re going into a new country, language, or currency. We also develop new products with new features, so we launch new features every week. Our internal team focuses on building those new features, but we often find that some of those features can be developed for us by QASource. Basically, they’re completely embedded into our team and contribute to our development efforts as if they were full-time employees.

What is the team composition?

We initially started working with two people from QASource for QA purposes. Since then, we’ve expanded our team to nine people, who now help us develop and QA the technology that we’re building. Their team includes two QA engineers and seven software engineers.

How did you come to work with QASource?

QASource came highly recommended to us by our CPO. During our vetting process, we also considered other companies we had previously worked with for short engagements. However, we knew that this would be a long engagement, so we were looking for a fitting candidate.

When we first engaged with QASource, we found out that they were extremely flexible and willing to work within our current processes, demonstrating that they would grow and change with us as our processes changed. They were also able to adapt to our needs quickly without us having to change much, so we onboarded them. We hadn’t found those characteristics in other partners; other candidates didn’t seem so flexible. 

How much have you invested with them?

We’ve spent between $750,000–$1 million.

What is the status of this engagement?

We started working together around Q4 2021, and our engagement is ongoing.

RESULTS & FEEDBACK

What evidence can you share that demonstrates the impact of the engagement? 

QASource has been a huge part of our success to date. When they joined, the metric we were the most concerned with was the number of defects that we were shipping in our code and whether that number could be reduced. Throughout the process, we introduced a new metric: the number of automated tests that we were able to create.

Thanks to QASource, we’ve definitely increased our automated tests every week. The tests we’re building are increasing, and we continue to cover more and more cases in an automated way. Ultimately, having more coverage in our tests allows us to decrease the number of defects our code has when released.

How did QASource perform from a project management standpoint?

Part of QASource’s team works within our time zone, so their responsiveness and communication are incredibly good. We also have a few other resources from QASource working in a different time zone, and they still do great work while we’re off duty or sleeping. We use Shortcut for project management, through which we also keep in communication.

We have a monthly check-in cadence, where we review what we’ve produced during the month. During these meetings, QASource asks for feedback, which is a very good practice they use to ensure we’re happy with their work.

What did you find most impressive about them?

Aside from their flexibility, the resources that QASource has is impressive. They have the quantity and quality of resources to establish long-term relationships with their clients. For example, if one of their team members can no longer work with us, they immediately bring in a new team member to continue working with us without any disruption.

If I were in a different organization and needed additional resources to augment my staff, I would certainly go back to QASource. They have everything we need from an external team, and we’ve been very pleased with their work.

Are there any areas they could improve?

I don’t have any constructive feedback for QASource; they’ve been great.

Do you have any advice for potential customers?

Have a clear scope of your project and give QASource clear instructions on what you want. They really thrive in an environment where the product team provides them with a clear set of outcomes that they’re looking to achieve so that they can develop the products correctly.

BACKGROUND

Please describe your company and your position there.

I am the QA Director for a Software company servicing the Utility space

OPPORTUNITY / CHALLENGE

For what projects/services did your company hire QASource?

We needed help to add automation tests to our existing suites while also accelerating the manual testing done to make timely releases.

SOLUTION

How did you select this vendor and what were the deciding factors?

We selected this vendor after interviewing a few, because of their automation expertise and being able to overlap with our workday. This allowed us to interact with the offshore team during normal business hours.

Describe the project in detail and walk through the stages of the project.

Their team was able to create and maintain automated tests for our software. They are also integral in learning the product and executing manual tests. UI Automation tests were created in Java using BDD. API Level testing has just begun also using Java.

How many resources from the vendor's team worked with you, and what were their positions?

We work with two automation engineers/

RESULTS & FEEDBACK

Can you share any outcomes from the project that demonstrate progress or success?

The automation tests have saved hours of regression testing. Their manual testing has also shaved days off our release cycle.

How effective was the workflow between your team and theirs?

We work on tight deadlines and the team participates in daily meeting to share their progress and questions with the rest of the team. In addition there is a posted daily and monthly progress report. They also share their questions with the development team through Teams when appropriata

What did you find most impressive or unique about this company?

They were able to work quickly to understand the product and produce useful tests and test cases.

Are there any areas for improvement or something they could have done differently?

Any improvement has been addressed in a timely manner after it's been shared with the team.

BACKGROUND

Introduce your business and what you do there.

I'm the head of mobile technology services at a health and wellness brand.

OPPORTUNITY / CHALLENGE

What challenge were you trying to address with QASource?

We wanted to take the load off of our product managers and designers, so we were looking for an affordable way to do QA testing of our mobile application.

SOLUTION

What was the scope of their involvement?

They review the features of the app before they go into production. They also review weekly content, do continuous testing of the app, and make sure there are no bugs.

What is the team composition?

We work with one QA resource and our account manager.

How did you come to work with QASource?

They were recommended to us by someone who had worked with them previously. We were trying to cut the costs, so we wanted to outsource an offshore team rather than hire someone full-time.

How much have you invested with them?

We've spent between $50,000–$60,000.

What is the status of this engagement?

We started working with them in November 2017 and the relationship is ongoing.

RESULTS & FEEDBACK

What evidence can you share that demonstrates the impact of the engagement?

They've helped us with everything from spelling errors and issues with image sizing to bugs and crashes. We're satisfied with their work.

How did QASource perform from a project management standpoint?

They're very organized and give us regular daily and weekly status reports. We mostly use Slack, Pivotal Tracker, and Excel documents to stay on track with the project. QASource has been willing to integrate with our current process and team structure.

What did you find most impressive about them?

They're very professional and willing to understand the product. Also, they're proactive, which is especially helpful when there is a lack of direction on our end. I can be sure they're always working on something to fill the hours we're paying them for.

Are there any areas they could improve?

We have limited resources, so the onboarding process took a little bit of time. But once they understood the product, they've been quite self-sufficient.

Do you have any advice for potential customers?

Get everyone up to speed on the product. That will make the process more efficient.

BACKGROUND

Introduce your business and what you do there.

I work for a cybersecurity firm, which provides an endpoint security application to military and commercial clients. I’m the QA manager, so I run both the QASource team and our local team.

OPPORTUNITY / CHALLENGE

What challenge were you trying to address with QASource?

We had a large quantity of software that needed testing and not enough internal staff to handle all the work.

SOLUTION

What was the scope of their involvement?

When I came on board, they only handled manual regression testing, but I was pleased with their work and tasked them to help our internal team with feature testing as well. We’d create a Jira ticket listing a feature we wanted them to develop and they’d write the test cases and perform exploratory testing for us. It’s worked out so well that the QASource team now communicates directly with our developers.

What is the team composition?

I have seven people on a team at their site and another team of six that are working locally with us.

How did you come to work with QASource?

They were already working with us when I came on board, but one of my predecessors researched the company and flew out to India to meet them and pick out the members of our initial team. He was really impressed by the quality and training of their engineers, as well as the maturity of their work.

How much have you invested with them?

It’s somewhere between $10,000–$50,000.

What is the status of this engagement?

We started working with them in April 2017 and the partnership is ongoing.

RESULTS & FEEDBACK

What evidence can you share that demonstrates the impact of the engagement?

They frequently turn around projects overnight, which means results and bug reports come in quickly. It’s really sped up our process. Their teams provide a range of technical expertise and everyone is excellent at manual testing. Some are also proficient at the backend work of building environments and performing API testing, which has been helpful to us.

How did QASource perform from a project management standpoint?

They do very detailed testing and communicate well. They send a daily status update via email listing everything they’ve accomplished and we manage our projects using Jira and Hipchat. They’re also part of our Agile team and they meet with our scrum team. In addition to that, we teleconference once a week so we can touch base with the team at their office in India. We also have a monthly conference call where we compare our performance with the past couple of months, analyze trends, and address any larger issues that come up. We’re very proactive about communicating with them so we can ensure that they’re successful in their work.

What did you find most impressive about them?

Their commitment to frequent communication has helped make this a successful partnership. Another noteworthy thing is that their team’s staff turnover has been extremely low. We’ve only had one person transition off the team and they always have a backup person who can take over to ensure that there are no service disruptions.

Do you have any advice for potential customers?

In order to maximize success, make sure you commit to frequent and regular communication. It’s too easy to lose pieces or make your team confused if you aren’t willing to stay on top of that.

BACKGROUND

Introduce your business and what you do there.

Our company provides legal, risk, and matter management software. I’m the senior VP of product development in charge of our 115-person global team.

OPPORTUNITY / CHALLENGE

What challenge were you trying to address with QASource?

Our challenge was common for medium-sized businesses where QA automation is usually an afterthought. We realized as our company grew that we hadn’t focused on this area, which is why we reached out to QASource.

SOLUTION

What was the scope of their involvement?

We deploy matter management systems for Fortune 500 corporations. They’re complex implementations, some of which span multiple years. QASource initially supplied an offshore testing team, but we’ve since expanded their scope beyond testing automation. They now provide product development and custom services implementation on top of testing. They’ve worked on at least 30 complex product implementations such as systems, applications, and products (SAP) rollouts.

What is the team composition?

They’ve provided developers, testers, and DevOps specialists. We’ve tried to keep our internal and outsourced teams small, no larger than three developers and two QAs. We have multiple teams of this kind working on different projects, totaling 45–50 resources. The model seems to work well. We handle project management internally.

They deploy an additional shadow resource at their own cost when putting together a team of five people. That person takes over when another team member leaves or needs to take time off.

They have dedicated managers for various functions: QA, development, and IT for different needs. We’ve also worked with a U.S.-based account/relationship manager. We have monthly meetings during which we go through checklists of items in need of improvement.

How did you come to work with QASource?

One of our heads of engineering who had previously collaborated with QASource introduced us to them.

How much have you invested with them?

The cost of their service over the past 10–12 years has been in the tens of millions of dollars.

What is the status of this engagement?

We started working with them 10–12 years ago and the relationship is ongoing.

RESULTS & FEEDBACK

What evidence can you share that demonstrates the impact of the engagement?

They’ve been an integral part of our deployment projects for very large global companies. They’ve added a tremendous amount of value to our business. We’ve had good and bad years, but we’ve always been able to reach out to them for scalability needs. Hiring resources locally is very difficult for us, but QASource has a large, available talent pool that we can quickly tap into. I can add resources with a week’s notice.

How did QASource perform from a project management standpoint?

We use Jira and HipChat for our daily communication and task tracking since we’re an Atlassian shop. We have local scrum masters with QASource’s teams. We have daily standups at the beginning of our working day which corresponds to the end of their working day.

What did you find most impressive about them?

A team usually fails because of a lack of good talent, but QASource has a good process in place for funneling the right resources. I’ve had deep discussions with them and they’ve explained how they source members from local schools in Chandigarh, India, and then provide ongoing training before graduation. They test the candidates based on that training and only pick the best performers. This is a huge differentiator that I haven’t seen in other offshore businesses.

We work with other offshore and nearshore vendors, but their management has become problematic. QASource has a strong management team in place that follows through on their commitments. We don’t need to constantly follow up with them.

Are there any areas they could improve?

They could provide on-site liaisons to act as leads for our offshore teams. They were willing to do this when I raised the issue a few years ago but at an additional cost.

Do you have any advice for potential customers?

You need to treat them as an extension of your team, including them in daily standups and allowing on-site visits with managers. Having in-person meetings will lead to a better understanding. Things took a turn for the better after I started sending my managers to their location and bringing their resources here.

BACKGROUND

Introduce your business and what you do there.

I work for an integration and automation company, dealing mainly with hospital infrastructure. I’m the senior VP of research and development.

OPPORTUNITY / CHALLENGE

What challenge were you trying to address with QASource?

We had a couple of internal QA staff members and were looking to add to that.

SOLUTION

What was the scope of their involvement?

Our iteration process is usually one to two months long. Their job is to take the nightly builds that get published. They start with a smoke and sanity test, then they go onto what we call work item testing that focuses on specific pieces of functionality. If that’s all complete, they do general exploratory testing.

They started with our enterprise messaging application. Now, they’re working on our web-based position data access application. We consider them part of our permanent staff and will always have work for them.

What is the team composition?

We started with one, then added two further resources. We’re going to cut back down to two at the end of this month. When we need to ramp up our testing efforts for a big initiative, they’re able to give us what we need. When their people have time off for vacation, they’ve been able to augment that with backup resources so that we don’t lose a lot of staff time when our full-time people aren’t available.

How did you come to work with QASource?

It was really difficult to find qualified testers in our area. We looked at outsourcing and found 5–6 different companies of which QASource was one. We based our decision on cost and accessibility. The references from their clients were really good too.

How much have you invested with them?

$150,000–$200,000.

What is the status of this engagement?

We started working together in July 2016 and the work is ongoing.

RESULTS & FEEDBACK

What evidence can you share that demonstrates the impact of the engagement?

We look at capacity versus the actual work. We know they helped and put the effort in, but it’s tough to say if we didn’t have them, that X number of bugs that have gone by us.

How did QASource perform from a project management standpoint?

It's been really good. Our QA manager sets up weekly meetings with them. We have a daily scrum with all of our research and development (R&D) staff every morning and a monthly meeting with our account manager and project manager. If we have questions, concerns, or need any attention, they’re very good at getting back to us in a timely fashion. We use IM and email.

What did you find most impressive about them?

We’re very pleased with our relationship with them. The project management is a very important piece. We’ve been very satisfied.

Are there any areas they could improve?

We looked at their offerings to try to automate test cases, but because we have our own automation tool, we’ve decided to go that route instead. We’re educating them now on that process and wanting them to actually use our tool to do the automation tests. They could improve on that.

Do you have any advice for potential customers?

Be clear from the start. We were just starting out our QA department and they helped us to put a structure in place. Make a good plan for the staff, how they need to structure their day, and integrate them with your team.