Testinium

Introduce your business and what you do there.

We’re a total beverage company, and I’m responsible for the digital customer experience of the company.  

What challenge were you trying to address with Testinium?

The whole application and technology that our customers were using directly were managed by me. The challenge was to stay relevant to our customer base and optimizing our efforts to conduct mobile app testing and performance testing of our services.

What was the scope of their involvement?

Our phases were divided into two projects. In the first project, we conducted a performance test of our services including our online order feature, photo recognition feature, and customer loyalty feature. 

Then, the second project was about automating the web and mobile device testing process. The phases in this project were divided into three priorities, and each time, the team developed and ran test scenarios using Testinium's testing automation suite. All the testing scripts were developed by the testing team. The main purpose was for us to get automated reports and identify bugs. Once problems were identified, we immediately fixed them.  

What is the team composition?

We worked with Suayip (CTO), senior test engineers, and other test engineers. 

How did you come to work with Testinium?

I knew Testinium for a long time, and they were working with one of the biggest global systems for mobile communication (GSM) operators in Turkey and leading e-commerce companies. I also knew them from my industry, and they were beating other global automation suites with their technical competency. I also liked that they were local.

How much have you invested with them?

We spent $100,000. 

What is the status of this engagement?

For this project, we worked with them from January–December 2020. Today, we still work with them on an on-demand basis. 

What evidence can you share that demonstrates the impact of the engagement? 

Thanks to the test automation project conducted by Testinium, the main result was a 6% decrease in our technical bugs. Since we were able to detect the bugs in our different platforms, we received a 50% decrease in our customer complaints. 

We highly appreciated that their senior management was always hands-on with our project, and they took good care of us by staying closely connected. 

How did Testinium perform from a project management standpoint?

Their response time was really quick, and we had no project management issues. Above all, they always kept their time commitment, and their project estimation was on point. 

What did you find most impressive about them?

What stood out the most was Testinium’s testing automation suite — the tool was so detailed and easy to use. Our internal team didn’t need to undergo training to familiarize themselves with the features of their tool.

Are there any areas they could improve?

Their device pool always changed during the engagement so it might be better if they try to keep up with the most used mobile devices in the field. Further, they could also secure old devices that still use in automation scenarios since those devices are hard to find; they're not available in the market anymore.

Introduce your business and what you do there.

I work at a big e-commerce company as the head of software development. We outsource more than 200 employees, who all report to me.

What challenge were you trying to address with Testinium?

We work very hard to prepare our systems and software for Black Friday. Testinium provides tools for conducting tests during the year.

What was the scope of their involvement?

We perform application testing and QA with Testinium's tools. Loadium is the main tool we use for our load testing, and we use it heavily — almost twice a week. They also have other tools that we use, such as a QA dashboard to maintain our code quality. We’ve also hired a couple of people from Testinium to do manual testing. Finally, they’ve aided us in hiring employees internally.

What is the team composition?

Currently, there are four manual testers working with us. Before that, a couple of years ago, we hired many more people from them. There’s also a maintenance resource that assists with the QA dashboard tool.

How did you come to work with Testinium?

There aren’t many competitors here in Turkey — it’s a very small environment. Testinium was one of the biggest companies, and that’s why we chose them. We investigated another internationally known testing company, but Testinium provided superior pricing for the same functionality.

How much have you invested with them?

We spend about $10,000 a month on the human resources we hire, €10,000 (approximately $12,000 USD), for the QA dashboard, and Loadium is a pay-as-you-go service.

What is the status of this engagement?

Our relationship started around two years ago in April 2018. We hired some people one-and-a-half years ago and started using their tools around the same time. There isn’t a deadline on this collaboration, and we’re going to keep using their tools as long as we are writing code.

What evidence can you share that demonstrates the impact of the engagement?

We don’t really track any metrics from the tools, as we use them to obtain information and monitor our applications. The staff Testinium provides are already trained, and they follow up to see if there any issues with the assigned resources. We haven’t had any problems though; the guys are performing well.

How did Testinium perform from a project management standpoint?

Since we don’t really have a project with them, there’s not much I can say, but they are easy to reach when I need to talk to them.

In the beginning, we involved four staff members from Testinium. Two of them were working for our mobile teams and another two for our website teams. Testinium has no problems switching out their people in case we don’t want to work with someone, but we haven’t had that issue.

What did you find most impressive about them?

They're helpful and quick to resolve any issues.

Are there any areas they could improve?

I don’t know.

Do you have any advice for potential customers?

Anyone working with them won’t have any issues. I advise other companies to work with them, as I’m really happy with their service.

Introduce your business and what you do there.

I’m the technology department unit manager for a bank in Turkey. I’m responsible for the DevOps transformation and software development lifecycle in our company.

What challenge were you trying to address with Testinium?

We needed help with quality and testing processes. Most tests were done manually, and it became the bottleneck in our software development lifecycle. Even though the software was rapidly developed, the testing process took too much time. We needed testing automation to increase the speed of our process.

What was the scope of their involvement?

We worked with Testinium to address this ineffectiveness, and increase the automation level and the quality of our testing process. There are around 150 agile teams developing solutions for our customers, and Testinium assigned a QA person in each team. 

QA engineers are responsible for monitoring quality metrics and writing test automation. QA mentors help and guide engineers when they hit a challenge. Meanwhile, the heads of QA are developing the infrastructure and software needs in these processes. Tools and processes are necessary when writing test automation, and QA heads help establish or write new functions in those tools. 

We start with the teams to assess their quality and test automation capability. After this assessment, we assign them a level from one to five. After six months, we aim to get the level 1 teams to level 3. We look at the number of automation, unit tests, test scenarios, coverage, general quality, and response time of the code. 

The teams are assigned to a phase, and we provide support. Every two weeks, we announce the points that they get and the position in the league. We’re trying to make a gamification process between the teams where they compete with each other.

We are currently in phase three and plan to do phases four and five soon.

What is the team composition?

There was one coordinator, 12 QA engineers, four mentors, and two heads of QA working with us during our pilot phase, which lasted six months. After that, we increased the number to 23 engineers, six mentors, and two heads of QA. Now we’re in phase three, and again we increased the number of resources to 34 engineers, 12 mentors, and two heads of QA.

How did you come to work with Testinium?

Our general manager had worked with Testinium in his previous two jobs.

How much have you invested with them?

We have spent $1.5 million on the project.

What is the status of this engagement?

We started working together in February 2020, and it’s ongoing.

What evidence can you share that demonstrates the impact of the engagement?

There are many KPIs, and at each assessment level, we increased our expectations. We see 100% and sometimes 200% improvements in these metrics in the teams.

How did Testinium perform from a project management standpoint?

We follow a communication plan, and we have biweekly calls with Testinium. Every 2-3 months, we make a presentation to the upper management. We have a weekly status meeting with the teams where we review the week’s roadmap. Our agile standup meetings last about an hour, and we exchange feedback. Testinium works in a Kanban style. Their communication skills are good, and they’re always reachable through our Whatsapp group.

What did you find most impressive about them?

The gamification they facilitated is fantastic, and we got the performance we needed. When teams are at the bottom of the league, their general managers or VPs will pressure them, and we see those teams get higher positions. It’s really competitive — teams can go from the top of the league to the bottom in one month.  

Are there any areas they could improve?

Sometimes, they want to sell products, but we don’t want to buy them because we already have alternatives in our inventory. They don’t insist that much, but it’s always on their agenda. I think that’s normal, but we don’t want to buy anything extra.

Do you have any advice for potential customers?

Most of the banks in Turkey have the same problem right now with testing, and they contacted me about this project to ask for my advice. I tell them that we are happy with our experience with Testinium, and I recommend working with them. Your team culture should also change because you shouldn’t be at the same point where you started when outsourced resources leave. You have to educate your team by looking at Testinium’s resources and what they do.

Please describe your company and your position there.

n11.com - CTO

For what projects/services did your company hire Testinium, and what were your goals?

To support our test processes

How did you select Testinium and what were the deciding factors?

Their know-how, expertise and references in this industry

Describe the scope of work in detail, including the project steps, key deliverables, and technologies used.

• Providing resources for manual test
• Helping us the shape our performance test environment
• Measuring quality of our development processes
• Consulting in improving the CI/CD processes

How many people from the vendor's team worked with you, and what were their positions?

>15

Can you share any measurable outcomes of the project or general feedback about the deliverables?

The performance tests conducted with Testium helped us a lot to detect some bottlenecks in our platform. With a set of improvements, we've been able the traffic increase during the Black Friday period.

Describe their project management style, including communication tools and timelines.

They were quite disciplined and aligned with the project timeline.

What did you find most impressive or unique about this company?

They mainly address one specific area, the testing, in whole development cycle create deep expertise.

Are there any areas for improvement?

?

Please describe your company and your position there.

Domino's Pizza - CTO

For what projects/services did your company hire Testinium?

For the online ordering platform both for web and mobile app develepmenn CI/CD

What were your goals for this project?

To create quality and efficiency to get rid of the unestimated bugs and crashes

How did you select Testinium?

I used to work with Testinium in my previous positions which are Insider company and Hurriyet newspaper

Describe the project in detail.

Project goal is creating an e-commerce platform with a large scale, fault tolerant and max availability

What was the team composition?

Team is about 25 people including, fe, be developers, app developers, QAs, cloud and DevOps engineers

Can you share any outcomes from the project that demonstrate progress or success?

Downtime and crash rates have %79 improved.

How effective was the workflow between your team and theirs?

Testinium was embeded to CI/CD pipeline for flawless SDLC

What did you find most impressive about this company?

The words given by company are always trustworthy and keep safe both mine and the strategy of the company

Are there any areas for improvement?

We continue to work on the project in harmony and create more success stories.