Pragmatic LLC.

BACKGROUND

Please describe your company and your position there.

I'm the tech lead in a 300+ person company. We provide web and app based streaming services.

OPPORTUNITY / CHALLENGE

For what projects/services did your company hire Pragmatic LLC.?

Providing quality assurance and automated testing for our services and apps.

SOLUTION

How did you select Pragmatic LLC.and what were the deciding factors?

We spoke to a number of vendors and the quality of knowledge of Pragmatic really shone through.

Describe the project in detail and walk through the stages of the project.

At a high level, Pragmatic were responsible for improving our confidence in our releases and in our test suites. They did this by stripping away dead wood tests, rewriting existing tests and filling in gaps in testing. They also liaised across the company to better understand where defects were being found and helped us to track them down. They introduced new ways of testing and helped to upskill the team.

How many resources from the Pragmatic LLC.  team worked with you, and what were their positions?

1 quality assurance engineer who covered all aspects.

RESULTS & FEEDBACK

Can you share any outcomes from the project that demonstrate progress or success?

Defects are now much lower than before. When tests fail, we get useful feedback that allows us to trace issues. Tests help us to identify problems with our architecture.

How effective was the workflow between your team and theirs?

Perfect. They behaved like part of the team. Our goals were their goals. They went the extra mile to provide quality results. They spoke up when they didn't need to. When they needed to learn new things, they did.

What did you find most impressive or unique about this company?

They really cared. They understand that they can maximise value to the client by walking with the client.

Are there any areas for improvement or something they could have done differently?

None.

BACKGROUND

Please describe your company and your position there.

I'm the CTO of a luxury goods e-commerce site

OPPORTUNITY / CHALLENGE

For what projects/services did your company hire Pragmatic LLC., and what were your goals?

We were building a new e-commerce store with the assistance of a development agency but we wanted to have third-party testing and validation of the core functionality to ensure the development team were "kept" honest and to ensure we could continue to test the site even if we changed development teams.

In addition, we wanted to build a suite of regression tests, manual and automatic, to ensure we could validate the site functionality as we added more functionality.

SOLUTION

How did you select Pragmatic LLC. and what were the deciding factors?

We were introduced to Pragmatic via an e-commerce consultant we were working with who is well known in the community. After a screening call with the agency we decided to engage them.

Describe the scope of work in detail, including the project steps, key deliverables, and technologies used.

We described the testing requirements we had in an initial kick-off meeting and shared our MVP feature list and product designs. Throughout the project the Pragmatic team would test weekly the core functionality and new deliverables quickly raising issues with the development team and escalating to the management team where critical or showstopper issues were encountered.

How many people from the vendor's team worked with you, and what were their positions?

2, the Founder/CTO and a mid-level tester.

RESULTS & FEEDBACK

Can you share any measurable outcomes of the project or general feedback about the deliverables?

It is still early days in our store but in terms of deliverables a number of key issues have been identified and resolved as a result of Pragmatic's work.

Describe their project management style, including communication tools and timelines.

We primarily engaged with the team over Slack but someone joined a weekly call with the development agency. To be honest this wasn't enough as they weren't close enough to the changing landscape and so later we have invited them to more regular stand-ups.

What did you find most impressive or unique about this company?

I am impressed how thorough they are and their ability to quickly triage and escalate the right issues that require immediate attention. In addition they always provide detailed test reports, steps to re-create, and videos to help debug. They are also just great, down to earth people.

Are there any areas for improvement?

I wish we could have more automated testing but that's more a factor of time than any reflection of Pragmatic.

BACKGROUND

Please describe your company and your position there.

My company develops hardware products and technology for digital drawing, note taking and digital ink. My position in the company is Director Software and E2E QA.

OPPORTUNITY / CHALLENGE

For what projects/services did your company hire Pragmatic LLC.?

Pragmatic were hired to perform load and performance testing for estore project

What were your goals for this project?

The goals of the project were to improve user experience, performance and stability

SOLUTION

How did you select this vendor?

We looked online for vendors with experience in performance and load testing. We had a meeting with vendors and got more information about projects they had executed, tools used, collaboration process and reporting

Describe the project in detail.

We had a meeting with Pragmatic and provided them with technical information about platform, environment and architecture. Areas that were not clear were clarified and they started with scripts preparation and execution. Report with issues found was provided. When issues were fixed we executed tests again

What was the team composition?

On our side test lead worked with lead from Pragmatic side.

RESULTS & FEEDBACK

Can you share any outcomes from the project that demonstrate progress or success?

We were able to improve the performance of the site and it became faster and stable.

How effective was the workflow between your team and theirs?

The work between both teams were very well and smooth in terms of communication and execution.

What did you find most impressive about this company?

They are fast and flexible. Their goal is to support the client not only with execution of respective project, but also to provide suggestions and consultancy how to additionally improve it.

Are there any areas for improvement?

No

BACKGROUND

Please describe your company and your position there.

I'm a CTO of IndigoVerge and we develop product and services for our clients.

OPPORTUNITY / CHALLENGE

For what projects/services did your company hire Pragmatic LLC.?

We hired Pragmatic to help us with establishing our own QA dedicated team.

What were your goals for this project?

The main goal was to setup a proper workflow, tools and environment for our QA team. For a period of 3 months we wanted guidance and supervision services to ensure the proper setup.

SOLUTION

How did you select this vendor?

We placed a job offer for a QA and then we met Pragmatic in the role of hiring agency that helped us to review candidates and hire the right one.

Describe the project in detail.

Device and application management platform for industrial IoT

What was the team composition?

We hied a junior QA and we setup a training program that has been provided by Pragmatic. The training was on-side and last month of the training on-line, upon request.

RESULTS & FEEDBACK

Can you share any outcomes from the project that demonstrate progress or success?

Within 3 months we managed to setup and implement manual testing procedures and we began to implement automation testing for one of our projects using several tools. Since then we constantly enhance the tests and we did several releases based on test results.

How effective was the workflow between your team and theirs?

Having in mind that they were at the office side by side with us ensures a smooth workflow and know-how transfer.

What did you find most impressive about this company?

I valued their fast response times and how easy it is to work with them. Also they shared their own experience and made a setup that was a combination of the project specifics and the best practices they apply in the QA field.

Are there any areas for improvement?

I cannot think of any at the moment.

BACKGROUND

Introduce your business and what you do there.

I was the test manager for a global e-commerce company.

OPPORTUNITY / CHALLENGE

What challenge were you trying to address with Pragmatic LLC.?

We needed additional testing resources. We already had a partner in Romania and needed a company to supplement that office’s output.

SOLUTION

What was the scope of their involvement?

Pragmatic LLC. did a two-week onboarding session at our office. Services included automation testing and technical testing for multiple solutions and services. The team leveraged automation for APIs.

What is the team composition?

I worked directly with Milen (CEO, Pragmatic LLC.) and Alexsandar (CTO, Pragmatic LLC.). I served as the project coordinator.

How did you come to work with Pragmatic LLC.?

I met their CEO through LinkedIn. The team was looking for clients that matched our company profile. They then submitted a proposal. We hired Pragmatic LLC. because they had superior training for automation testing and coding.

What is the status of this engagement?

We started working together in 2014 and the project ended in 2016.

RESULTS & FEEDBACK

What evidence can you share that demonstrates the impact of the engagement?

The automation framework that we designed in 2014 is still in operation. We continue to use a number of tests and API checks that run from the framework.

How did Pragmatic LLC. perform from a project management standpoint?

They were the most responsive team I’ve worked with. We went on site and their team visited our office for varying lengths of time. Whenever we needed resources with particular skill sets, Pragmatic LLC. would find the best people the best rates. Overall, the team delivered quality work and were first class.

What did you find most impressive about them?

They had integrity and delivered on their promises. The team would tell us in advance if they could meet a deadline. If they couldn’t, they would provide alternative solutions. What’s more, the team used accessible language to explain the project’s technical aspects.

Are there any areas they could improve?

No, I can’t think of anything.

Do you have any advice for potential customers?

Clients should understand the requirements before requesting them. Pragmatic LLC. needs the rules laid out to build frameworks. The team will provide honest responses to any queries.  

BACKGROUND

Introduce your business and what you do there.

I’m the VP of product and development at Whiteboard Technology. We provide CRM management and automation services to loan officers in the home buying sector.

OPPORTUNITY / CHALLENGE

What challenge were you trying to address with Pragmatic LLC?

We wanted to automate our internal testing processes to increase the quality of our builds and products. We brought in Pragmatic LLC to help us introduce testing environments that are separate from our production environments.

SOLUTION

What was the scope of their involvement?

The engagement is split into two phases—architecture creation, then strategy building. Pragmatic LLC helped us assemble a testing environment outside of our production processes. They architected the platform’s framework and conceptualized the process. Together, we decided on a collection of tools and approaches to use in the testing strategy. Now that we have the environment’s architecture, Pragmatic LLC is designing test cases for all of our current functionalities.

So far, they’ve worked with a variety of tools and technology. For the actual testing components, they used Selenium, Buddy and Docker Swarm. The team also worked with several dynamic APIs to automate the process. They used a few test management tools to store data on our procedures and test results.

What is the team composition?

For the first phase, we worked with two senior architects and a senior test engineer. Now that the build is complete, we’ll continue working with just the test engineer to design cases.

How did you come to work with Pragmatic LLC?

Initially, I was just looking to hire a single offshore resource for this project. I shared my needs with a friend, who connected me with the CEO of Pragmatic LLC. He described his experience in the industry, and we discussed the scope of work. He agreed to the terms, but also explained that he owned a company that provided all of the services I needed. I did the math and spoke with my CEO, and we decided that it made more sense to engage the full company.

What is the status of this engagement?

We started working together in September 2018, and it’s an ongoing project. We’ve signed a six-month agreement.

RESULTS & FEEDBACK

What evidence can you share that demonstrates the impact of the engagement?

We’ve noticed improvements in two significant areas. First, the quality of our solutions has improved. With the new testing processes, we can catch unwanted bugs before we start production. Second, in partnering with Pragmatic LLC, we’re able to internally allocate more time to core responsibilities like feature development and architectural challenges, without sacrificing any quality. We’ll likely extend the partnership beyond this project.

How did Pragmatic LLC perform from a project management standpoint?

Pragmatic LLC is excellent. They’re very knowledgeable in this field, and it’s clear that they know what they’re doing. The team is professional and helpful, willingly addressing all of my questions and concerns no matter how trivial. We keep an open communication channel across Slack, email, and video conference tools. Everyone involved is up to date on our status and aware of any challenges we’re currently facing. The team is also very flexible. I made some changes to the scope a month into the project, and they quickly adjusted their workflow.

What did you find most impressive about them?

I really appreciate their expertise and customer-centric approach. They put themselves in our shoes to better understand the importance of this project.

Are there any areas they could improve?

Honestly, nothing comes to mind right now.

Any advice for potential customers?

Trust their team. Give them the freedom to tap into their extensive skill set. In an open and collaborative partnership, Pragmatic LLC will bring a lot to the table. Their services are extremely beneficial.

BACKGROUND

Introduce your business and what you do there.

I'm director of professional services for an e-commerce digital agency. We work with some of the biggest brands in sports, fashion, and retail.

OPPORTUNITY / CHALLENGE

What challenge were you trying to address with Pragmatic LLC?

We needed additional capacity and a level of expertise.

SOLUTION

What was the scope of their involvement?

We do a number of e-commerce implementations with Salesforce, Magenta, and other platforms. The Pragmatic team was mainly responsible for developing automated QA tests. Their main two KPIs were how stable is the framework that they developed and is it usable. We also looked at whether we needed to spend a lot of time on maintenance.

What is the team composition?

Several automation QA people were assigned to our account. I also worked directly with one of Pragmatic’s founders.

How did you come to work with Pragmatic LLC?

The founders used to work for me before they decided to open their own business. One of the main reasons I went with them is because I already knew their capacity and skill set.

What is the status of this engagement?

The project ran from November 2017–January 2018.

RESULTS & FEEDBACK

What evidence can you share that demonstrates the impact of the engagement?

They completed the project and I was pleasantly surprised by how well they did. They were really able to get to the root causes of some of the issues that we were having. Not only did they come up with a proposal for improvements, they implemented those fixes as well. It went even better than expected, and I would definitely use them again in the future.

How did Pragmatic LLC perform from a project management standpoint?

They were very professional and able to communicate quite well in English even though they’re based in Bulgaria. We primarily used Skype and Slack to communicate along with email. One of the founders even spent some time working from our office in order to cooperate even better with my team.

What did you find most impressive about them?

The team members had an impressive technical skill set. We were also really pleased by the way they proactively proposed enhancements. They told us how we could improve different areas and then implemented these improvements.

Any advice for potential customers?

Any time you work with a company from another part of the world, you should be aware of cultural differences. Trust Pragmatic because they are honest and a good company.

BACKGROUND

Introduce your business and what you do there.

I’m the CTO for EasySecure International, which offers a cloud application that is mainly used for time and attendance and access control. I’m responsible for product development, consultancy, and customer support.

OPPORTUNITY / CHALLENGE

What challenge were you trying to address with Pragmatic LLC.?

We try to post a new release every month, which means we have a lot of testing to do. We were looking for a way to automate this.

SOLUTION

What was the scope of their involvement?

They’ve been doing the QA for us. We have about 1,200 automated tests running every night now. They provide the manual testing for us as well. They also give us advice concerning whether our graphical user interface is consistent or compliant with the overall standards.

What is the team composition?

We work with Nikolay (QA Automation Engineer, Pragmatic LLC.). They trained him first and then hired him for us. He works with Aleksandar (CTO, Pragmatic LLC.), one of the co-owners of the company, who is very experienced. They worked together for the first three months. Now Nikolay is doing most of the work.

How did you come to work with Pragmatic LLC.?

When my partner was looking for automated testing, Pragmatic LLC. turned up in our Google search. We started to investigate them. When we were mostly convinced that they could be a worthwhile partner for us, we visited them in Bulgaria. They showed us their company and their skills for a whole day. That was an eye-opener. We found out that they could contribute much more than QA.

What is the status of this engagement?

We started working together in February 2018 and the work is ongoing.

RESULTS & FEEDBACK

What evidence can you share that demonstrates the impact of the engagement?

The 1,200 tasks that are now automated give us much more time to do the manual testing and focus on specific problems or challenges.

How did Pragmatic LLC. perform from a project management standpoint?

It’s well managed and under control. They stick to the agreements we make. We communicate on a weekly basis through Slack. They’re very nice to work with.

What did you find most impressive about them?

We’re now working with all kinds of project tools that they handed over to us like Slack, Jira, and GitHub. We use those tools together with our developers whereas before we only used email. Due to the experience with Pragmatic LLC., our whole development process is much more transparent.

Are there any areas they could improve?

We asked them to deliver reports every Friday to see what’s going on. Sometimes I have to ask again to get that information.

Do you have any advice for potential customers?

Meeting face-to-face is helpful in starting a business relationship. Take the initiative to go visit them.