QualityLogic

Introduce your business and what you do there.

I’m at Tubi, a video streaming and entertainment service. Within the organization, I’m the senior director of product for core systems and foundation, the department responsible for our core video experience, international product experience, privacy compliance, and internal tools.

What challenge were you trying to address with QualityLogic?

Our goal was to increase our throughput of linear channels on Tubi. At the time, we were averaging about 15 channel launches per month, and our goal was to hit 200 linear channels by September 30th, 2022.

In order to hit that goal, we needed to increase the number of linear channels we were launching per month. However, our bottleneck was that we lacked QA support to QA our outputs before launching them into our platforms. Thus, we decided to hire QualityLogic to increase our QA capacity and hit that goal.

What was the scope of their involvement?

QualityLogic has provided us with QA services for our platforms’ linear channels, which are similar to live TV channels. To launch those channels, we first get the stream of assets from partners such as ABC, NBC, and FOX, and our engineering team uploads those assets. Then, QualityLogic provides QA to ensure videos come through our 25 different platforms correctly.

Their QA process includes testing on all those platforms. Whenever they find an issue with things like subtitles, sound, or video quality, they create tickets and work with our engineering team to ensure that those issues are resolved.

Our goal was to launch 200 channels, but halfway into the project, we realized we had extra bandwidth, so we’ve also tasked QualityLogic with other things. Since hitting our goal, their team has been helping us support our existing channels, and they provide ongoing QA to ensure we don’t have any issues with our platforms.

What is the team composition?

We worked with two full-time QA agents, a part-time project manager, and a part-time program manager.

How did you come to work with QualityLogic?

Our director of QA at Tubi found them, contacted them, and handed them over to me to handle the contract negotiations. One of the things that stood out to me was that they were in the US, so we would be able to work the same hours.

How much have you invested with them?

So far, we’ve invested about $60,000, but we’ve budgeted around $250,000 for the year.

What is the status of this engagement?

We started working with them in July 2022, and our engagement is ongoing. We plan to use them for the rest of our fiscal year and beyond.

What evidence can you share that demonstrates the impact of the engagement? 

QualityLogic has exceeded our expectations. In order to hit our goals, we needed to increase our channel launch capacity from 12–15 channels per month to over 20 channels. However, they achieved a much higher throughput, reaching 30 channel launches per month and hitting our goal in half the time we had budgeted. By late September 2022, we already had 202 channels. They were a pivotal part of helping us achieve our goal; we wouldn’t have achieved it without them.

How did QualityLogic perform from a project management standpoint?

QualityLogic moves very fast. We’ve spent half the money we had originally budgeted because they’ve done more than we thought they would do, demonstrating that they’re a hyper-focused team. We’re very pleased with how fast they get the job done, which ultimately saves us money and allows us to have extra bandwidth to work on other projects.

Due to security issues, we can’t provide QualityLogic with direct access to Shortcut, our internal project management system. Instead, their team uses Excel to create tickets and pass them over to us to then create the tickets in our internal system. However, we’re working on creating a third-party integration that will enable them to pass those tickets directly to Shortcut so that we don’t have to enter each one manually.

As for communication, we have a shared Slack channel through which QualityLogic communicates directly with our engineering team. Additionally, their team joins relevant meetings via Zoom, and we have monthly and quarterly business review meetings with them.

What did you find most impressive about them?

QualityLogic has been very flexible and professional; they’ve done a fantastic job all around. They also have really talented people who have added to our culture in a positive way and become valued members of our team, ultimately improving our productivity.

Additionally, I really like that they’re able to scale their team up and down very quickly, and they don’t try to secure a long contract. Instead, we’re able to work with them month by month based on our needs.

Are there any areas they could improve?

No.

Do you have any advice for potential customers?

If QualityLogic joins your team, allow them to join completely. To get the most out of them, you’ll need to treat them as part of your team and give them access to the resources they need to do their job. When people are happy, they’re more engaged with their work, and they’re ultimately more productive.

Introduce your business and what you do there.

I’m a director at a robotics company.

What challenge were you trying to address with QualityLogic?

We engaged QualityLogic to help us with QA testing for our application.

What was the scope of their involvement?

QualityLogic helped us with manual QA testing. The team performed regression and production testing and documented their findings in test cases. Once they identified bugs, they took screenshots and videos so our development team knew what to look for. Then, our product managers reviewed the test cases, and our development team implemented changes to address the bugs. When QualityLogic identified new bugs that affected the application’s functionality, they advised our product team to reopen the story and triage the bug. 

What is the team composition?

We worked with two QA engineers and a project manager from QualityLogic. 

How did you come to work with QualityLogic?

Our company engaged QualityLogic before I joined. Our team had heard positive feedback about QualityLogic through word of mouth, so we approached them to help us with the project. 

How much have you invested with them?

We invested between $10,000–$15,000 in the partnership with QualityLogic. 

What is the status of this engagement?

We worked with QualityLogic between December 2021–August 2022.

What evidence can you share that demonstrates the impact of the engagement? 

We measured QA success by looking at the success of each release. They reported all the bugs efficiently and caught most of them before each release. The QualityLogic team delivered thorough test cases and performed immaculate work. We had some problem cases, but those were exceptions.  

How did QualityLogic perform from a project management standpoint?

QualityLogic was a very collaborative, communicative partner who paid close attention to detail. They were kind and dependable, which made the process easier for us. We had daily stand-ups via Google Meet and used Jira to manage the project. We also communicated via Slack.

What did you find most impressive about them?

I was most impressed by their proactive communication style, reflected in the precision of their test cases. QualityLogic consistently followed up with us and took the initiative to loop in our developers as they saw fit. When they identified particularly problematic issues, they’d advise us to block the next release, which was helpful. Throughout the partnership, they kept our customers in mind, which helped us in the long run. 

Are there any areas they could improve?

I have nothing QualityLogic could improve on.

Do you have any advice for potential customers?

I’d recommend engaging with QualityLogic to experience the great outcomes for yourself.

Introduce your business and what you do there.

I’m the CTO of Direct Care Innovations. We’re a healthcare technology company offering SaaS-based software to service providers in the Medicaid and Medicare markets.

What challenge were you trying to address with QualityLogic?

We were in the testing phase for multiple projects and didn’t have enough internal resources to perform quality tests. We wanted to outsource to a vendor who had the skills we lacked.

What was the scope of their involvement?

QualityLogic test all of our new developments and bug solutions. They conduct tests on our products’ functionalities, performance, and security. Their team uses their own testing tools and show us analytics on the success of our products. The reports show if bugs have been solved and if a system functions correctly. When they find an issue, they screencast with our developers to make the locating and reproducing process simpler.

What is the team composition?

They provide us with five full-time testers and one part-time tester, who also functions as the project lead. Their team is on constant retainer, so there is always a member available to help us.

How did you come to work with QualityLogic?

I searched for vendors online after performing marketing research. One of our system analysts found them, and we preferred their smaller composition compared to the other behemoth-like companies. We also liked their flexibility toward an open-ended arrangement with an undefined scope of work.

How much have you invested with them?

We’ve spent around $150,000, so far.

What is the status of this engagement?

We started in March 2017, and we keep them on retainer for our testing needs.

What evidence can you share that demonstrates the impact of the engagement?

We’ve been able to increase the number of test cases we get through drastically. Along with quantity increasing, our KPIs show that the thoroughness of the tests is much better than we were doing previously. Our customers have given us positive feedback on the enhanced usability of our system’s overall performance and security.

How did QualityLogic perform from a project management standpoint?

They provide us with detailed documentation about the issues they find and give us quality suggestions for solutions. We receive daily updates from the project lead, which is one of the best parts of working with them. They deliver everything on time and within the budget we set.

What did you find most impressive about them?

Their entire team is professional, knowledgeable, and willing to go the extra mile—we consider them a partner. We rely on them heavily, and they’ve never let us down, so we find the project to be money well-spent.

Are there any areas they could improve?

Although there have been inevitable bugs that went undetected, it’s infrequent, and they give us a reasonable explanation. Our work is complex, and there will always be some issues that slip through—we can’t blame them for that.

Introduce your business and what you do there.

We are a large associate development company, with locations all around the world. I work in software quality.

What challenge were you trying to address with QualityLogic?

We needed to support a new file format in our machines. There were some new specifications that were being developed at the same time we were developing our machines. We needed QualityLogic to help develop a suite of files that covered the specifications. We also needed help developing the tools to test the files with our machines.

What was the scope of their involvement?

We had two main goals. We handed QualityLogic the specifications that were in development for the new file format. First goal: QualityLogic needed to develop a set of files, a plot suite, that covered all of the specifications, including the possible variants, the positive and negative clauses, and everything that the specification says. The specifications were very complex with a core module and several extensions. In the end, there were 300 different plots that covered all the specifications. Our proposal was to make sure that our machines were compliant with the specifications. If a machine is going to consume this kind of file format and be considered "compliant with the spec," it needs to accept certain kinds of files and reject other kinds of files. If all the positive cases pass and all the negatives cases fail, we could be sure that we were compliant with the specification.

The second goal was to develop a set of tools so that our team didn’t need to test our machines manually. Every week, there are 300 untested files. QualityLogic developed a set of tools that were automatically able to connect with our machines, send the files, and report the results. The evaluation needed to be formed according to both our quality standards and those of our development team. Every time our development team made a change to the code, they had to make sure that it was compliant with the specifications.

What is the team dynamic?

I had one person that I interfaced with, and I also spoke to a senior engineer. We discussed requirements and technical matters and also budget, scope, etc. I know they had a team of developers and testers behind them, but I only interacted with these two people.

How did you come to work with QualityLogic?

They had worked with my company 15 years ago for another project. We recalled that their work was very good, so we decided to contact them. It was an internal recommendation from a colleague.

How much have you invested with them?

Our initial budget was around $200,000. We expanded the budget a couple of times because we wanted to perfect some tools and add more capabilities with them, to a final of $250,000.

What is the status of this engagement?

We started working with QualityLogic in the spring of 2016; the first project is finished and the second project is ongoing. The first set of standards was completed in seven or eight months.

What evidence can you share that demonstrates the impact of the engagement?

The most valuable metric is that we ended up with 300 valid test cases developed by QualityLogic. We also encountered 20-25 issues in our code that we found thanks to this plot suite.

We didn’t intend to qualify the specifications, but QualityLogic found several issues in the specifications themselves. They were in contact with the authors so they could correct them. There was a lot of success in the requirements. That was very valuable because if we had built the code based on wrong specifications, it would have cost us a lot of effort to re-build later on. That was not expected, but QualityLogic reviewed all the specification and asked questions and proposed improvements which was very helpful.

How did QualityLogic perform from a project management standpoint?

We had a meeting every week with a call. In every call, they told us the progress in the development and the new plots. We discussed requirements and priorities. If they needed to choose between different options for a tool, for example, we discussed the best approach. They would inform us on the budget and the cost of doing a certain functionality. They were very transparent. We didn’t have any problem with priorities or time estimations. If they said they were going to do something in a certain time, they made sure they did. We used GoToMeeting for our communication.

What did you find most impressive about them?

Their commitment is impressive. They were very involved in the project. They cared about the success. They understood our problem and wanted to know how we were going to use the product in order to adequate their job. They went above and beyond our expectations. They were good at avoiding failures and giving suggestions. In the beginning, I was the only interface with my company, but in the end, we saw that it was better if they spoke directly to our engineers. Now there’s very frequent communication between their team and ours. Their work is going well, so there will be other opportunities working with us.

Are there any areas they could improve?

No. If I had any suggestions of improvement for them, I told them and they fixed it right away.

Introduce your business and what you do there.

Our organization provides online invitations to events, and I am a mobile QA manager for our apps on both iOS and Android.

What challenge were you trying to address with QualityLogic?

We were a little bit shorthanded and were looking for some help with our regression testing and regular tickets, so we reached out to QualityLogic for that.

What was the scope of their involvement?

Our biggest pain point was that our regression testing was taking a long time, and we were essentially able to just hand that off to QualityLogic for their team to run with. Our product had developed a lot of new features in a fairly short amount of time, but they were able to really build out nuanced test cases to deal with the complexity of the application and then run manual testing from there.

What is the team composition?

There are three individuals from their team on our project, two of whom work full time on QA and one who is the project manager.

How did you come to work with QualityLogic?

I scheduled phone calls with several companies that I found on the internet, and we really liked their business model what their team had to say so we decided to work with them.

What is the status of this engagement?

We started in October 2019 and the engagement is ongoing.

What evidence can you share that demonstrates the impact of the engagement?

With the help of QualityLogic, we’ve been able to accelerate our update schedule from around monthly releases to every two weeks. That’s a really significant jump for us, so we are highly satisfied with the quality of the work so far.

How did QualityLogic perform from a project management standpoint?

For the most part, communication has been very straight to the point and direct, which we really appreciate. Not having to filter through a lot of business talk saves us a lot of time and makes sure we are able to address each problem or improvement that they raise. We use a JIRA board to track progress as we go.

In addition, their tickets are incredibly detailed, which makes it easy to read through them and understand exactly what is going on. That allows for minimized back and forth over specific issues so that we can use Slack and email to discuss larger changes and strategies without distraction.

What did you find most impressive about them?

I’ve worked with other remote companies that handle QA before, and I think that QualityLogic has been much better prepared than almost any others. They fundamentally understand QA, so I don’t have to double-check their work at all.

Are there any areas they could improve?

I’ve been really happy with them, so I don’t have anything I think they need to improve.

Do you have any advice for potential customers?

Provide as much feedback as you can early on if there’s anything you want from them that isn’t getting done. As long as you give clear directions and communicate throughout the project, you’re going to be satisfied with the work that they do.

Introduce your business and what you do there.

Stand Up To Cancer is a division of the Entertainment Industry Foundation, a 501(c)(3) charitable organization that raises funds to support collaborative cancer research.

What challenge were you trying to address with QualityLogic?

We had a very large-scale event that usually sees a very high level of transactional traffic. We needed QualityLogic to help make sure that all of our forms were functioning correctly and that there were no surprise issues that would be encountered by any of our users.

What was the scope of their involvement?

We engaged QualityLogic to run transactional testing as well as cross browser and device testing. We began by outlining what we were concerned about, and then QualityLogic looked over the plans, suggested what we should test, and then ran a full suite of functional and cross browser tests for us to make sure pages displayed correctly and that there were no bugs or submission issues.

We've worked with QualityLogic for three years and the team that we work with has always stayed consistent. It's made it very easy to engage with them. They're pretty familiar with our forms so it's easy to go back to them time and time again. We worked heavily with Don Walker, Paul Morris, Don Burke, and several other people on the team, and they've always been really great.

How did you come to work with QualityLogic?

Someone we had worked with knew of them and said that they might be worth considering. When we first engaged with them, we looked at several different testing companies, and we just liked the style of QualityLogic. They were very technical, but easy to work with. It was our first engagement with a firm to handle this kind of work as a neutral third party.

What is the status of this engagement?

We started working with them August 2014 and have done several projects together since that time.

Could you share any evidence that would demonstrate the productivity, quality of work, or the impact of the engagement?

For each engagement, they’ve found at least 20 reasonably significant bugs that could be addressed. On a recent project, we ended up with zero user complaints of any issues on the forms.

How did QualityLogic perform from a project management standpoint?

They’re very proactive about communicating what they’re working on and their schedule estimation is always very accurate. They are very flexible in timelines and the testing, and they really understand digital work cycles. When we had delays or things that came up, the team was really professional and easy to work with. Our correspondence has been mostly over the phone and email, and then also working with tracking through shared documents.

What did you find most impressive about QualityLogic?

What’s always impressed me is the accuracy and thoroughness of their work and their flexibility. They are usually able to help us make sure that we can walk into any page launch feeling confident that there aren’t any issues that we don’t know about. They’re a good third party sanity check for us.

Are there any areas QualityLogic could improve? ​​​​​​

Not that I can think of.

Introduce your business and what you do there.

I am the QA manager of an app development group working on behalf of a large media company. We focus on multiple platforms, including mobile over-the-top content on iOS, tvOS, Android mobile and TV, Amazon Fire TV, and a few others, including the web.

What challenge were you trying to address with QualityLogic?

We were growing fast, and needed testing help. QualityLogic was one of a few vendors who were available to us.

What was the scope of their involvement?

We assigned QualityLogic some projects and discussed how we wanted the testing to be done. They were open to suggestions and new testing methods. They learned the process from us and took ownership of the testing for a wide array of our apps. This has included all-manual testing, not just black-box (having users interact with the apps), but also with a more complex approach. Based on a user’s location, they will receive different streams for live viewing. For instance, if someone wanted to see a particular New York station, the app would recognize that they are in that location, and give them the right feed. This is challenging to do with black-box testing because we can’t manipulate locations, and need to go into a gray-box area, looking at the APIs and manually manipulating responses so that we receive the desired results.

Another example: our app can handle five shows on a page; we wanted to see what would happen if there was a sixth show. Rather than us programming a test for each use case, QualityLogic has looked at our API and its responses, manually added shows, and assessed the reactions.

The composition of the team has changed recently. We have recently expanded to 50 people, but my interactions are with the project manager and the leads. We have also decreased the team size when there were fewer projects in a fiscal year.

We have worked with other testing vendors, but not at the scale of QualityLogic. They hold around 75-80% of all our projects.

How did you come to work with QualityLogic?

We used to have two different teams in-house: a web team and a mobile player team, which I was a part of. QualityLogic was contacted by someone on the web part. The teams have since merged, and I took over the QA department. I contacted QualityLogic and started giving them business minimally, at first. After they were onboarded around processes and functionalities, we started giving them more projects.

What is the status of this engagement?

We started working with QualityLogic 2 ½ or 3 years ago for my department. They may have started working for the company a little earlier.

Could you share any evidence that would demonstrate the productivity, quality of work, or the impact of the engagement?

Over the last year, we haven’t had any major issues in production after the testing performed by QualityLogic. Software will always have bugs, but we haven’t had any major ones that would make us stop everything and rerelease the code. QualityLogic has full coverage of test cases, and the work is done to a level that makes us confident in whatever they provide signoff on.

How did QualityLogic perform from a project management standpoint?

I deal directly with Joe Walker, their program management director, who is extremely flexible and able to present a plan for any challenges I present. I’ve also talked with one of their QA managers.

What did you find most impressive about QualityLogic?

The type of testing we do is not standard, not something which could be done by low-cost resources in India, for example. With QualityLogic, we have access to testers working at engineer levels. They’re willing to adapt, and they’re quick learners. We can teach something to one person, who will then relay it to the whole team.

Are there any areas QualityLogic could improve?

There were things to fix in the beginning, and we provided QualityLogic with ideas on how to change things around, but we haven’t had any production issues in the last six months to a year. When we first started, if there was anything missed on their side, they owned the mistake, which was a positive for us. We still provide feedback to them, but we haven’t had any recent issues.

Introduce your business and what you do there.

I’m the CTO and VP of engineering for a company that makes mobile and SaaS [software-as-a-service] solutions for emergency and clerical communication management. 

What challenge were you trying to address with QualityLogic?

We needed testing for our mobile apps and web app in a way that could be repeated without staffing our own team. We also needed enough physical devices to test everything properly. We’ve worked with other quality companies that focus on test automation. Despite all the automated tests as we build and execute, we still need manual, ad hoc, and exploratory testing. 

What was the scope of their involvement?

The relationship was already in place when I joined the company. Now, they provide testing services for our applications. As we add more features, they incorporate new tests and have built up a library of those to determine functionality. If new issues are found, they integrate with our defect-tracking system directly. They’re able to enter what type of issue they find, how to reproduce them, and which accounts and users they’re acting on behalf of when they found it. We have other communication channels between them, the development teams, and the support teams in order to refine and triage items and determine what actions we should be taking.

We had a single main point of contact. At the beginning of the year, he started joining us twice a week in our daily standup meetings. He comes to the office and sits in our scrum. That’s working really well. A few other people are involved too.

How did you come to work with QualityLogic?

They’re a local company, and I think a manager of ours at the time ran into them somewhere. I’m not exactly sure how they got connected because that predated me.

How much have you invested with QualityLogic?

We have spent between $150,000 and $250,000 so far.

What is the status of this engagement?

We started working with them in June 2014 and the work is ongoing. 

Could you share any evidence that would demonstrate the productivity, quality of work, or impact of the engagement?

We have a great relationship, and they do excellent work for us. We have urgent timing on some of the releases that we make to enterprise customers. These tasks have time-sensitive events where we’re releasing things for use around the world. On some occasions, QualityLogic has been asked to help us test things really quickly or over the weekend. They’ve always been responsive and worked with us on those urgent requests.

How did QualityLogic perform from a project management standpoint?

We use JIRA for tracking all of the R&D work. They interface directly with JIRA and our Atlassian accounts. We use HipChat for communication about testing and qualification. We also use email and phone to communicate, and our project manager attends our meetings twice a week.

What did you find most impressive about QualityLogic?

They’ve developed a keen understanding of our applications. They know how they work as well as we do, maybe even better than some people on our team because of their broad usage. They take ownership of what they do and are quite proactive.

Are there any areas QualityLogic could improve?

I can’t think of anything. They offer good service. They don’t get into test automation, but that’s not a downside really.  

Introduce your business and what you do there.

I’m the VP of engineering at BrightLine, an ad tech company within the OTT space.

What challenge were you trying to address with QualityLogic?

We were lacking internal QA support, so we hired QualityLogic.

What was the scope of their involvement?

QualityLogic tests different software development kit (SDK) products for different platforms, including Apple TV, Android, Roku. They test them on different devices as well. We’ll usually have a kickoff session to gain an understanding of the product and what it should do and how things should flow. 

The team then creates test cases and proceeds to do tests on a per-platform basis. Throughout that process, if there are any questions or ambiguities, they’ll stay engaged to clarify them.

What is the team composition?

We typically work with one QA lead, but it could be a combination of two at any given point.

How did you come to work with QualityLogic?

Our previous CTO onboarded them, and I picked up where he left off.

How much have you invested with them?

The cost ranges depending on the level of support we need. Typically, we engage them for at least 20 hours per week, but it can go up to 40 or more hours. We spend $5,000–$10,000 per month with them, and the total cost so far is in the $50,000–$200,000 range.

What is the status of this engagement?

We started working with QualityLogic around October 2018, and our engagement is ongoing.

What evidence can you share that demonstrates the impact of the engagement?

It’s been tremendous. Before working with QualityLogic, we had a single QA teammate internally, and we were experiencing a lot of bugs and complaints from media partners. 

We saw a huge decrease in complaints after we brought QualityLogic onboard, and we saw a big increase in the quality of our releases. It’s been a tremendous impact overall because they’ve helped us decrease bugs, processes, and things of that nature.

How did QualityLogic perform from a project management standpoint?

We discuss what should be tested during the week, and they give us daily updates on any issues they find and report them in Jira. QualityLogic also gives detailed descriptions of how they found the issue and where, along with how to reproduce it. They have a very good process for that.

We have weekly sessions every Monday, and we have a lot of communication over email, which is pretty clear. We also hop on phone calls if we can’t get the understanding through email. 

What did you find most impressive about them?

This is the first QA team I’ve worked with while at the company, but they’ve definitely been better in terms of process and tracking compared to our internal resources.

Are there any areas they could improve?

It’s not necessarily a bad thing, but, in terms of the collaboration between our companies, having more scheduled sessions would help.

Although we had weekly meetings, more frequent sessions throughout the week would be good as well. It can take me longer to get to an email if I’m swamped with work, so having calls and screen shares would make it easier to describe elements of the project.

Do you have any advice for future clients of theirs?

QualityLogic is very in-depth with the products they QA, but we may have an urgent release, where identifying blockers becomes most important. It’s important to understand the product in terms of what’s critical and what isn’t and how that can impact other aspects.

Introduce your business and what you do there.

I’m the technical and managing director of the OpenADR Alliance. We’re a membership-based industry standards group for the smart goods space which includes devices like smart thermostats and electric-vehicle charger stations.

What challenge were you trying to address with QualityLogic?

One of our core organizational functions is to organize tests and certify products according to our standards. Almost nine years ago, we made the decision to develop and own a specific certification test tool that we would use to certify products. We needed a partner to work with us to develop it.

What was the scope of their involvement?

We’ve been working with QualityLogic for almost nine years now. They initially built the test tool for us including all of the testing, the test plan, and some test cases. We now co-own the test tool with them and QualityLogic serves as an endorsed partner that can test products for our members based on our standards and certify them. They’ve delivered that tool to a number of our member companies. They provide consulting services and additional development support to our members upon request. We also maintain a working relationship to update the software every couple of years and do other, smaller projects as needed.

What is the team composition?

We work directly with a lead engineer who serves as a project manager and with the general manager. There are other programmers who work behind the scenes.

How did you come to work with QualityLogic?

We put out an RFP to a number of test equipment companies as well as test houses. We didn’t want to just hire a programmer to build the test tool; we wanted to establish a partnership. We understood that we would need support and modifications over the years and that a long-term arrangement would be more effective. We eventually selected QualityLogic based on a combination of their experience, their pricing, and the partnership arrangement they offered.

How much have you invested with them?

We’ve spent about $400,000.

What is the status of this engagement?

We started working together in July or August 2011 and it’s ongoing.

What evidence can you share that demonstrates the impact of the engagement?

We haven’t specifically tracked performance but the test tool works very well. I get very few questions about it, if at all, and we’ve had a positive response.

How did QualityLogic perform from a project management standpoint?

They’ve performed well. Working directly with one of their lead developers has worked well for us. He’s always been on top of things. They’ve worked well according to timelines and plans.

What did you find most impressive about them?

I would say two things. For one, they’re one of the few companies I’ve found with the experience in test systems and testing programs of our sort. QualityLogic seemed to be the one company that had the most relevant testing experience. They also wanted to dive into the smart goods space, which was great for us.

Also, they have the right people on their team. Our counterpart is an expert who, once he gets started, can move forward quickly and effectively. He’s incredibly knowledgeable and easy to communicate with.

Are there any areas they could improve?

I can’t think of anything in particular. We’ve been working together for such a long time and everything moves the way we’d like it to. They could potentially be a bit better at tracking their hours. It would be nice to have advance warning if we’re going to exceed our estimates. But I know it’s sometimes hard to quantify work when you’re building something from scratch. The standards were changing as they built the test tool.

Do you have any advice for potential customers?

The challenge is always to scope a project well, which can be tricky given so many unknowns. Make sure your project is fully outlined and most aspects are clear. That way there won’t be any surprises.