What evidence can you share that demonstrates the impact of the engagement?
The tests that we’ve launched are still running. We have some conclusive behaviors that are ready to call, but we haven’t permanently implemented any changes from the test.
The thing about A/B testing in general and ConversionAdvocates’ work specifically is that it’s less about finding a winning design and more about learning along the way. Then, we can incorporate those learnings into future designs and tests. We’ve had plenty of those; for example, one of their insights was that our booking flow didn’t provide enough confidence, information, and context to somebody who’s less familiar with our service. We’ve noticed that the designs we’re testing that they designed are performing really well with paid SEM. The insight there is that those are customers who don’t know us as well as direct traffic, Facebook traffic, and those types of channels. Having that additional context in our booking flow has been a critical learning for us that we’ll carry forward in our final design.
How did ConversionAdvocates perform from a project management standpoint?
They do all the project management in Asana. They keep Asana super organized and do a really great job of tracking progress in that tool. Asana is a new tool for me personally; I’d heard of it but never used it. It’s easy to pick up, and ConversionAdvocates keeps it up to date, so it’s always available whenever I want to look into their work. In addition, they send weekly updates on the status of each ongoing test.
At some point, I decided that it would be beneficial to have weekly calls. I meet with two of them and loop in the data analyst when necessary. I find it really helpful to have those calls; if we don’t need them, we cancel them, but they allow us to get into more detail about specific initiatives or feedback that we have. I also find it helpful to offer design feedback verbally rather than asynchronously. Managing every piece of the project could be a full-time job if I were their only client, so I don’t know how they do it for me.
What did you find most impressive about them?
Their data analysis reports are where they shine. They segment things in many different ways so that you can really understand behaviors across channels, sources, devices, etc. The qualitative insights that you can tie to those quantitative insights are also really valuable.
I think any company could be helpful by just offering a new perspective from outside of one’s own day-to-day operations, but ConversionAdvocates has taken it a step further. They’ve been able to offer insights that we may not have otherwise discovered because of their segmentation and deep analysis of our product.
Are there any areas they could improve?
It took us a little bit to get into the swing of things and our working pattern, but that’s true of any engagement, contractor, or new employee at a company. I think that adding the weekly meetings with them by video has been a really critical piece of making sure that we can hit our stride.
Earlier in our conversations, they’d suggested starting with a one-month engagement to make sure that it was a good fit all around, but I’m not sure that any company can reasonably expect to see results in the first month. In fact, I’d say it took about 2–3 months for us to really hit our stride in our working relationship and be able to make some change. My recommendation to them would be to have a weekly call sooner rather than later, because since we’ve implemented that, we’ve had a much smoother experience keeping multiple project tracks moving forward.
Any advice for potential customers?
As I mentioned before, one month is not enough, and three months was barely enough to start seeing results. We started with a three-month engagement, and we just renewed our contract. Every business is different, but at least for ours and others in the space, I think three months is the minimum you should expect.
Make sure you keep the lines of communication open. Share a lot about your history, strengths, and where you need help.