Beyond the Arc

BACKGROUND

Please describe your company and position.

I am the Executive Director-Global AI/ML of Johnson Controls

Describe what your company does in a single sentence.

JCI manufactures and services equipment used in commercial buildings

OPPORTUNITY / CHALLENGE

What specific goals or objectives did you hire Beyond the Arc to accomplish?

• Build customer churn models using ML

SOLUTION

How did you find Beyond the Arc?

Predictive Analytics World

Why did you select Beyond the Arc over others?

• Great culture fit
• Company values aligned

How many teammates from Beyond the Arc were assigned to this project?

2-5 Employees

Describe the scope of work in detail. Please include a summary of key deliverables.

Beyond the Arc provided us POC churn models for two of our LOBs in Global Services. They led numerous data discovery sessions with key business stakeholders and helped us collectively understand the importance of business context in Machine Learning. We received monthly lists of at-risk customers, predicted via ML, from them until an organic team could be stood up internally.

RESULTS & FEEDBACK

What were the measurable outcomes from the project that demonstrate progress or success?

I started in the role I was in with no data science team, no data engineering team, no constructed datasets and a company that had never executed a machine learning project before. BTA helped us uncover the data (in very old source systems), get it formatted (snapshotted) correctly and delivered from it monthly lists of at risk customers. We were, from there, able to begin to learn how to retain at risk customers from the lists.

Describe their project management. Did they deliver items on time? How did they respond to your needs?

BTA not only delivered projects on time, they did so with challenging timelines. In fact, their performance under difficult circumstances made quite an impression on numerous internal team members.

What was your primary form of communication with Beyond the Arc?

Virtual Meeting

What did you find most impressive or unique about this company?

The comprehensive approach to problems solving.

Are there any areas for improvement or something Beyond the Arc could have done differently?

Honestly, no. The performance and services provided were flawless and demonstrated significant competence in the space and, more importantly, tailored solutions to our business needs.

BACKGROUND

Please describe your company.

We are a very large and well-known international financial institution.

What is your role and responsibilities?

I’m a strategic planning manager. I work specifically focused around the customer experience, and how we handle customers. I work with some different teams to do analysis on feedback that we get from customers through different channels. 

We take general feedback from customers, and see if there’s something we can do to change communication to make it clear or change a process to make it easier. My team looks at all customer issues, and we try to improve the customer experience. Our job is to keep the customer satisfied.

OPPORTUNITY / CHALLENGE

What was the business challenge you were trying to address with the work with Beyond the Arc?

We wanted to analyze customer information through different data sources regarding customer feedback. Beyond the Arc was really instrumental in helping us look at that analysis to identify emerging issues. They helped us create a sustainable and repeatable process to address these issues that are apparent from data analysis.

SOLUTION

What was your process for selecting Beyond the Arc?

They came highly recommended to us, so we selected them out of a group of three to five firms.

Can you tell me a bit about the scope of the project? 

Beyond the Arc looks at the data, then they stage and prep the data to see what it’s telling us. They’ll utilize different data sources depending on what we’re looking for. 

They have developed good rules on how to present the information. They usually create a model around that piece of it, and then validate the information so they feel confident in the report. 

They also provide an output like a report or presentation to help us achieve our customer satisfaction goals. They are experienced, and can explain the steps of a new process to improve. If it’s something repeatable, they can explain it to our team. They sometimes provide training to help our team adopt best practices.

What technologies were used?

I’m not sure, but I think SAS and SPSS.

Can you give a sense of the size of the initiative in dollar terms?

I am not allowed to disclose that number, but it’s a fairly large engagement.

When will the work be completed?

The work is ongoing, and we have no plans to stop it soon. The work has been very valuable to my team, and we will likely expand it later.

RESULTS & FEEDBACK

What were the results of the project?

We track metrics on different projects. Beyond the Arc is great at finding trends and issues in the data. They have helped us improve customer satisfaction. With many projects, the goal is to reduce customer feedback. They have helped us in this regard greatly.

What is unique or special about Beyond the Arc compared to some other vendors?

Their team is so professional and easy to work with. The gentleman that works with me full time is great to work with and professional. They understand our business and quickly became part of the team. They have a close relationship with members of my team. I can only say positive things about them.

Looking back on the project, is there any area that you think they could improve upon or maybe you would do differently?

I really don’t because they have been very successful here. I may have something to suggest for improvements in the future. Now, we are very satisfied with their modeling because it is accurate and we enjoy their work attitude.

BACKGROUND

Can you provide a brief description of your company?

We are primarily a social networking/dating site. We have around 500 employees.

It is primarily US based. We also have an office in Taiwan. We have smaller offices in Europe and South America. Those offices are relatively small, they deal with banks in the local region and do tax work.

What is your role and responsibilities?

I’m the vice president of business operations, which basically means that I’m in charge of everything technology-based that touches revenue. I manage our billing team, which does a lot of the code, and talks with the credit card companies and does the building order forms. I run the business analytics group, which does a lot of site conversion rate work and financial reporting. I do the user fraud piece. That probably makes up the bulk of what I do.

OPPORTUNITY / CHALLENGE

What was the business challenge you were trying to address or your goal for that project?

The business piece we’re doing right now is based on credit card fraud. My company is trying to predict what orders will want a refund. There are two types of refund. One is when the customer calls us, and just wants their money back. Those are called credits.

There is another type where the user will call the bank. The user calls their bank, and contests the charges. That is called a chargeback. Visa and MasterCard each have limits on the amount of the percentage of sales that are allowed to be chargebacks. It’s roughly 1 percent for Visa, 1 1/2 percent for MasterCard. We basically try to keep our chargebacks numbers well below that number so that we don’t run into trouble.

So, you are trying to predict which customers to avoid?

Yes. We are doing it in two different ways. Number one is, we’re trying to predict which users to decline as they hit the submit button.

The other type is for orders the user doesn’t hit submit for. If it’s a monthly product, and we just auto renew, we’re going to have it approved before renewing. For many orders, we are going to rescore 24 to 72 hours later. Therefore, we want to find out who to proactively credit back. On the website, each user has a future score or risk calculation. My company can decide to reject the order, cancel the product, and refund the charge.

SOLUTION

Can you describe the scope of Beyond the Arc’s work?

Beyond the Arc’s work was mostly around training me to use the product with this one goal specifically in mind. Rather than just go through a generic training class on how to use SPSS Modeler, we worked with this specific data. Beyond the Arc’s team came into my office, and sat next to me three days a week for several weeks. I have more time scheduled with them in November. The bulk of their work was training me to use the product, watching me work, and making suggestions on which models and variables to use.

I am the only trainee at the moment. We’re working with a couple of others on my team. I have somebody building the data and somebody interacting on the real time component. So, Beyond the Arc is working with several others on my team, but I would say it’s 80 percent just me at the. Once I get comfortable, we’ll probably expand it out to a larger team.

You use IBM SPSS? Are there any other technologies used?

Correct. No, that is all.

Can you assess IBM SPSS’s effectiveness for your organization?

So far, it seems to be great. I don’t know if it’s better than other software for the statistical piece but it makes it a lot easier for data gathering, manipulation and cranking out multiple models. I haven’t used SAS, which I would say is probably its primary competitor. SPSS saved me from doing a lot of manual work that I had to do before with Microsoft Excel and other software packages.

What was your process for selecting Beyond the Arc to work with?

We started with IBM during the presales engagement, and brought in a contractor who I worked with for six months on the actual modeling. Instead of training me, they actually built the first set of models, put them in place, and later trained me.

That contractor didn’t work out as well, so I asked IBM for a replacement candidate targeted more at training me to do it as opposed to having a third-party group build the models for us. IBM recommended Beyond the Arc.

Can you provide a sense of the size of the initiative with Beyond the Arc in a dollar figure?

The overall initiative is roughly half a million dollars.

When do you think it will all be completed?

We will probably wrap up the work with them in January. I’ll probably have some kind of ongoing engagement five or so hours a month for a while. The bulk of the training will, hopefully, be done in the next couple of months.

RESULTS & FEEDBACK

Do you have any metrics or stats to track improvement?

Absolutely. Our chargeback rate is what we use as our primary indicator for this. I’m also looking at the percentage of orders that are scored as risky, and the accuracy of the scores. These three primary metrics are the main indicators of success.

How has Beyond the Arc performed so far?

So far, they’ve done great. They are all very professional, very responsive, and quite competent.

What’s really unique or special about them compared to other vendors?

I’ve really liked their technical lead’s approach. He’s been great to work with. I think it’s the staff that makes them stand out.

Is there anything that you think they should improve upon or maybe you would do differently?

Not yet. So far, I think they’ve done well. I would say the one thing that I could use, is more time. I would like more time face to face with their technical lead, Bruce. I want to bill more time with him, and have him full time here.