You are here

Cloud Services

Interview with TWG on Choosing the Best Cloud Platform

 
Clutch spoke with Derek Watson, the CTO of TWG, about advising clients when migrating to the cloud, finding the best cloud and analytics platforms, and security on the cloud.
 

Learn more about TWG on their Clutch profile or at twg.io.

Background

Can you provide a brief description of your company and the role that you play there?

TWG is a software development agency. Our mission is to help our clients achieve their missions. Whatever their business objectives and goals are, it’s our job to help them fulfill that mission through software. My role here is CTO, and I’ve been with the company for about six years. I help to lead our engineering team of around 55–60 software developers. I help architect some of our solutions and work with a lot of our clients in the early stages of their relationship with TWG. I put teams together internally and make sure that we’re delivering high-quality and high-value solutions for our clients.

Often, companies come to us with a new business idea, and we help them with strategy, product design, market testing, and user research to validate that business idea, and then, we’ll design and build the software product. The end product is often web or a mobile application on top of a backend data platform in the cloud. This is the typical story of a company that wants to bring a new product to market. On our website, you can see a short video of our approach to this type of product development.

On the other hand, a lot of our clients are enterprises that have large technology investments but need to move into the future and modernize their code base and product and the way that they host and deliver them. We also help this type of client. 

Challenge

What are some challenges that companies face when migrating to the cloud?

In a lot of cases, there aren’t a lot of challenges. This is the case when the client type is a more modern company that wants to start a technology platform from scratch. We will recommend putting a lot of their services in the cloud and that is fairly non-contentious. We run into challenges with modernizing companies that have more established technology footprints, like a company that’s been investing in software or IT for a long time. There are a few things that come up:

1. Budgeting

The first challenge is perhaps one of the simpler ones, but it’s actually quite tricky. The nature of dynamic, hourly billing is a big mental shift for a lot of people who manage IT budgets internally. It’s hard to predict costs when they’re tied directly to load and demand. However, there are savings and performance gains when operating a budget that way, so we strongly encourage our clients to try and become comfortable with this type of budgeting.

2. Security perception

Another challenge that comes up almost all the time is security. This is a concern particular to clients who have an internal IT department and their own server closet, whether it’s on-premise or in a shared data center somewhere in their city. The feeling that companies have is that there is a degree of security when infrastructure is at home. It’s in their basement, on a server, and they know who their IT person is, and they can talk to that person. But, if we talk to them about putting things in AWS and in Google Cloud, fear creeps in. The perception of security in the cloud is a real issue.

3. Data residency

Another challenge is data residency. We operate in Canada and some of our clients are in Canada, but a lot of our hosting options are in America. But, that’s improving. This year, an AWS data center opened in Quebec. Some clients are concerned with privacy laws, especially regulated industries such as insurance or medicine which have regulations for data privacy and data residency. Differing laws state by state and province by province complicate the matter. 

4. Shift in responsibility

The final challenge is a shift in work roles. For a client who has an existing IT department and a server closet which they’ve been operating themselves, we take some of that responsibility away from that team and put it in the cloud and ask that they rely on managed services. We find that paying cloud providers is more cost-effective, more reliable, and probably better performing. But, that takes budget away from a team that’s been investing in their own capabilities and technology, probably for many years. That can be emotionally difficult to deal with so we need to put care into those conversations.

Platforms

Do the companies usually come in with preferences of what cloud platform they want to use?

We have our own preferences, but in almost every case, our recommendations are also the most obvious choice that our clients would make on their own. There are three cloud platforms that we look at over and over again: AWS, Google Cloud, and Azure. From our perspective, AWS is the market leader. Many clients expect to be on AWS. Over and over again, the conversation naturally goes there. So, for most of our clients, AWS is the choice. 

If Google Cloud continues at the rate of their growth and market penetration, they will be neck and neck, or even beyond where AWS is from a popularity perspective for putting new deployments into the cloud. The reason we choose Google Cloud from time to time is their approach to hosting dockerized applications and Kubernetes [Google product for orchestrating containers in the cloud]. It is the best in class. When we rely on Docker technologies, we sometimes prefer Google Cloud. We also prefer Google Cloud for machine learning because the way that they make TensorFlow [open-source machine learning framework] services available is unparalleled. 

Microsoft Azure comes up a lot too. If a company is already working with Microsoft technology, that becomes the number one choice pretty instantly. We find that Azure is great for Windows and .NET hosting.

When advising on a cloud platform, we also consider if the client has a business relationship, partnership, or hosting credits with a company. Sometimes Google Cloud or Azure will provide a free tier of hosting beyond what the public gets for special programs for entrepreneurs and small businesses. There are incentives for them to host a considerable amount of technology on a special free tier. Those incentives often are very worthwhile for our clients so we’ll make technology recommendations that fit those business incentives.

Are there any platforms you think could be improved upon?

While Azure is the best in class for Windows-based hosting, it is not as good as the other two for open source projects, Node applications, Java applications, and especially Linux-based platforms. The platform has room for improvement in terms of serving the open source community, although it’s in their brochure, so I trust that they’ll get there. 

It’s hard to criticize Google Cloud. They just need more popularity and mindshare. Once they are more of the recommendation on the tip of people’s tongues, they’ll be in a better place. 

AWS is great, but there are a lot of criticisms that I could level against the confusing array of options that they give their customers. There are 12 ways to do everything on AWS and the user interface is byzantine and confusing. But, it’s a known quantity, and once you get used to it, you can see the forest for the trees. 

Hilariously, there is a webpage called “Amazon Web Services in Plain English” which gives all of the different AWS services a more understandable name. It is kind of a joke website, but I refer people to it when they’re trying to get their heads around AWS. Amazon is scrambling to provide more modern machine learning options, but they are quickly catching up on their ability to provide open-ended neural networks. 

What are some of the factors that companies need to consider when selecting a storage platform?

There are so many products available these days. There are a lot of alternate databases, products, and choices. We strongly encourage classic relational databases such as PostgreSQL, Microsoft SQL Server, or Oracle. Those are traditional choices for the heart of the applications that we build for our clients, and they are very reliable and consistent databases for companies’ transactional workflow. 

The biggest factor is the nature of the data and the workflow around the data. It could be transactional, which is to say that the data will power your application on a daily basis. For example, if users are going to be putting data records in, your administrators are going to be getting them out and operating your business on top of this data. For transactional data, we like to choose traditional relational databases. But, on the other hand, when we have analytical workflows, it’s very important to use databases optimized for analysis. 

What are some of the factors that companies need to consider when selecting a storage platform?

This is something that’s really interesting to us because we ourselves need business intelligence (BI). We have a very complicated business model and work with many different clients and a large team that is all dynamically working on different things, so we had to get good at BI ourselves.
 
When we consult with our clients on BI, we recommend the cloud for almost everything. The first thing you need to do is get your data into a place where you can analyze it. In the 80s and 90s, there were tools called ingestion pipelines and ETLs that were very popular for extracting your data, transforming it, and loading it into a data warehouse so it can be analyzed. These days, the ingestion pipelines and ETLs are in the cloud. For example, AWS has a product called Kinesis that is excellent at ingesting data if you have a constant stream of events coming out of your application or a large number of users all speeding data up into one place. Another product, Glue, is more of a traditional ETL in the cloud service that allows a bunch of data sources to be transformed and aggregated into a central warehouse for analysis. For storing bigger sets of data and doing more complicated queries, we often recommend AWS Redshift and Google BigQuery.
 
Some of our clients have greater data processing requirements, such as a client building a recommendation engine or something with a machine learning component. In these cases, the database needs to be constantly analyzed and data pulled and processed, and then the results have to be put back into either the transactional database or the analytical database. For this, we have used tools like Spark and Hadoop and AWS services like Elastic MapReduce to help us with the Always On workflows. Being in the cloud, analytics work scales up to the demand that you have. From a cost perspective, it’s really effective.
 
Finally, there are visualization tools such as Tableau, Chartio, Periscope, and Looker. It is worth noting that these are applications that used to run on your Windows desktop ten years ago, and the company finance team would have to go through charts and export PDFs to email them around. Today, these are all software-as-a-service web applications and entire companies think about KPIs and look at dashboards. To become a data-driven company, everyone needs access. Using cloud web applications allows for this and can create that culture of data awareness. BI tools being moved to the cloud has been a really great shift.

Security

What should companies take into consideration in terms of security on the cloud?

Nothing is perfectly secure and I would never say a silver bullet exists in terms of security because there is always a degree of risk. But, in almost every case, the perceived risk of moving to the cloud is greater than the actual risk.

It is almost always true that a cloud provider like AWS is tremendously more secure than what a client had in their own IT department. The kind of security practices, ISO certifications, infrastructure, and equipment that cloud providers have in their data centers is typically lightyears beyond what an individual organization and IT department is able to achieve. 

To illustrate physical security, we ask questions like: “Can I walk into the server room?” “Can I grab a hard drive?” “Can I plug in a USB key?” “Does your IT manager have it on a portable hard drive at their house?” The answers to those questions are often “Yes.” It’s a perceived security that companies feel when they operate their own IT that is often dwarfed when you look at what AWS or another service provides. The cloud starts to look very attractive once you start looking at it and comparing to what you might have in your own server closet