DS STREAM

BACKGROUND

Please describe your company and position.

I am the Data Science Manager of Kpler

Describe what your company does in a single sentence.

Bring data transparency into the Energy & Commodities trading market

OPPORTUNITY / CHALLENGE

What specific goals or objectives did you hire DS Stream to accomplish?

• The project aimed to optimize the Feature Store ingestion process, redesign its architecture, and enhance model execution workflows to improve efficiency, reliability, and scalability.

SOLUTION

How did you find DS Stream?

• Online Search
• Referral

Why did you select DS Stream over others?

Referred to me

How many teammates from DS Stream were assigned to this project?

1 Employee

Describe the scope of work in detail. Please include a summary of key deliverables.

Feature Store ingestion process optimization [July-August]
Review current how Feature Store ingests data from Production databases process and add documentation on Confluence. Documentation should include: [week 1]
Architectural design 
Interactions between Feature Store and other services if any
Add monitor on data ingestion steps, identify faulty parts and come up with suggestions about what can be improved [week 2]
Fix ingestion process to make sure no data is lost, errors and failures are managed [week 3-6]
Add unit tests on critical parts of the process
review database access (identify root cause of pool exhaustion), define solution
identify if leveraging pgbouncer would solve database pool problem
potentially collaborate with Kpler’s DevOps to setup read replica

Output:

process description / documentation, presentation to the crew
data ingestion improvement and optimization (potentially change of ingestion mechanism if necessary)

Feature Store redesign [August-September]
Review the Feature Store API using KAI and how the API is used by different models and add documentation
Now all the pipelines around Feature Store has been analysed, formulate proposals on what could be improved including architecture, technologies and different processes
Implement the redesigned system

Output:

process description / documentation, presentation to the crew
redesigned Feature Store

Analysis of the model execution [October/November]
Validate and document each stage of the model training and inference process
make sure that pods run by API are always traceable (logs are not lost due to OOM or unexpected pod disappearance)
make proposal about how to improve the current model training and retraining process

Output:

process description / documentation
strengthen database access, eliminate access problems

RESULTS & FEEDBACK

What were the measurable outcomes from the project that demonstrate progress or success?

Scalability: Redesigned Feature Store API for better performance

Data Processing Time: Reduced from 4 hours to 10 minutes

Reliability: Achieved zero data loss and improved log traceability

Database Performance: Resolved connection pool exhaustion

Describe their project management. Did they deliver items on time? How did they respond to your needs?

We have regular checkups and the deliveries are on time

What was your primary form of communication with DS Stream?

• Virtual Meeting
• Email or Messaging App

What did you find most impressive or unique about this company?

Agility and availability: the initial plan we had was subject to changes as things go, and they stayed available to listen to our needs and adjusted on the fly

Are there any areas for improvement or something DS Stream could have done differently?

Stronger expertise and force of proposal could have changed the way we collaborated: we had our idea of what we need and how things could be done, but we were all ears if DS Stream had different and better proposals especially about new technologies to introduce. 

BACKGROUND

Please describe your company and your position there.

I am a seasoned Product and Technology leader looking after the Sportside distributed development team.

OPPORTUNITY / CHALLENGE

For what projects/services did your company hire DS Stream?

We hired DS Stream to help us migrate our databases so they were scalable for 2 million plus users as well as implement business intelligence and data warehousing for our data.

What were your goals for this project?

The goals were to achieve scalability and visibility across all of our data.

SOLUTION

How did you select DS Stream?

DS Stream were highly recommended by a contact I have in the industry and they were invited to be a part of the RFP Process in which they were successful.

Describe the project in detail.

Sportside is very excited to have partnered with DS Stream. Sportside's plans for growth and expansion are being supported through serverless architecture, which is hyper-scalable, and our data has to scale in the same manner.

The specialists from DS Stream had to think outside the box, coming up with creative solutions and implementing cutting edge data solutions within our AWS infrastucture, simplifying complexity and making Sportside's stack more cost effective.

What was the team composition?

We initially were using 2 Cloud Data Specialists, with a team leader and a project manager. They worked within our current development team and liaised with other members of Sportside.

RESULTS & FEEDBACK

Can you share any outcomes from the project that demonstrate progress or success?

Our database is now resilient and scalable and we are confident the work completed will ensure Sportside can scale to 2m plus users.

How effective was the workflow between your team and theirs?

The workflow was brilliant. They worked within the existing team processes without any issues, and also helped us improve our database workflows through close collaboration with the Sportside technology leadership.

What did you find most impressive about this company?

Their ability to think outside the box, collaborate with an existing team and find the best solution for Sportside. They were very impressive with their thoroughness of research and approach to kicking off the project.

Are there any areas for improvement?

Working with the DS Stream team didn't highlight areas of improvement, rather they helped us grow and improve as a wider team. We see them as a partner not a supplier.

BACKGROUND

Please describe your company and your position there.

We are consultancy firm providing a comprehensive range of services and solutions in Data Warehouse & Business Intelligence platforms focusing on processing medical data.

OPPORTUNITY / CHALLENGE

For what projects/services did your company hire DS Stream?

This project enabled our customers to better planning resources and gaining higher profits from business activities due to use AIML algorithms.

What were your goals for this project?

To provide configuration and environment setup in private cloud on data from CRM and medical sources.

SOLUTION

How did you select DS Stream?

The solution was based on application consisting of several components developed in private cloud. The following tools were used: AWS, Triton, Oracle, Airflow, Hive, Hue, Sqoop, Docker, S3, EC2, Scala

Describe the project in detail.

In response to our needs, DS Stream provided a Data Engineering project in which data allowed machine learning analysis and creation of models to predict customer behavior. A team of qualified and experienced engineers were highly committed and listened carefully to the client's needs which has resulted in a productive cooperation.

What was the team composition?

We had about 4 people on contract.

RESULTS & FEEDBACK

Can you share any outcomes from the project that demonstrate progress or success?

The feedback was great. The project fulfil all clients requirements and there are planned new phases to further extend functionalities of this solution.

How effective was the workflow between your team and theirs?

We had a team of experienced people working with us, who are perfectly familiar with communicating with the client and responding quickly to all requests

What did you find most impressive about this company?

Their commitment, knowledge, and very good communication resulted in high performance and a comfortable work atmosphere.

Are there any areas for improvement?

No, I can’t think of any, which is unusual.

BACKGROUND

Introduce your business and what you do there.

I’m the CEO of a sports analytics company.

OPPORTUNITY / CHALLENGE

What challenge were you trying to address with DS Stream?

We were looking for a consulting partner to provide us with guidance and implementation of workflow scheduling service for multiple data processing pipelines. Since we don’t have experience with these technologies, we needed a partner to help us choose the right solution for our needs.

SOLUTION

What was the scope of their involvement?

It wasn’t a very big project, probably more of a consulting gig or a quick prototype. Initially, we reviewed several possible workflow scheduling tools. DS Stream recommended a powerful open-source tool that was developed by a high-traffic online marketplace.

After looking at our requirements for pipeline processing, they designed the most critical parts of the scheduling. They also implemented some of the pipelines.

What is the team composition?

We worked with 2–3 people from their team, including a project manager.

How much have you invested with them?

We spent less than $10,000 on their services.

What is the status of this engagement?

We worked together from July–August 2019. We’re considering hiring them for our next project.

RESULTS & FEEDBACK

What evidence can you share that demonstrates the impact of the engagement?

This project has increased our efficiency and reduced the number of data processing failures. Previously, we were running paper pipelines semi-automatically. Now, we’ve managed to automate 90%–95% of the work, so we don’t need to spend as much time running it.

How did DS Stream perform from a project management standpoint?

It was fine. They delivered what we expected. We didn’t have any issues.

What did you find most impressive about them?

They have excellent technical specialists compared to other companies. The people we work with are quite similar in terms of seniority. It wasn’t like with other companies where you get one or two juniors and a senior. We got a highly experienced team from day one.

Are there any areas they could improve?

No, everything was okay.

Do you have any advice for potential customers?

It’s worth clarifying the requirements and the expected results from the beginning. Let them work on the best solution. Don’t provide your ideas for a solution because you may get something better because they’re quite experienced.