A Modern Data Ecosystem

Key Facts

Brands like New Balance, TaylorMade, and Audi spend millions of dollars each year to sponsor top athletes. However, calculating the ROI of those sponsorships can be a major challenge for the sponsors. Our client is changing that through their sponsorship analytics platform. This platform enables sponsors to analyze the effectiveness of their sponsorships using social and digital media data.

When our client wanted to centralize their data, run more accurate reports, and reduce back-end operating costs, they turned to 2nd Watch for help.

The Business Challenges

The front end of our client’s analytics solution provided their customers with highly valuable access to robust analytics that enabled them to understand the value of their sponsorships.

However, the back-end architecture was costly, inefficient, and prone to failure. The SVP of technology’s vision was to improve the current architecture while reducing costs and supporting future growth.

 

Project Goals

• A modern data architecture that would fix data movement errors and historical reporting issues, as well as support future growth.
• Evaluation of current tools and technologies to find opportunities for optimization and cost savings.
• Optimization of current Snowflake and ETL architecture to provide future cost savings.
• Creation of a data platform foundation for reporting and advanced analytics.

The Solution

The approach to this project was two-fold:

• Step 1: Create a data strategy based on our client’s current state and future goals.
• Step 2: Build a scalable modern data ecosystem with best-in-class technologies and custom applications.

Step One: Data Strategy – Understanding the Current State and Desired Future State

Working closely with the SVP of technology, the 2nd Watch team conducted interviews with stakeholders throughout the organizations to:

• Understand their current challenges and pain points.
• Review their current data architecture and tech stack.
• Establish their near-term and long-term goals.

The outcome was a modern data architecture strategy that outlined:

• The current and future needs of both technical and business users.
• The current state of their data architecture.
• Recommendations on current and new technologies.
• Recommended future state architecture.
• Short-, mid-, and long-term recommendations.
• A high-level road map to achieve their goals.
• Expected ROI of the project.

Step Two: Building the Solution

The client leveraged 2nd Watch’s team to build a solution to replace costly migration tools by implementing a multipurpose container environment leveraging AWS Step Functions and Fargate for orchestration. The low-cost solution was ideal for running data loads in parallel, which improved data refresh rates and the replacement of data migration tools.

In addition, the new design centralized the disparate source data into a centralized data warehouse. This provided a way for the client to see how their reports and data changed over time (“Why did my valuation go down?”). We did this by loading copies of the source data into a raw layer in Snowflake. We then modeled the raw data into a second layer optimized for analytics.

Finally, 2nd Watch assisted in optimizing the data warehouse in Snowflake. We implemented a change data capture approach to streamline data updates and reduce costs versus the previous truncate and load method. These optimizations decreased costs and increased concurrency for data loads. Additionally, we streamlined and simplified the reporting queries to meet the end consumers’ expectations.

The Outcome

Over the course of three months, the 2nd Watch team delivered an enterprise-ready data solution that:

• Migrated data from numerous sources into a centralized cloud-based data warehouse.
• Streamlined cumbersome processes resulting in more efficient and reliable operations.
• Replaced the data migration tool with a custom solution that resulted in an estimated annual savings of $24,000.
• Implemented a data architecture that allows the client to sunset existing infrastructure and gain future cost savings of hundreds of thousands of dollars.
• Enabled auditing and historical views to make the data movement process more robust and reduce downtime due to errors.
• Laid the foundation for future growth and advanced solutions such as machine learning.

The Result

The 2nd Watch and client team worked side by side to build a custom ETL solution with container orchestration using AWS Step Functions and Fargate. The solution allowed the client to address a costly component of their current architecture that resulted in an immediate annual savings of $24,000, as well as reducing errors caused during the data movement process. In addition, after sunsetting other components of their existing architecture, it is expected that the client can gain additional cost savings of hundreds of thousands of dollars a year.

Before:

After:

The solution we implemented resulted in immediate cost savings, faster processes, and more reliable access to data.

Our client’s new data ecosystem will provide ongoing cost savings and enable their front-end analytics platform to deliver powerful analytics and insights more efficiently.