Optimized Data Pipelines

The Overview

The client is a mass media company with a forward-thinking technical strategy. Despite a robust commitment to technology, they were losing competitive advantage by not taking control of the data these systems produced. 2nd Watch helped them identify useful data assets and built a cutting-edge analytics platform that solved these problems.

The Business Challenge

2nd Watch partnered with the client to assess their current Ad Proxy environment to address visibility and performance issues that have limited their ability to gain additional Advertising revenue opportunities.

The client was facing a potential problem of timely scalability and performance during high peak intervals in their Ad Proxy architecture. The application is designed to use Google Ad Manager as the primary ad server and send simultaneous requests to Amazon Ad Publishing service, Publica, and Springserv.

2nd Watch assessed the current architecture against KPI’s and performed an Observability Health Check to understand the internal state of the environment based on its external outputs and behavior, such as logs, metrics, traces, and events. An Observability Health Check typically involves reviewing the observability architecture and infrastructure, and the instrumentation and monitoring practices of the system or application. The goal of this process is to identify gaps in observability that may be preventing teams from detecting, diagnosing, and resolving issues quickly and effectively.

The 2nd Watch Solution

2nd Watch partnered with the Client to remediate and refactor the Ad Proxy environment through:

1. Increasing visibility of the existing architecture to provide granular and attributable reporting.
2. Building a Proof of Concept (POC) architecture that targets increasing visibility and to improve performance through implementing burstable events with increased exposure.
3. Building an automated curation ETL to scrub sensitive data and provide a concise and standardized format.
4. Providing a performance reporting dashboard for on-going reporting and analytics.

How AWS was Part of the Solution

2nd Watch provided insights into the existing solution by adding an Ingress Controller to their AWS EKS cluster which facilitates OSI Layer 7 visibility into requests. We then created a Data Lake backed by Amazon S3 and exported log data to the Raw Zone of a Data Lake. We created AWS Glue jobs to perform curation ETLs and landed desensitized and curated data in the Curated Zone. Finally, data was ingested into Snowflake where it can be aggregated and modeled to facilitate reporting of key KPIs in Power BI. These capabilities enable the customer to leverage complete visibility to understand application behavior and capture lost opportunities while maximizing overall performance.

The POC architecture is provided in two primary models: Multi-region AWS EKS and Amazon CloudFront + AWS Lambda@Edge. The former increases availability while also reducing latency, key factors in revenue potential. We recommended an AWS Global Accelerator in combination with deploying multiple AWS EKS clusters across target regions based on request sources. The latter maximizes this potential through serverless edge computing across multiple regions across the nation and that offers scale and performance while operating at the unit-level – offering even greater visibility and control.

The Outcome

We provided automated delivery of managed services and capabilities for on-going analysis and data engineering efforts. We increased visibility into existing products resulting in better business performance metrics. We provided multiple options for future state architecture resulting in the company having the ability to quickly pivot, as needed.

Lessons Learned

The client is still early in their cloud adoption journey having come from Roku only a few years prior to our engagement. Throughout their experience, they continue to grow and learn as new challenges arise and additional information is gathered. Observability was a key component missed in the initial stages of development of this architecture and the insights gleaned from newly available data have opened many opportunities for technical and business-focused teams alike.

About 2nd Watch

2nd Watch’s team of experts works with global brands in financial services, retail, fast-moving consumer goods, media and entertainment, manufacturing IT, healthcare, and education. We have been an AWS Premier Consulting Partner since 2012, an audited Managed Service Provider since 2015, and an AWS Training Partner. We currently hold the AWS DevOps, Data and Analytics, Financial Services, Microsoft Workloads, Cloud Operations, and Migration and Migration Delivery Competencies.