Leading National Retailer

About the Business

This national retailer is one of the world’s leading residential furniture producers, marketing furniture for every room of the home. The corporation’s branded distribution network is dedicated to selling products and brands, and includes approximately 350 stand-alone furniture gallery stores and more than 550 independent comfort studio locations, in addition to in-store gallery programs. For more than 90 years, the brand has focused on the core values of honesty and integrity.

The Business Challenges

The retailer struggled to integrate their data due to silos spread across different source systems and a newly acquired company. Data teams had to dedicate most of their time toward creating dashboards for business users that would only access single data sources. In addition, each data mart was provisioned for a specific business intelligence (BI) tool, such as Domo and Pentaho. These inefficient processes were compounded by unclear governance and undefined teams that made it difficult for analysts to collaborate on and share data.

Data teams lacked the centralized platform and tools required to fulfill use cases for demand forecasting, eCommerce integration, and MDM. Without sufficient internal DataOps expertise, the retailer required help to design, deploy, and implement the pipelines and platform necessary for data ingestion, curation, and consumption at scale.

The 2nd Watch Solution

After assessing the retailer’s existing infrastructure against their goals, 2nd Watch data management consultants focused on creating a solution that not only provided high-quality data sets for immediate use cases — demand forecasting, ML, eCommerce integration, and MDM — but also established a reusable, scalable data platform for future strategic initiatives. To achieve this, 2nd Watch leveraged a variety of modern tools based on cost, ease of use, and ability to satisfy workflow requirements.

The retailer needed a single tool for ingesting data to a Snowflake “raw” layer for staging. The 2nd Watch development team suggested HVR, a Fivetran company, to incrementally stream data from the enterprise resource planning (ERP) solution, customer relationship management (CRM) solution, and pre-defined SFTP (Secure File Transfer Protocol) flat files based on change logs. With HVR managing the majority of the ingestion work, users are able to train on one tool and monitor the active data flows using a single application.

After staging the data in Snowflake, our engineering team designed a “business” layer to gather all relevant information for a given domain on a per-source basis. For example, to collect customer attributes, we created multiple “CUSTOMER_SOURCE_N” tables to stage information before mastering the domain at an enterprise level using Ataccama. Using the mastered data, we integrated each data thread into a single source of truth designed as a star-schema data warehouse, which provides governed, secure, and consistent data sets to downstream users. To deploy transformation SQL scripts, job orchestration within Snowflake, and CI/CD data operations, 2nd Watch leveraged dbt, integrated with GitHub.

Finally, we strategically exposed the data sets to end users in a variety of ways, including Snowflake external shares, a Domo connector for BI, macro unloads to S3 buckets (and later SFTP folders), and Java-based APIs for application development.

The Business Benefits

With a fully developed data pipeline and unified data platform,
the retailer is able to take advantage of their data. Not only are data teams empowered to fulfill use cases, but they are doing it on a regular basis. Our client has developed a cadence for engineering and scaling based on new use cases, formalized data governance, and CI/CD best practices for the data ecosystem and DataOps. Now, teams participate in bi-monthly agile sprints for developing data features in the Snowflake data lake house that serve downstream business users and their needs.

The data lake house, enabled by the Snowflake deployment, gives the organization a raw and curated view of important business domains, including sales, supply chain, and customer information. With the ability to ingest, transform, and consume enterprise-level data in a centralized and digestible location, users can access, explore, share, and collaborate across teams to make smarter business decisions.

One of those use cases was to provide extracts for training and productionalizing o9 ML models for demand forecasting to better track inventory and drive manufacturing. The second use case was the deployment of an API for application integration for the eCommerce website. The API integration enables customers to view the status of their orders for improved customer service and experience. A third use case fostered an MDM feedback loop or domains that were mastered with Ataccama. This created one “golden” record per customer or other domain that was brought back into the data warehouse.

In total, the engagement between 2nd Watch and the retailer empowered their data team to be functional. Moving forward, they will continue to build on their capacity to generate and deliver insights within their unified platform. Ultimately, through collaboration, data exploration, sharing,
and analysis, current and future engineering can grow and scale with the interests of business users and the retailer’s overall goals.