It’s true that in a very short period of time, you can get an operational data warehouse loaded with all your source data. Wondering how to build a data warehouse that yields these results? Snowflake and its technology partner ecosystem, such as partners like Fivetran, allow you to take your databases and SaaS applications, replicate them to Snowflake, and see results in a significantly shorter timeframe than we’ve seen before. Write some SQL views in Snowflake against this data, point your favorite BI tool at it, and get lightning-fast results.
With the right plan in place, you can (aggressively) deliver that first business “wow” in six to eight weeks. 2nd Watch typically recommends at least two analytical virtual warehouses in enterprise deployments: one for data discovery and one for more structured and governed reporting.
The practice of having both a data discovery layer and a governed layer for your Snowflake deployment not only saves you time in the initial build, but it creates a continuous integration and deployment pattern. It finally makes a dent in the notion that a data warehouse cannot be “agile.”
Future State Snowflake Data Architecture
Future State Snowflake Data Architecture
With this approach, you not only achieve governance and speed-to-insight, but you’ve also cut down your Snowflake consumption costs. Running complex queries at run-time, every time, can get expensive. Snowflake’s caching can help here, but if you’re constantly running a complex join across over 20 tables, that physical layer might help you move from an XL-sized Snowflake warehouse to a L or M. In the long run, those cost savings will add up. When “best practice” or risk avoidance isn’t enough to justify this approach, it’s possible the dollar savings might speak for themselves.
The initial six- to eight-week path assumes a focus on the data discovery layer, as depicted below. Loading all your data into a data discovery layer should be the primary development activity in your Snowflake proof of concept (POC) or pilot. Here are some tips:
- Find several source systems that have the data your stakeholders need.
- Begin the process of rapidly loading into your Snowflake data discovery layer.
- Write iterative SQL in Snowflake views to build your business logic.
- Connect your BI tool to Snowflake and build a very simple dashboard.
- Get feedback from business, with the new dashboard as a facilitation tool.
- Repeat this process – more feedback is better.
You’ve not only increased the speed to market, but you’ve also enabled developers and business users to execute an analytics project in a completely new way.
Highlighted: Speed-to-insight approach, with transformations in Snowflake
Here are some additional considerations to keep in mind:
Security Considerations: It’s important to remember that when creating a data warehouse, data security and compliance can’t be overlooked. Using Snowflake and Fivetran, we ensure robust security protocols are in place, safeguarding your sensitive data while complying with necessary regulations.
Data Quality: Another critical aspect of this process is guaranteeing the quality of the data. By employing rigorous data quality checks and validation methods during the loading phase, we ensure that the data in your Snowflake data discovery layer is accurate, consistent, and reliable.
Scalability: Our approach to data warehousing is not only efficient but also scalable. As your business grows and the amount of data increases, Snowflake’s cloud-native architecture can easily adapt to your expanding needs while maintaining cost savings.
Training and Support: Adapting to a new data warehouse can be a challenge, but our team provides comprehensive training and continuous support to ensure a smooth transition. We’re here to guide you through every step of the process, from initial setup to advanced data analytics.
Alternative Tools: While we prefer Snowflake for its efficiency and scalability, there are other technology options available, like Redshift and BigQuery. However, Snowflake stands out due to its unique capabilities, such as separate storage and compute resources, robust security, and excellent performance.
ROI and Performance Metrics: After setting up your data warehouse, we’ll provide you with tools to measure its performance and the return on your investment. We’ll help you monitor query performance, storage utilization, and other key metrics that align with your business goals.
Integration with Existing Systems: We also take care of integrating Snowflake with your existing systems and workflows. Our team will devise a plan to ensure minimum disruption to your business during this integration phase.
This all comes with a caveat: Yes, you can write all your logic in Snowflake and do everything you need from an analytics standpoint, but it will limit you in the long run. Every data project has sacrifices to be made, whether it’s time, features, cost, or flexibility. You need to balance these with a long-term vision for how your data will be used across the organization. Snowflake will get you part of the way there, but a good plan and strategy will take you even further.
In the meantime, if you have any questions or want to learn more about how to build a data warehouse using Snowflake, contact us today.
