Snowflake Framework
Our framework has been used by numerous customers to reduce delivery time of new data warehouses.
Description
With more than 20 years’ experience in Data and Analytics, we know how to help organizations make better decisions with their data.
The Altis metadata driven Snowflake framework provides configuration driven processes to:
- Land data from source
- Run change data capture (CDC) on source data
- Load Type 1 and 2 dimensions and facts
Provided the schema of the source table/view and target table are the same, the framework will dynamically generate SQL to calculate changes between the source and target, expire old records in the target table and insert new records.
Where definitions of the source table/view are available, warehouse tables to store the historicised data will be automatically generated. New columns added to the source will also be automatically added to the target table.
Our framework has been used by numerous customers to reduce delivery time of new data warehouses. Once setup, the framework has allowed them to schedule and load hundreds of tables from source systems into their Snowflake warehouse through a once off configuration. Data mart creation time has also been reduced to a simple configuration process once the business logic has been signed off.
Features
- Metadata driven change data capture (CDC) and append processes
- Dynamic SQL generation for CDC and append processes removing the need to code individual ETL jobs to load warehouse tables
- Auto-generate warehouse tables based on table definitions from source systems. Add new columns as they get added into the source system.
- Custom metadata driven jobs for other repeatable processes can be created
- Conforms to industry best practice for ETL solutions
Example
Click here to read how Altis helped BRG deliver a metadata-driven ETL framework:
Webinar: Altis Snowstream – Stream Data Ingestion for Snowflake
Join the team of Altis Data Engineers behind “Altis Snowstream” for a webinar. Altis Snowstream is a purpose-built tool for ingesting and integrating streaming data into Snowflake with CICD support. The tool is built using native Snowflake SQL components.
Watch the webinar recording here.
