Jul 22, 2023

Layered Data Architecture with Veronika Durgin - AgileData #44

In this episode of the Agile Data N’ Info Podcast, host Shane Gibson sits down with Veronika Durgin, an experienced data professional whose extensive career spans from database administration and performance tuning to modern Snowflake analytics and Data Vault modeling.

In this episode, the conversation explores the concept of a layered data architecture pipeline and the importance of an ELT (Extract, Load, Transform) approach. Veronika explains how her specific categorization of data into Bronze, Silver, and Gold layers operates as a flexible framework that makes data quality and clear team ownership first-class citizens. By treating raw, immutable source data as the Silver layer, using Data Vault for the modeled Gold layer, and reserving Bronze for decentralized analytics and discovery, this framework ensures that your data architecture remains highly scalable and clear enough to support true business agility.

The discussion also dives deep into the complexities of team design, clarifying the crucial role of keeping software engineers focused on building application capabilities rather than writing analytics. Veronika breaks down how adopting robust data modeling practices prevents data teams from falling into the trap of modern technical debt—like the dreaded 15-nested-CTE “spaghetti code” transformations. This approach achieves clear, governed data communication by definitely shifting the focus toward understanding core business definitions before serving it all up in a presentation layer, affectionately dubbed the “data buffet”.

Whether you are curious about how to weave a Bronze, Silver, and Gold data architecture into your everyday work, how avoiding custom-built ingestion engines can actually strengthen your pipeline, or want to learn why simple data policies are far more effective than heavy governance committees, this episode is packed with valuable insights for data practitioners

You can get in touch with Veronika via LinkedIn or over at https://thinksolverepeat.substack.com/

If you want want to read a summary generated with GenAI, head over to:

https://agiledata.substack.com/i/204924358/google-notebooklm-briefing

If you want to download the transcript for the podcast, head over to:

https://agiledata.substack.com/i/204924358/transcript

Listen to more podcasts on applying Agile Data patterns over at https://podcast.agiledata.io/

Read more on the Agile Data Way of Working over at https://AgileDataGuides.com/

If you just want to talk about making magic happen with agile and data you can connect with Shane @shagility on LinkedIn.

Buy the Green Book now!

Simply Magical Data Ways of Working

Comment (0)

No comments yet. Be the first to say something!