Snowflake

JULY 27, 2023

In this guest blog post, HomeToGo’s director of data, Stephan Claus, explains why the company migrated to Snowflake to meet its data needs. This article is based on Stephan’s presentation during the Snowflake Data World Tour 2022. Join us at this year’s world tour to learn more about the latest innovations to Snowflake’s Data Cloud.

Data Architecture 52

Data Architecture Architecture Building Structured Data 52

Data Engineering Weekly

MARCH 19, 2023

link] Uber: Setting Uber’s Transactional Data Lake in Motion with Incremental ETL Using Apache Hudi Uber writes a comprehensive guide on running incremental ETL using Apache Hudi. Hadoop put forward the schema-on-read strategy that leads to the disruption of data modeling techniques as we know until then.

Data Engineering 57

Data Engineering Data Engineer Engineering Media 57

Data Engineering Podcast

JUNE 22, 2021

We spend countless hours designing the data model, updating engine versions, and tuning performance. Andy Pavlo researches autonomous database systems, and out of that research he created OtterTune to find the optimal set of parameters to use for your specific workload. Then what do you do? Send a CSV via email?

Database 100

Database MySQL PostgreSQL Data Warehouse 100

Data Engineering Podcast

AUGUST 10, 2021

Summary All of the fancy data platform tools and shiny dashboards that you use are pointless if the consumers of your analysis don’t have trust in the answers. It builds your customer data warehouse and your identity graph on your data warehouse, with support for Snowflake, Google BigQuery, Amazon Redshift, and more.

IT 130

IT Building Data Warehouse Python 130

Propel Data

OCTOBER 23, 2022

Learn how to use the incremental model in dbt to manage data streams in your Snowflake warehouse.

Data Analytics 52

Data Analytics Building Data Management 52

Workfall

AUGUST 14, 2023

Two powerful tools that have emerged to simplify the management of data pipelines are DBT (Data Build Tool) and Airflow. In this blog, we will explore the benefits of using DBT and Airflow together and how they make data pipeline management easier than ever before. By combining DBT and Airflow , you get the best of both worlds.

Data Pipeline 62

Data Pipeline Raw Data Data Database 62

Data Engineering Podcast

JUNE 14, 2021

Summary When you build a machine learning model, the first step is always to load your data. To speed up the process, why not build the model inside the database so that you don’t have to move the information? But what about incremental sync, API quotas, error handling, and all of the other details that eat up your time?

Machine Learning 100

Machine Learning Database Data Warehouse Hadoop 100

Data Engineering Podcast

JULY 19, 2021

While those are useful tools, it is far better to prevent data errors before they become an outsized issue. It builds your customer data warehouse and your identity graph on your data warehouse, with support for Snowflake, Google BigQuery, Amazon Redshift, and more. Then what do you do? Send a CSV via email?

Management 100

Management Data Warehouse Data Pipeline Data 100

Data Engineering Podcast

JULY 12, 2021

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management You listen to this show to learn about all of the latest tools, patterns, and practices that power data engineering projects across every domain. What kinds of training have you found to be necessary/useful for those end-users?

Designing 100

Designing Data Warehouse Data Governance Consulting 100

Data Engineering Podcast

JULY 5, 2021

In this episode Bart and Rich Wood explain how SaaSGlue is architected to allow for a high degree of flexibility in usage and deployment, their experience building a business with family, and how you can get started using it today. This is a fascinating platform with an endless set of use cases and a great team of people behind it.

Systems 100

Systems Management Data Warehouse Programming Language 100

Data Engineering Podcast

JUNE 8, 2021

By making those pieces high quality and flexible they can be used in surprising ways that the original creators couldn’t have imagined. One such component that has gone above and beyond its originally envisioned use case is BookKeeper, a distributed storage system that is optimized for durability and speed. Then what do you do?

Data Warehouse 100

Data Warehouse Hadoop Metadata Architecture 100

Data Engineering Podcast

JULY 27, 2021

They explain how Pulsar is being used by their customers, the work that they have done to scale the administrative workload for multi-tenant environments, and the challenges of operating such a data intensive service at large scale. What are the core use cases that you are aiming to support with Astra Streaming? Then what do you do?

Building 100

Building Management Kafka Data Warehouse 100

dbt Developer Hub

FEBRUARY 7, 2022

You can follow along with Donny's data modeling technique for identity resolution in this dbt project repo. It includes a set of demo CSV files, which you can use as dbt seeds to test Donny's project for yourself. In short, a jaffle is: "A toasted sandwich with crimped, sealed edges. See above: Tasty, tasty jaffles.

Data Warehouse 52

Data Warehouse Datasets Data SQL 52

dbt Developer Hub

APRIL 19, 2023

Dimensional modeling is one of many data modeling techniques that are used by data practitioners to organize and present data for analytics. Other data modeling techniques include Data Vault (DV), Third Normal Form (3NF), and One Big Table (OBT) to name a few. This helps ensure that the modeled data is easily usable.

Building 145

Building PostgreSQL BI Database 145

Monte Carlo

JULY 27, 2023

We have already predicted that data teams would spend 30 percent more time on data platform cost optimization initiatives in 2023 and provided thoughts on 3 Snowflake cost optimization strategies that aren’t too crazy, but wanted to get additional insights across multiple platforms from some of data’s top leaders.

Data 52

Data Architecture Data Warehouse Data Engineering 52

Christophe Blefari

MARCH 15, 2024

I'd like to do a bit of user research about yato, if you consider using it drop me a message please. Databricks invests in Mistral AI — Mistral successfully positioned as the main OpenAI rival by being integrated in all major data platforms (Azure, Snowflake previously). I'm speechless. This is Croissant.

Metadata 272

Metadata Datasets Data Data Warehouse 272

Data Engineering Podcast

NOVEMBER 19, 2023

Summary The dbt project has become overwhelmingly popular across analytics and data engineering teams. Dustin Dorsey and Cameron Cyr co-authored a practical guide to building your dbt project. In this episode they share their hard-won wisdom about how to build and scale your dbt projects. What was your path to adoption of dbt?

Project 147

Project Data Lake High Quality Data SQL 147

Data Engineering Podcast

APRIL 9, 2023

When it was difficult to wire together the event collection, data modeling, reporting, and activation it made sense to buy monolithic products that handled every stage of the customer data lifecycle. pre-built dbt packages for common transformations, etc.) Can you describe what you mean by a "composable CDP"?

Data Lake 147

Data Lake Data Warehouse Machine Learning Data 147

Data Engineering Podcast

NOVEMBER 26, 2023

You should be able to keep the familiarity of SQL and the proven architecture of cloud warehouses, but swap the decades-old batch computation model for an efficient incremental engine to get complex queries that are always up-to-date. __init__ covers the Python language, its community, and the innovative ways it is being used.

Architecture 147

Architecture Data Lake High Quality Data SQL 147

dbt Developer Hub

AUGUST 17, 2022

When running a job that has over 1,700 models, how do you know what a “good” runtime is? While there are many possible answers depending on dataset size, complexity of modeling, and historical run times, the crux of the matter is normally “did you hit your SLAs”? The model fct_dbt_invocations takes, on average, 1.5 hours to run.

Data Warehouse 59

Data Warehouse Datasets Cloud Coding 59

Christophe Blefari

MARCH 8, 2024

While I'm not good at marketing it there is a few number of you using it every month but I think it could be used way more. Architecture I said it, while being a new feature to the blog this is as well an educational projet I can use to showcase technologies. The Explorer ([link] But I want to go further.

Data 130

Data Algorithm Database Education 130

dbt Developer Hub

NOVEMBER 3, 2022

Let’s discuss how to convert events from an event-driven microservice architecture into relational tables in a warehouse like Snowflake. We use Snowflake as our data warehouse where we build dashboards both for internal use and for customers. This data would become the main dbt sources used by our report models in BI.

Kafka 52

Kafka ETL Tools BI Database 52

Data Engineering Podcast

JUNE 28, 2021

It builds your customer data warehouse and your identity graph on your data warehouse, with support for Snowflake, Google BigQuery, Amazon Redshift, and more. With RudderStack you can use all of your customer data to answer more difficult questions and then send those insights to your whole customer data stack. Then what do you do?

Data Analysis 100

Data Analysis Scala Data Warehouse Kafka 100

dbt Developer Hub

NOVEMBER 28, 2021

Often, this is framed in terms of popularity and adoption - i.e., "how many people use our stuff" and "how many people love it". But there are tactical uses for this data as well. My models process the data so that it's easy to perform analysis and spot trends. In this post, I'll walk you through our stack and sources.

Raw Data 52

Raw Data Metadata Database Datasets 52

Data Engineering Podcast

OCTOBER 20, 2021

In this episode Boris Jabes, CEO of Census, explains how the work of synchronizing cleaned and consolidated data about your customers back into the systems that you use to interact with those customers allows for a powerful feedback loop that has been missing in data systems until now. Census supports incremental syncs from the warehouse.

Data Warehouse 100

Data Warehouse Data Lake Business Intelligence Data Engineering 100

DataKitchen

APRIL 13, 2021

We have also included vendors for the specific use cases of ModelOps, MLOps, DataGovOps and DataSecOps which apply DataOps principles to machine learning, AI, data governance, and data security operations. . Please let us know if we have forgotten anyone or if you have any comments (marketing@datakitchen.io). Process Analytics.

Consulting 123

Consulting Machine Learning Data Science Data Pipeline 123

phData: Data Engineering

JANUARY 3, 2022

Source Control Management Infrastructure as Code Build/Deploy Strategy Continuous Integration and Delivery (CI/CD) Data Quality and Validation Workflow Management Data Modeling Monitoring and Logging Business Continuity So How Do I Build a DataOps Strategy? How do I know where this data came from or how it’s being used?

IT 52

IT AWS Software Engineer Software Engineering 52

Monte Carlo

JANUARY 23, 2024

In 2024, data engineers can run big jobs very quickly thanks to the compute power of BigQuery, Snowflake, Firebolt, Databricks, and other cloud warehousing technologies. It used to be fairly cheap to run on-prem, but in the cloud, you have to be mindful of your compute costs ,” Maxime says. So, pick your poison?

Data Engineering 98

Data Engineering Data Engineer Engineering Data 98

phData: Data Engineering

JULY 18, 2023

While rarely a straightforward process, we’ve hit nearly every bump along the way and have learned so much, and we’re excited to share those insights with you! Business-Focused Operation Model: Teams can shed countless hours of managing long-running and complex ETL pipelines that do not scale.

Data Warehouse 52

Data Warehouse Pipeline-centric Government Data 52

Monte Carlo

MAY 17, 2023

Because the technology is so extensible, there have been a wide array of suggestions–some more grounded than others–for how it can be used. To help clarify, we reviewed hundreds of data observability deployments to identify 61 real data observability use cases and benefits. Table of Contents Core Data Observability Use Cases 1.

Data 52

Data Data Pipeline Data Engineering Data Engineer 52

Monte Carlo

MAY 17, 2023

Because the technology is so extensible, there have been a wide array of suggestions–some more grounded than others–for how it can be used. To help clarify, we reviewed hundreds of data observability deployments to identify 61 real data observability use cases and benefits. Table of Contents Core Data Observability Use Cases 1.

Data Pipeline 52

Data Pipeline Data Data Engineering Data Engineer 52

AltexSoft

MARCH 14, 2023

A Modern Data Stack (MDS) is a collection of tools and technologies used to gather, store, process, and analyze data in a scalable, efficient, and cost-effective way. Software engineers use a technology stack — a combination of programming languages, frameworks, libraries, etc. — to build products and services for various purposes.

IT 59

IT Data Warehouse Data Governance Data Lake 59