Monte Carlo

JANUARY 25, 2023

In this article, Chad Sanderson , Head of Product, Data Platform , at Convoy and creator of Data Quality Camp , introduces a new application of data contracts: in your data warehouse. In the last couple of posts , I’ve focused on implementing data contracts in production services.

Data Warehouse 52

Data Warehouse Data High Quality Data Metadata 52

Data Engineering Podcast

OCTOBER 14, 2019

Summary Managing a data warehouse can be challenging, especially when trying to maintain a common set of patterns. What are some of the challenges and mistakes that are common among engineers and analysts with regard to versioning and evolving schemas and the accompanying data?

Data Warehouse 100

Data Warehouse PostgreSQL AWS Programming Language 100

Grouparoo

JANUARY 11, 2022

This article looks at the options available for storing and processing big data, which is too large for conventional databases to handle. There are two main options available, a data lake and a data warehouse. What is a Data Warehouse? What is a Data Lake?

Data Lake 52

Data Lake Data Warehouse Unstructured Data Raw Data 52

Monte Carlo

APRIL 16, 2022

The data warehouse is the foundation of the modern data stack, so it caught our attention when we saw Convoy head of data Chad Sanderson declare, “ the data warehouse is broken ” on LinkedIn. Treating data like an API. Immutable data warehouses have challenges too.

Data Warehouse 52

Data Warehouse Architecture Data Government 52

Netflix Tech

DECEMBER 21, 2020

By Anupom Syam Background At Netflix, our current data warehouse contains hundreds of Petabytes of data stored in AWS S3 , and each day we ingest and create additional Petabytes. Some of the optimizations are prerequisites for a high-performance data warehouse. that got added and deleted in the commit.

Data Warehouse 81

Data Warehouse Metadata Algorithm Data 81

Christophe Blefari

MARCH 1, 2023

dbt Core is an open-source framework that helps you organise data warehouse SQL transformation. dbt was born out of the analysis that more and more companies were switching from on-premise Hadoop data infrastructure to cloud data warehouses. This switch has been lead by modern data stack vision.

Data Warehouse 278

Data Warehouse SQL Metadata Raw Data 278

Data Engineering Podcast

FEBRUARY 18, 2024

Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). To start, can you share your definition of what constitutes a "Data Lakehouse"?

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Christophe Blefari

JANUARY 20, 2024

Data engineering inherits from years of data practices in US big companies. Hadoop initially led the way with Big Data and distributed computing on-premise to finally land on Modern Data Stack — in the cloud — with a data warehouse at the center. Who are the data engineers?

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

ProjectPro

AUGUST 11, 2021

“Data Lake vs Data Warehouse = Load First, Think Later vs Think First, Load Later” The terms data lake and data warehouse are frequently stumbled upon when it comes to storing large volumes of data. Data Warehouse Architecture What is a Data lake?

Data Lake 52

Data Lake Data Warehouse Cloud Hadoop 52

Knowledge Hut

APRIL 19, 2023

Step 3: Cleanse data: The extracted data is then cleaned to remove inconsistencies, errors, and duplicates from the given dataset Step 4: Combine data: The cleaned data is then combined into a single location, such as a data warehouse or a data lake.

Process 59

Process Data Mining Aggregated Data Portfolio 59

Monte Carlo

AUGUST 30, 2023

Your test passes when there are no rows returned, which indicates your data meets your defined conditions. Often, teams run custom data tests as part of a deployment pipeline, or scheduled on production systems via job schedulers like Apache Airflow, dbt Cloud, or via in-built schedulers in your data warehouse solution.

SQL 52

SQL Datasets Database High Quality Data 52

Lyft Engineering

OCTOBER 3, 2023

We will then introduce our in-house product, Verity, and showcase how it serves as a central platform for ensuring data quality in our Hive Data Warehouse. In future posts we will discuss how Verity addresses data quality elsewhere in our data platform. What and Where is Data Quality?

Big Data 84

Big Data Metadata Data Warehouse Data 84

ProjectPro

DECEMBER 7, 2021

Generally, data pipelines are created to store data in a data warehouse or data lake or provide information directly to the machine learning model development. Keeping data in data warehouses or data lakes helps companies centralize the data for several data-driven initiatives.

Data Pipeline 52

Data Pipeline Architecture Kafka AWS 52

Cloudera

DECEMBER 12, 2018

After having rebuilt their data warehouse, I decided to take a little bit more of a pointed role, and I joined Oracle as a database performance engineer. I spent eight years in the real-world performance group where I specialized in high visibility and high impact data warehousing competes and benchmarks.

Data Warehouse 42

Data Warehouse Relational Database Hadoop BI 42

Data Engineering Podcast

FEBRUARY 19, 2023

Summary Cloud data warehouses have unlocked a massive amount of innovation and investment in data applications, but they are still inherently limiting. Because of their complete ownership of your data they constrain the possibilities of what data you can store and how it can be used.

IT 147

IT Data Lake Metadata Data Warehouse 147

Monte Carlo

AUGUST 9, 2023

Integration tools (Fivetran, Matillion) help extract data from internal and external sources and load it into a data warehouse. Teams also use Git to collaborate, perform code reviews, use branches to segregate work, and automate builds and tests through pipeline integrations.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Pipeline 52

Advancing Analytics: Data Engineering

JUNE 21, 2019

Hear me out – back in the on-premises days we had data loading processes that connect directly to our source system databases and perform huge data extract queries as the start of one long, monolithic data pipeline, resulting in our data warehouse. Till next time.

Data Warehouse 52

Data Warehouse Business Intelligence Data Data Validation 52

Airbnb Tech

NOVEMBER 28, 2023

However, for all of our uncertified data, which remained the majority of our offline data, we lacked visibility into its quality and didn’t have clear mechanisms for up-leveling it. How could we scale the hard-fought wins and best practices of Midas across our entire data warehouse?

Data Warehouse 98

Data Warehouse Metadata Data Certification 98

Data Engineering Weekly

MAY 26, 2024

The idea will infact open flexibility in programming data over the SQL-based data warehouse systems. link] Jonathan Merlevede: Quack, Quack, Ka-Ching - Cut Costs by Querying Snowflake from DuckDB Many data pipeline process incremental data, which is not a lot in many business cases.

Data Engineering 57

Data Engineering Data Engineer Engineering Google Cloud 57

Data Engineering Podcast

JANUARY 7, 2022

In this episode Metaplane founder Kevin Hu shares his working definition of the term and explains the work that he and his team are doing to cut down on the time to adoption for this new set of practices. Are you bored with writing scripts to move data into SaaS tools like Salesforce, Marketo, or Facebook Ads?

BI 100

BI Data Warehouse Metadata SQL 100

Data Engineering Podcast

NOVEMBER 14, 2021

Datafold also helps automate regression testing of ETL code with its Data Diff feature that instantly shows how a change in ETL or BI code affects the produced data, both on a statistical level and down to individual rows and values. The market for "data quality tools" has been growing and gaining attention recently.

Data Warehouse 100

Data Warehouse BI Data Workflow Data 100

Data Engineering Podcast

DECEMBER 28, 2022

Datafold shows how a change in SQL code affects your data, both on a statistical level and down to individual rows and values before it gets merged to production. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows.

Data Lake 130

Data Lake Data Warehouse Data Pipeline MongoDB 130

Striim

JULY 7, 2023

Sarah Krasnik · Published in Towards Data Science · Oct 7, 2022 Photo by American Public Power Association on Unsplash Co-written with John Kutay of Striim Data warehouses emerged after analytics teams slowed down the production database one too many times. Data warehouses optimize for precisely this using OLAP.

PostgreSQL 52

PostgreSQL MySQL Data Warehouse Data Pipeline 52

Snowflake

NOVEMBER 21, 2023

Now a household name, the company is on a mission to become the definitive online food provider, and its services already cater to 10 international markets across Europe, the Middle East, and Asia. Deliveroo uses Snowflake to ingest all of its important business data. “We

Cloud 85

Cloud Food Data Warehouse Data 85

Towards Data Science

NOVEMBER 4, 2023

Often it is a data warehouse solution (DWH) in the central part of our infrastructure. Data warehouse exmaple. It’s worth mentioning that its data frame transformations have been included in one of the basic methods of data loading for many modern data warehouses. Consider this example below.

Data Engineering 83

Data Engineering Data Engineer Engineering BI 83

ProjectPro

MAY 13, 2016

As the appetite for Hadoop and related big data technologies grows at an exponential rate, it is not out to spell the death of data warehousing. Data warehousing as a technology is evolving. Data Warehouse – Decide Which One to Use When Hadoop vs DataWarehouse Shocking Headlines like “Is data warehouse dead?”,

Data Warehouse 40

Data Warehouse Hadoop Unstructured Data Big Data 40

Christophe Blefari

MAY 18, 2023

uMetric manages definition, discovery, computation, verification and serving. Writing unit test for data science — Pragmatic guide about unit tests. Retro on data science by DJ Patil — DJ Patil has been US Chief Data Scientist. The eng - director gap problem.

Data 130

Data BI Consulting Data Science 130

Data Engineering Weekly

APRIL 14, 2024

The highlight of the design is the unique blending of batch and real-time events with the common feature definition. link] Intel: Four Data Cleaning Techniques to Improve Large Language Model (LLM) Performance If someone asks me to define LLM, this is my one-line definition. High-quality data is the cornerstone of LLM.

Data Engineering 62

Data Engineering Data Engineer Engineering Business Intelligence 62

Knowledge Hut

MARCH 7, 2024

This implies two things, one, the data coming from one source is out of date when compared to another source. Two, it creates a commonality of data definitions, concepts, metadata and the like.

Big Data 98

Big Data Bytes Data Governance Raw Data 98

Data Engineering Podcast

DECEMBER 26, 2021

In the same way that application performance monitoring ensures reliable software and keeps application downtime at bay, Monte Carlo solves the costly problem of broken data pipelines. Start trusting your data with Monte Carlo today! Hightouch is the easiest way to sync data into the platforms that your business teams rely on.

Data Engineering 100

Data Engineering Data Engineer Engineering Data Warehouse 100

Data Engineering Podcast

APRIL 12, 2021

Datafold also helps automate regression testing of ETL code with its Data Diff feature that instantly shows how a change in ETL or BI code affects the produced data, both on a statistical level and down to individual rows and values. RudderStack’s smart customer data pipeline is warehouse-first.

Data Warehouse 100

Data Warehouse Data Pipeline BI Data 100

Monte Carlo

MARCH 28, 2023

As part of this movement, Fivetran and dbt fundamentally altered the data pipeline from ETL to ELT. Hightouch interrupted SaaS eating the world in an attempt to shift the center of gravity to the data warehouse. Other common light transformations done within the ingestion phase are data formatting and deduplication.

Data Warehouse 126

Data Warehouse Raw Data Data Pipeline Software Engineer 126

Data Engineering Podcast

DECEMBER 21, 2021

Summary One of the perennial challenges of data analytics is having a consistent set of definitions, along with a flexible and performant API endpoint for querying them. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows.

Data Analytics 130

Data Analytics BI Computer Science SQL 130

Knowledge Hut

JUNE 30, 2023

Investigate real-time data processing methods by employing distributed systems. Master the art of data modeling and developing scalable data architectures. Ingestion of data, processing of data, machine learning, and graph processing are a few topics covered in the book.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Monte Carlo

JANUARY 23, 2024

Every team has their own analytic domain they own, forcing decentralized team structures around broadly standardized definitions of what “good” data looks like. The data warehouse is the mirror of the organization in many ways. Data debt or data pipeline chaos. So, pick your poison?

Data Engineering 98

Data Engineering Data Engineer Engineering Data 98

Data Engineering Podcast

SEPTEMBER 11, 2022

Interview Introduction How did you get involved in the area of data management? How does the garbage in/garbage out problem manifest in data warehouse/data lake environments? What are the different places in a data system that schema definitions need to be established?

Systems 100

Systems Metadata Building MongoDB 100

Cloudera

SEPTEMBER 29, 2022

In the data fabric implementation, the concepts in data mesh map to real-world artifacts in a data architecture. Corresponding to the data mesh example in Figure 4, D1, D2 are tables in a data warehouse. A2 is an app built as a spark job orchestrated to run when some data shows up.

Architecture 74

Architecture Data Architecture Metadata Data Warehouse 74