Data Engineering Podcast

JANUARY 26, 2020

Summary Data pipelines are complicated and business critical pieces of technical infrastructure. Unfortunately they are also complex and difficult to test, leading to a significant amount of technical debt which contributes to slower iteration cycles. Can you start by explaining what Great Expecations is and the origin of the project?

Data Pipeline 100

Data Pipeline PostgreSQL Media Data Validation 100

Data Engineering Podcast

MAY 19, 2019

Summary In recent years the traditional approach to building data warehouses has shifted from transforming records before loading, to transforming them afterwards. The data build tool (dbt) is designed to bring battle tested engineering practices to your analytics pipelines.

Data Analytics 100

Data Analytics Building Engineering Data Warehouse 100

Monte Carlo

DECEMBER 13, 2023

You can’t turn around in the data space without running into hot takes about GenAI. What do data leaders say when the cameras aren’t rolling and the board decks aren’t public? What do data leaders say when the cameras aren’t rolling and the board decks aren’t public? Will it make our jobs obsolete?

BI 59

BI Metadata Data Technology 59

dbt Developer Hub

JANUARY 23, 2023

Testing the quality of data in your warehouse is an important aspect in any mature data pipeline. One of the biggest blockers for developing a successful data quality pipeline is aggregating test failures and successes in an informational and actionable way. Recently, we overhauled our testing framework.

Metadata 52

Metadata High Quality Data SQL Data Integration 52

Zalando Engineering

NOVEMBER 19, 2023

What this article is about What kind of changes we had to make to the codebase How we did the actual upgrade What challenges we faced How we did the data transfer How the data was kept in sync What this article is not A step-by-step guide on how to upgrade Elasticsearch (read on to find out why). x also represented a choice.

Scala 86

Scala Java Coding IT 86

Monte Carlo

DECEMBER 13, 2023

You can’t turn around in the data space without running into hot takes about GenAI. What do data leaders say when the cameras aren’t rolling and the board decks aren’t public? What do data leaders say when the cameras aren’t rolling and the board decks aren’t public? Will it make our jobs obsolete?

BI 52

BI Metadata Data Technology 52

DataKitchen

JANUARY 18, 2023

Any good plumber can give you some hard-won tips on managing data pipelines effectively, insights that might save your career from going down the drain. Good testing, like exercise and veganism, is the subject of fervent talk and half-hearted action. There are lots of reasons good people test inadequately. Of course not.

Data Pipeline 52

Data Pipeline Data AWS Data Engineering 52

dbt Developer Hub

MAY 16, 2022

But this means the name has a life outside of dbt and supports the many end users who will potentially never know about dbt and where this data came from, but still access the database objects in the database or business intelligence (BI) tool. With these goals in mind, they must form a hypothesis and prove their point with data.

BI 52

BI Data Warehouse Database SQL 52

Cloudera

OCTOBER 28, 2020

Cloudera services logs offer a breadth of information to assist in cluster maintenance; from assisting in security checks, auditing tasks, and validation for performance tuning and testing tasks – to name a few. . This example solution will outline the rerouting/filtering of data, but an overall reduction is possible via this workflow.

Kafka 65

Kafka SQL Professional Services Consulting 65

Cloudera

SEPTEMBER 28, 2020

Cloudera delivers an enterprise data cloud that enables companies to build end-to-end data pipelines for hybrid cloud, spanning edge devices to public or private cloud, with integrated security and governance underpinning it to protect customers data. Data Science and machine learning workloads using CDSW.

Cloud 131

Cloud Kafka Professional Services Metadata 131

dbt Developer Hub

AUGUST 2, 2023

For Postgres/Redshift/Databricks {{ config ( materialized = 'materialized_view' , ) }} For Snowflake: {{ config ( materialized = 'dynamic_table' , ) }} note We are only supporting dynamic tables on Snowflake, not Snowflake’s materialized views (for a comparison between Snowflake Dynamic Tables and Materialized Views, refer docs.

Datasets 52

Datasets Kafka SQL Cloud 52

Monte Carlo

MAY 17, 2023

Bad pull requests may not be the root of all evil—but they’re certainly the root of some gnarly data quality incidents. With this new feature, users can: Quickly and easily correlate PRs on Github to data issues, accelerating incident resolution. GitHub PRs break data pipelines. The good news? Call it a fact of life.

Data Pipeline 52

Data Pipeline Coding Data Engineering Data Engineer 52

Monte Carlo

MAY 17, 2023

Bad pull requests may not be the root of all evil—but they’re certainly the root of some gnarly data quality incidents. With this new feature, users can: Quickly and easily correlate PRs on Github to data issues, accelerating incident resolution. GitHub PRs break data pipelines. The good news? There’s a better way.

Data Pipeline 52

Data Pipeline Data Engineering Data Engineer Data 52

dbt Developer Hub

APRIL 30, 2023

I, Sung, entered the data industry by chance in Fall 2014. My next gig was in consulting where I bootstrapped my way into data engineering and had to learn the whole gamut below. Armed with them, I finally saw the magic of the modern data stack and what problems it could solve. I wanted to build prestige in my career.

Data Engineering 64

Data Engineering Data Engineer Engineering Finance 64

dbt Developer Hub

FEBRUARY 7, 2022

You can follow along with Donny's data modeling technique for identity resolution in this dbt project repo. It includes a set of demo CSV files, which you can use as dbt seeds to test Donny's project for yourself. To do so, they want to build out their sales motion to target companies with active gaggles. Enter JaffleGaggle.

Data Warehouse 52

Data Warehouse Datasets Data SQL 52

dbt Developer Hub

MAY 9, 2022

The main goal of this article is to spread awareness of these endpoints as the docs are being built & show you how to use them. Please keep in mind that the collection is not an evergreen doc. Building “starter projects” they can deploy as templates for new teams onboarding to dbt from a learning standpoint.

Cloud 52

Cloud Management Government Data Warehouse 52

Cloudera

JANUARY 31, 2024

Replication is a crucial capability in distributed systems to address challenges related to fault tolerance, high availability, load balancing, scalability, data locality, network efficiency, and data durability. It forms a foundational element for building robust and reliable distributed architectures.

Management 69

Management Kafka Big Data Cloud 69

Edureka

APRIL 19, 2024

June 22, 2023 V6 December 21, 2023 For Other Models: Version Release Date –beta August 22, 2022 test/testp August 28, 2022 Niji December 20, 2022 Niji 5 April 2, 2023 Niji 6 January 29, 2024 Niji is a collaboration between Midjourney and Spellbrush, which is tuned to produce anime and illustrative tasks. May 3, 2023 V5.2

Algorithm 52

Algorithm Deep Learning Machine Learning Media 52

dbt Developer Hub

JULY 2, 2023

Data Vault 2.0 is a data modeling technique designed to help scale large data warehousing projects. So why Data Vault? Have you experienced a data warehousing project with 50+ data sources, with 25+ data developers working on the same data platform, or data spanning 5+ years with two or more generations of source systems?

Cloud 52

Cloud Data Warehouse Data BI 52

dbt Developer Hub

MARCH 9, 2022

Special Thanks: Emilie Schario, Matt Winkler dbt has done a great job of building an elegant, common interface between data engineers, analytics engineers, and any data-y role, by uniting our work on SQL. Should we increase the SQL surface area to build ML models? I like to call this interoperability a “baton pass.”

Machine Learning 52

Machine Learning Raw Data SQL Python 52

Monte Carlo

FEBRUARY 9, 2022

Is your data quality improving? What is your most used data? Where in the pipeline are your most frequent data issues occurring? With Snowflake Secure Data Sharing, building custom workflows and dashboards to answer these questions has never been easier. Which tables should we test?

Raw Data 52

Raw Data Analytics Architecture Data Data Analytics 52

Monte Carlo

SEPTEMBER 6, 2022

Our biggest priority at Monte Carlo is to make the lives of our customers easier by reducing data downtime and helping them accelerate the adoption of reliable data at their companies. 75% of respondents reporting that it takes four or more hours to detect a data quality incident.

Data Lake 52

Data Lake Machine Learning Data Science Data 52

Ripple Engineering

FEBRUARY 2, 2021

The Ripple Data Engineering team is expanding, which means higher frequency changes to our data pipeline source code. This means we need to build better, more configurable and more collaborative tooling that prevents code collisions and enforces software engineering best practices.

Building 52

Building Python Google Cloud Coding 52

Data Engineering Weekly

OCTOBER 30, 2022

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Sign up free to test out the tool today.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Ingestion 52

Knowledge Hut

APRIL 12, 2023

For example, you want to build a Customer Relationship Management (CRM) application with advanced automated lead management capabilities. Understanding the meaning of MVP will help you build a valuable yet minimal version of your product. MVP is a valuable technique to ensure you get clear ideas about how best to build the product.

Certification 52

Certification Utilities Building Project 52

Grouparoo

DECEMBER 13, 2021

As part of our goal to make Data Engineer's lives easier by integrating more deeply with the Modern Data Stack , we've been working on new ways to trigger Grouparoo's Schedules to run. This year we've watched dbt take off as the open source data transformation tool of choice and were excited for their 1.0

Metadata 52

Metadata Data Warehouse SQL Utilities 52

Rockset

APRIL 28, 2022

It took us only a couple days to set up our data pipelines into Rockset and after that, it was pretty straightforward. The docs were great. We experienced all sorts of challenges that have tested us in a variety of ways. Trusting Your Crazy Ideas We at Rockset are actively in build mode.

Medical 52

Medical Data Pipeline Designing Database 52

dbt Developer Hub

JANUARY 14, 2023

There is no reference data for the real estate market here - how many houses are being sold, for what price? The only data source we have is Idealista, which is a portal where real estate agencies post ads. The builder: I’m a data freelancer who deploys end to end solutions, so when I have a data problem, I cannot just let it go.

Data Ingestion 52

Data Ingestion Google Cloud Python Cloud 52

Workfall

FEBRUARY 28, 2023

The ChatGPT responses are generated in real-time, based on a combination of my training data and the specific input it receives from users. That said, it is constantly improving and evolving, as OpenAI updates its training data and algorithms to improve its accuracy and capabilities. Who Can Use ChatGPT?

Datasets 62

Datasets Coding Algorithm Technology 62

dbt Developer Hub

NOVEMBER 28, 2021

So I challenge you after reading this article to test out your skillsets, think about the code that you find yourself reusing again and again, and build a package. Packages can be as complex as you would want; it’s just SQL hidden in the mix of reusable macros and expansive testing frameworks. What is a package even?

Building 52

Building SQL Coding Datasets 52

Rockset

MAY 27, 2021

The second one is more flexibility, especially now in the last few years with how data has changed, how the shape of the data coming from many different places tends to be completely different, and how it's being used for very different types of applications. How do we give you more schema-query flexibility ?

MongoDB 40

MongoDB Kafka Cloud SQL 40

dbt Developer Hub

APRIL 30, 2019

As the maintainers of dbt, and analytics consultants, at Fishtown Analytics (now dbt Labs) we build a lot of dbt projects. Instead, use this as a guide once you’ve already got an idea of what you’re building for how you should break the transformations up into separate dbt models. Staging models : The atomic unit of data modeling.

Project 40

Project Database-centric Raw Data Data Warehouse 40

dbt Developer Hub

FEBRUARY 12, 2024

What’s in a data platform? We, as data professionals, have poured ourselves into raising happy healthy data products, and we should be proud of the insights they’ve driven. We remember the angsty teenage years where tests kept failing, seemingly just to spite us. Where’s this data coming from?

Metadata 59

Metadata Raw Data BI Project 59

dbt Developer Hub

FEBRUARY 15, 2023

The new dbt Certification Program has been created by dbt Labs to codify the data development best practices that enable safe, confident, and impactful use of dbt. Jade (J): I’m a newly minted Montreal Analytics Analytics Engineer coming from a background as a data analyst with a few months of dbt experience. How did the test go? ​

Certification 119

Certification Consulting BI SQL 119

Zalando Engineering

JULY 17, 2017

As such, we need high quality labelled data sets that we can use to benchmark model performance and evaluate changes to the model. Not only do we need such data sets for final validation, but going forward, we also need methods to acquire high-quality labelled data sets for training models. In what ways is it better?

Machine Learning 40

Machine Learning Data Datasets Data Collection 40

Christophe Blefari

MAY 29, 2023

In this test clickhouse works better. The Future of Data — Everyone wants a piece of the pie; no one wants to bake. Data Modeling, architecture Pattern, tools and the future — part 3 of Simon's guide. States of data season — Airbyte's state of data , Databricks's , lakeFS's.

BI 130

BI Data Warehouse Data Data Pipeline 130

Rock the JVM

JUNE 20, 2022

using Slick how to use Postgres specific data types using slick-pg how to auto-generate Slick schema from database For this blog, we will build a basic database for movies and related entities. Apart from that, Slick provides compile time safety by mapping the database columns to Scala data types.

Scala 40

Scala PostgreSQL Database SQL 40