Data Engineering Podcast

OCTOBER 22, 2021

In this episode Oliver Laslett describes why dashboards aren’t sufficient for business analytics, how Lightdash promotes the work that you are already doing in your data warehouse modeling with dbt, and how they are focusing on bridging the divide between data teams and business teams and the requirements that they have for data workflows.

Engineering 100

Engineering Business Intelligence BI Data Warehouse 100

Data Engineering Podcast

JULY 3, 2022

In this episode they explain how the utility is implemented to run quickly and how you can start using it in your own data workflows to ensure that your data warehouse isn’t missing any records from your source systems. What are some of the ways that you have seen data-diff incorporated in the workflow of data teams?

Data Integration 100

Data Integration MongoDB Scala MySQL 100

Data Engineering Podcast

FEBRUARY 20, 2022

Summary The life sciences as an industry has seen incredible growth in scale and sophistication, along with the advances in data technology that make it possible to analyze massive amounts of genomic information. If you’ve learned something or tried out a project from the show then tell us about it!

Systems 100

Systems Software Engineer Software Engineering Data Warehouse 100

Data Engineering Podcast

JULY 17, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Management 130

Data Management Management Metadata MongoDB 130

Data Engineering Podcast

SEPTEMBER 11, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Pipeline 130

Data Pipeline Building MongoDB Scala 130

Data Engineering Podcast

JULY 31, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

IT 100

IT Metadata MongoDB MySQL 100

Data Engineering Weekly

DECEMBER 25, 2023

The Rising Impact of AI and Large Language Models 2023 witnessed a substantial impact of AI and large language models in data engineering. These technologies are increasingly automating processes like ETL, improving data quality management, and evolving the landscape of data tools.

Data Engineering 52

Data Engineering Data Engineer Engineering Unstructured Data 52

Data Engineering Podcast

JANUARY 1, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Warehouse 100

Data Warehouse Data Lake Hadoop SQL 100

Data Engineering Podcast

APRIL 5, 2021

She explains how the design of the platform is informed by the needs of managing data projects for large and small teams across her previous roles, how it integrates with your existing systems, and how it can work to bring everyone onto the same page. What portions of the data workflow is Atlan responsible for?

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100

DataKitchen

MAY 10, 2024

Development : Observability in development includes conducting regression tests and impact assessments when new code, tools, or configurations are introduced, helping maintain system integrity as new code of data sets are introduced into production.

Raw Data 40

Raw Data Data Ingestion Datasets Data 40

Data Engineering Podcast

SEPTEMBER 4, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

MongoDB 100

MongoDB Scala MySQL Data Ingestion 100

Data Engineering Podcast

OCTOBER 19, 2020

Summary In order for analytics and machine learning projects to be useful, they require a high degree of data quality. In this episode Barr Moses and Lior Gavish, co-founders of Monte Carlo, share the leading causes of what they refer to as data downtime and how it manifests. What is "data downtime"?

Machine Learning 100

Machine Learning Data Engineering Data Engineer Data 100

Data Engineering Podcast

MAY 3, 2021

Summary The Data industry is changing rapidly, and one of the most active areas of growth is automation of data workflows. Taking cues from the DevOps movement of the past decade data professionals are orienting around the concept of DataOps. Email hosts@dataengineeringpodcast.com ) with your story.

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100

Data Engineering Podcast

AUGUST 5, 2019

Announcements Welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

PostgreSQL 100

PostgreSQL MongoDB Metadata Media 100

Workfall

JULY 4, 2023

Reading Time: 8 minutes In the world of data engineering, a mighty tool called DBT (Data Build Tool) comes to the rescue of modern data workflows. Imagine a team of skilled data engineers on an exciting quest to transform raw data into a treasure trove of insights.

Data Warehouse 69

Data Warehouse SQL PostgreSQL Database 69

ThoughtSpot

SEPTEMBER 6, 2023

The pathway from ETL to actionable analytics can often feel disconnected and cumbersome, leading to frustration for data teams and long wait times for business users. And even when we manage to streamline the data workflow, those insights aren’t always accessible to users unfamiliar with antiquated business intelligence tools.

Data Workflow 91

Data Workflow Raw Data Data Lake Business Intelligence 91

Databand.ai

AUGUST 30, 2023

One key aspect of data orchestration is the automation of data pipeline tasks. By automating repetitive tasks, such as data extraction, transformation, and loading (ETL), organizations can streamline their data workflows and reduce the risk of human error.

Data Governance 52

Data Governance Data Pipeline Government Data Cleanse 52

Knowledge Hut

NOVEMBER 17, 2023

Each of these languages has its specific use cases and advantages within the context of Azure Data Engineering. Tailor your learning and mastery based on the Azure services you'll be primarily working with and the requirements of your projects. Azure Data Factory stands at the forefront, orchestrating data workflows.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

Data Engineering Podcast

NOVEMBER 19, 2023

Summary The dbt project has become overwhelmingly popular across analytics and data engineering teams. Dustin Dorsey and Cameron Cyr co-authored a practical guide to building your dbt project. In this episode they share their hard-won wisdom about how to build and scale your dbt projects.

Project 147

Project Data Lake High Quality Data SQL 147

DataKitchen

FEBRUARY 27, 2024

Embracing DataOps for Enhanced Data Journey Management The complexity of managing Data Journeys, especially in RAG and LLMs, underscores the importance of embracing DataOps principles. DataOps provides a framework for automating and optimizing data workflows, emphasizing collaboration, monitoring, and continuous improvement.

Data Engineering 48

Data Engineering Data Engineer Engineering Unstructured Data 48

Knowledge Hut

MARCH 28, 2024

Salary (Average) $135,678 per year (Source: Glassdoor) Top Companies Hiring Walmart, Verizon, AT&T Certifications Microsoft Certified: Azure Data Engineer Associate Job Role 8: Azure Data Platform Consultant The Azure data platform consultant offers the organizations how to benefit from the usage of Azure data services.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

DataKitchen

DECEMBER 20, 2020

If it takes months to spin up a development environment , then analytics projects will become irrelevant before they are completed. If the data team is always dealing with data errors and putting out fires, then they’ll be constantly pulled away from their highest priority projects. .

Engineering 59

Engineering Manufacturing Data Analytics Data Workflow 59

DareData

NOVEMBER 28, 2023

As the scope of the models and the data continues to scale, the role of a Data Scientist has evolved accordingly in the last years. This enables reproducibility, as you can easily retrieve and reproduce past experiments by referencing their logged information, ensuring that scientific breakthroughs in your project are not lost.

Data Science 52

Data Science Medical Machine Learning Data 52

Knowledge Hut

DECEMBER 28, 2023

This certification, often referred to as the Azure Data Engineer Associate certification, validates the competency of individuals in implementing Azure data solutions. It’s a testament to their ability to create scalable, efficient and secure data pipelines. What is the Azure Data Engineer Certification?

Certification 52

Certification Data Engineering Data Engineer Engineering 52

Monte Carlo

JULY 27, 2023

In terms of specific processing techniques for large scale data, 13% of job postings mentioned Spark, 17% mentioned Kafka/Kinesis, 17% mentioned Hadoop, and one posting referenced S3. Collaborate with data engineering and development teams to implement data quality best practices and optimize data workflows.

Engineering 52

Engineering Healthcare Scala Data Warehouse 52

dbt Developer Hub

MARCH 23, 2023

Visibility to a project : Audit_helper gives visibility to a refactoring process by showing how your code is delivering the same results in both row-wise and column-wise comparisons. Installation instructions ​ Let’s start by setting the stage for the audit_helper package to work in our project. and below 2.0.0. and below 2.0.0.

Data Warehouse 64

Data Warehouse Generalist SQL Coding 64

Data Engineering Podcast

APRIL 14, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management This episode is brought to you by Datafold – a testing automation platform for data engineers that prevents data quality issues from entering every part of your data workflow, from migration to dbt deployment.

Non-relational Database 147

Non-relational Database Relational Database Database Designing 147

DataKitchen

JANUARY 3, 2022

When the project scope and schedule are fixed, managers have to add more staff to keep up with the workload. . These groups tend to be more insulated from their customer’s actual needs, so they take many months to deliver, and when a project does get done, it usually requires rework. Some IT teams are fantastic.

Business Analyst 98

Business Analyst Data Lake Consulting Data Analytics 98

Hepta Analytics

FEBRUARY 14, 2022

Disadvantages of a data lake are: Can easily become a data swamp data has no versioning Same data with incompatible schemas is a problem without versioning Has no metadata associated It is difficult to join the data Data warehouse stores processed data, mostly structured data.

Data Ingestion 52

Data Ingestion Data Engineering Data Engineer Engineering 52

ProjectPro

DECEMBER 21, 2021

These include the skills needed for a machine learning career, a few machine learning projects for practice to develop those skills, and a clear grasp of the different types of machine learning careers available. Get Closer To Your Dream of Becoming a Data Scientist with 150+ Solved End-to-End ML Projects 2.

Machine Learning 52

Machine Learning Engineering Algorithm Computer Science 52

Monte Carlo

DECEMBER 13, 2023

Tip #1: Don’t be afraid to start small with data observability Our experts’ number one tip for getting started with data observability? Thanks to Monte Carlo, we can detect these issues very quickly — most of the time, even before our data workflow fails, so users aren’t impacted. Don’t get too complicated.

BI 52

BI Data Ingestion SQL Data Warehouse 52

Data Engineering Podcast

DECEMBER 3, 2023

This episode is brought to you by Datafold – a testing automation platform for data engineers that finds data quality issues for every part of your data workflow, from migration to deployment. Datafold has recently launched a 3-in-1 product experience to support accelerated data migrations.

Systems 130

Systems Designing Data Lake SQL 130

DataKitchen

MARCH 16, 2023

By reducing wasted time through minimizing cycle time, organizations can optimize their resource allocation, ensuring that data analysts, data scientists, and other stakeholders can focus on high-impact projects and maximize their contributions to the organization’s success. How does one track a team’s productivity?

Data Analytics 52

Data Analytics Manufacturing Data Process 52

Knowledge Hut

SEPTEMBER 29, 2023

One of the pivotal responsibilities is managing data workflows and pipelines, a core aspect of a data engineer's role. To truly understand the importance of data engineers, we can explore the hierarchy of needs in data science.

Certification 52

Certification Data Engineering Data Engineer Engineering 52

Data Engineering Podcast

MAY 12, 2024

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst is an end-to-end data lakehouse platform built on Trino, the query engine Apache Iceberg was designed for, with complete support for all table formats including Apache Iceberg, Hive, and Delta Lake.

Management 130

Management Data Lake High Quality Data Machine Learning 130

Data Engineering Podcast

FEBRUARY 18, 2024

Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). Multiple open source projects and vendors have been working together to make this vision a reality. Data lakes are notoriously complex.

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Data Engineering Podcast

MARCH 24, 2024

Summary A core differentiator of Dagster in the ecosystem of data orchestration is their focus on software defined assets as a means of building declarative workflows. With their launch of Dagster+ as the redesigned commercial companion to the open source project they are investing in that capability with a suite of new features.

Data Lake 162

Data Lake High Quality Data Hadoop Data Pipeline 162