Data Engineer, Data Pipeline and High Quality Data

Data Engineer

Data Pipeline

High Quality Data

Data Engineering Weekly #161

Data Engineering Weekly

MARCH 3, 2024

Editor’s Note: Chennai, India Meetup - March-08 Update We are thankful to Ideas2IT to host our first Data Hero’s meetup. There will be food, networking, and real-world talks around data engineering. link] Martin Chesbrough: How to Build a Modern Data Team?

Data Engineering

Data Engineering Data Engineer Pipeline-centric Engineering

5 Skills Data Engineers Should Master to Keep Pace with GenAI

Monte Carlo

FEBRUARY 27, 2024

If you’re a data engineer experiencing GenAI-induced whiplash, you’re not alone. On one hand, everyone’s talking about whether GenAI’s not-insignificant data engineering skills are going to automate away their jobs. They need robust data pipelines, high-quality data, well-guarded privacy, and cost-effective scalability.

Data Engineering

Data Engineering Data Engineer Engineering High Quality Data

Join 16,000+

Insiders

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Webinars

The Path to Product Excellence: Avoiding Common Pitfalls and Enhancing Communication

Peak Performance: Continuous Testing & Evaluation of LLM-Based Applications

From Developer Experience to Product Experience: How a Shared Focus Fuels Product Success

Understanding User Needs and Satisfying Them

Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You Need to Know

MORE WEBINARS

Trending Sources

The Symbiotic Relationship Between AI and Data Engineering

Ascend.io

FEBRUARY 28, 2024

The rise of generative AI is changing more than just technology; it’s reshaping our professional landscapes — and yes, data engineering is directly experiencing the impact. How does AI recalibrate the workload and priorities of data teams? How can data engineers harness the power of AI?

Data Engineering

Data Engineering Data Engineer Engineering Metadata

Webinars

The Path to Product Excellence: Avoiding Common Pitfalls and Enhancing Communication

Peak Performance: Continuous Testing & Evaluation of LLM-Based Applications

From Developer Experience to Product Experience: How a Shared Focus Fuels Product Success

Understanding User Needs and Satisfying Them

Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You Need to Know

MORE WEBINARS

How to become Azure Data Engineer I Edureka

Edureka

FEBRUARY 7, 2023

An Azure Data Engineer is responsible for designing, implementing, and maintaining data management and data processing systems on the Microsoft Azure cloud platform. They work with large and complex data sets and are responsible for ensuring that data is stored, processed, and secured efficiently and effectively.

Data Engineering

Data Engineering Data Engineer Engineering Programming Language

Intrinsic Data Quality: 6 Essential Tactics Every Data Engineer Needs to Know

Monte Carlo

JANUARY 10, 2024

On the other hand, “Can the marketing team easily segment the customer data for targeted communications?” usability) would be about extrinsic data quality. A data observability platform, like Monte Carlo’s, can help. It employs machine learning to empower data engineering teams to resolve data issues more rapidly.

Data Cleanse

Data Cleanse Data Engineering Data Engineer Engineering

Forge Your Career Path with Best Data Engineering Certifications

ProjectPro

FEBRUARY 21, 2023

With so many data engineering certifications available , choosing the right one can be a daunting task. There are over 133K data engineer job openings in the US, but how will you stand out in such a crowded job market? The answer is- by earning professional data engineering certifications! AWS or Azure?

Certification

Certification Data Engineering Data Engineer Engineering

7 Essential Data Cleaning Best Practices

Monte Carlo

APRIL 1, 2024

But, for data engineers, there’s something else that comes pretty close to the top of that list: clean data. Data cleaning is an essential step to ensure your data is safe from the adage “garbage in, garbage out.” Implement Routine Data Audits Build a data cleaning cadence into your data teams’ schedule.

High Quality Data

High Quality Data Datasets Data Data Pipeline

5 Hard Truths About Generative AI for Technology Leaders

Monte Carlo

JANUARY 3, 2024

But RAG development comes with a learning curve, even for your most talented data engineers. They need to know prompt engineering , vector databases and embedding vectors , data modeling, data orchestration , data pipelines and all for RAG. Data engineers. It’s 2024 now.

Technology

Technology Database Data Governance Data Engineering

How Fox Facilitates Data Trust with Governance and Monte Carlo

Monte Carlo

JANUARY 16, 2024

Table of Contents Solve data silos starting at the people-level Keep data governance approachable Oliver Gomes’ data governance best practices Manage and promote the value of high-quality data How will Generative AI impact data quality at Fox? The complexity of a modern data pipeline.

Government

Government High Quality Data Data Governance Entertainment

DataOps vs. MLOps: Similarities, Differences, and How to Choose

Databand.ai

JULY 17, 2023

By adopting a set of best practices inspired by Agile methodologies, DevOps principles, and statistical process control techniques, DataOps helps organizations deliver high-quality data insights more efficiently. Learn more in our detailed guide to DataOps tools (coming soon) What Is MLOps?

Data Pipeline

Data Pipeline Machine Learning High Quality Data BI

Data Quality Platform: Benefits, Key Features, and How to Choose

Databand.ai

JULY 11, 2023

There are several reasons why organizations need a data quality platform to ensure the accuracy and reliability of their data. With a data quality platform in place, decision-makers can trust the data they use, reducing the risk of costly mistakes and missed opportunities.

Data Cleanse

Data Cleanse Telecommunication High Quality Data BI

Data Observability Tools: Types, Capabilities, and Notable Solutions

Databand.ai

JULY 5, 2023

What Are Data Observability Tools? Data observability tools are software solutions that oversee, analyze, and improve the performance of data pipelines. Data observability tools allow teams to detect issues such as missing values, duplicate records, or inconsistent formats early on before they affect downstream processes.

Data Pipeline

Data Pipeline Data Lake Data Warehouse Datasets

[O’Reilly Book] Chapter 1: Why Data Quality Deserves Attention Now

Monte Carlo

AUGUST 31, 2023

As the data analyst or engineer responsible for managing this data and making it usable, accessible, and trustworthy, rarely a day goes by without having to field some request from your stakeholders. But what happens when the data is wrong? In our opinion, data quality frequently gets a bad rep.

Data Lake

Data Lake Data Pipeline Unstructured Data Data Warehouse

The Significance of O’Reilly’s Data Quality Fundamentals

Monte Carlo

OCTOBER 4, 2022

In November of 2020, O’Reilly Media first approached us with the idea to author Data Quality Fundamentals: A Practitioner’s Guide to Building More Trustworthy Data Pipelines. It was an inflection point for a fledgling company that had only just begun to establish the category of data observability.

High Quality Data

High Quality Data Data Pipeline Media Data

AI Implementation: The Roadmap to Leveraging AI in Your Organization

Ascend.io

JANUARY 10, 2024

AI models are only as good as the data they consume, making continuous data readiness crucial. Here are the key processes that need to be in place to guarantee consistently high-quality data for AI models: Data Availability: Establish a process to regularly check on data availability. Actionable tip?

Data Pipeline

Data Pipeline Government Data Governance Raw Data

Data Quality Engineer: Skills, Salary, & Tools Required

Monte Carlo

JULY 27, 2023

These specialists are also commonly referred to as data reliability engineers. To be successful in their role, data quality engineers will need to gather data quality requirements (mentioned in 65% of job postings) from relevant stakeholders.

Engineering

Engineering Healthcare Scala Data Warehouse

The Data Janitor Letters - September 2021

Pipeline Data Engineering

OCTOBER 15, 2021

Data engineering salon. News and interesting reads about the world of data. The First Rule of Machine Learning: Start without Machine Learning Eugene Yan, Applied Scientist, Amazon Having robust data pipelines and high-quality data labels also suggests you’re ready for machine learning.

High Quality Data

High Quality Data Machine Learning Kafka Data Pipeline

Data Integrity vs. Data Validity: Key Differences with a Zoo Analogy

Monte Carlo

MARCH 24, 2023

The key differences are that data integrity refers to having complete and consistent data, while data validity refers to correctness and real-world meaning – validity requires integrity but integrity alone does not guarantee validity. What is Data Integrity? How Do You Maintain Data Integrity?

Data Validation

Data Validation Data Integration Data Cleanse Data Pipeline

Data Quality Testing: 7 Essential Tests

Monte Carlo

DECEMBER 19, 2022

Data quality testing is the process of validating that key characteristics of a dataset match what is anticipated prior to its consumption. According to Gartner , bad data costs organizations on average an estimated $12.9 Download our Data Quality Testing 101 eBook So, what is data quality testing anyway?

High Quality Data

High Quality Data Bytes Data SQL

Data Validation Testing: Techniques, Examples, & Tools

Monte Carlo

AUGUST 8, 2023

By applying rules and checks, data validation testing verifies the data meets predefined standards and business requirements to help prevent data quality issues and data downtime. From this perspective, the data validation process looks a lot like any other DataOps process.

Data Validation

Data Validation Data Pipeline SQL Data

Data Integrity vs. Data Quality: 4 Key Differences You Can’t Confuse

Monte Carlo

JUNE 16, 2023

Data quality has broad applications across industries, but its importance and degree of quality required is also contextual to the use case. For example, in marketing, high-quality data can help businesses better understand their customers, allowing them to create more targeted and effective campaigns.

Data Integration

Data Integration Hospitality Banking Healthcare

How Tenable Built a Reliable Data Platform at Terabyte Scale with Monte Carlo

Monte Carlo

JUNE 12, 2023

Like many companies, Tenable’s data team structure has shifted over time, but they currently have a centralized data platform team that includes data engineering, data science, and a data ingestion team. The team built out manual tests to address data quality issues as they came up.

Kafka

Kafka Algorithm Data Pipeline High Quality Data

Making Email Better With AI At Shortwave

Data Engineering Podcast

APRIL 21, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex. Go to dataengineeringpodcast.com/dagster today to get started.

Data Lake

Data Lake High Quality Data Data Pipeline Machine Learning

An Open Letter to Data Ninjas - Yes, You Need To Implement Data Contract System

Data Engineering Weekly

NOVEMBER 11, 2022

I took a break from data engineering and focused on system monitoring and observability for some time. Little did I know I would be building data pipelines to measure system efficiency instead of measuring business efficiency. As the famous saying goes; Once a data engineer, always a data engineer.

Systems

Systems High Quality Data Consulting Metadata

Using Trino And Iceberg As The Foundation Of Your Data Lakehouse

Data Engineering Podcast

FEBRUARY 18, 2024

In this episode Dain Sundstrom, CTO of Starburst, explains how the combination of the Trino query engine and the Iceberg table format offer the ease of use and execution speed of data warehouses with the infinite storage and scalability of data lakes. Data lakes are notoriously complex. Your first 30 days are free!

Data Lake

Data Lake High Quality Data Data Warehouse Google Cloud

Build Your Second Brain One Piece At A Time

Data Engineering Podcast

APRIL 28, 2024

In this episode he explains the data collection and preparation process, the collection of model types and sizes that work together to power the experience, and how to incorporate it into your workflow to act as a second brain. Data lakes are notoriously complex. Go to dataengineeringpodcast.com/dagster today to get started.

Building

Building Data Lake High Quality Data Machine Learning

How Dr. Squatch Keeps Data Clean & Fresh with Monte Carlo

Monte Carlo

SEPTEMBER 22, 2022

Most employees have access to domain specific dashboards and will also engage the data team when their questions are deeper than a dashboard can answer. “We We have a sole data engineer and the rest of our team is pretty analytics focused. You can think of it as nine analytics folks and our singular hero on data engineering.

High Quality Data

High Quality Data Data Data Engineering Data Engineer

Ship Smarter Not Harder With Declarative And Collaborative Data Orchestration On Dagster+

Data Engineering Podcast

MARCH 24, 2024

In this episode Pete Hunt, CEO of Dagster labs, outlines these new capabilities, how they reduce the burden on data teams, and the increased collaboration that they enable across teams and business units. Data lakes are notoriously complex. Go to dataengineeringpodcast.com/dagster today to get started. Your first 30 days are free!

Data Lake

Data Lake High Quality Data Hadoop Data Pipeline

Designing A Non-Relational Database Engine

Data Engineering Podcast

APRIL 14, 2024

In this episode Oren Eini, CEO and creator of RavenDB, explores the nuances of relational vs. non-relational engines, and the strategies for designing a non-relational database. Dagster offers a new approach to building and running data platforms and data pipelines. Data lakes are notoriously complex.

Non-relational Database

Non-relational Database Relational Database Database Designing

Centralize Your Data Processes With a DataOps Process Hub

DataKitchen

NOVEMBER 4, 2021

It often takes months to progress from a data lake to the final delivery of insights. One data engineer called it the “last mile problem.” . In our many conversations about data analytics, data engineers, analysts and scientists have verbalized the difficulty of creating analytics in the modern enterprise.

Process

Process Data Process Pharmaceutical Data Lake

Celebrating the New Pioneers of Data Reliability

Monte Carlo

AUGUST 17, 2021

And your entire data engineering team is all in on this “data as code” movement. The flip side of this data-driven coin: your stakeholders (CEO and CTO included) ping you nearly every other hour to ask you: “is my data up-to-date?” To achieve this data trust, they turned to Monte Carlo.

Insurance

Insurance Retail Data Pipeline Portfolio

When And How To Conduct An AI Program

Data Engineering Podcast

MARCH 3, 2024

Colleen Tartow has worked across all stages of the data lifecycle, and in this episode she shares her hard-earned wisdom about how to conduct an AI program for your organization. Data lakes are notoriously complex. Visit dataengineeringpodcast.com/data-council and use code dataengpod20 to register today!

Programming

Programming Data Lake High Quality Data Data Pipeline

Find Out About The Technology Behind The Latest PFAD In Analytical Database Development

Data Engineering Podcast

FEBRUARY 25, 2024

Database

Database Technology Data Lake High Quality Data

Adding Anomaly Detection And Observability To Your dbt Projects Is Elementary

Data Engineering Podcast

MARCH 31, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Dagster offers a new approach to building and running data platforms and data pipelines. Go to dataengineeringpodcast.com/dagster today to get started.

Project

Project Data Lake High Quality Data Data Workflow

Reconciling The Data In Your Databases With Datafold

Data Engineering Podcast

MARCH 17, 2024

In this episode Gleb Mezhanskiy, founder and CEO of Datafold, discusses the different error conditions and solutions that you need to know about to ensure the accuracy of your data. Data lakes are notoriously complex. Visit dataengineeringpodcast.com/data-council and use code dataengpod20 to register today!

Database

Database Data Lake High Quality Data Data Workflow

Tackling Real Time Streaming Data With SQL Using RisingWave

Data Engineering Podcast

FEBRUARY 4, 2024

In this episode Yingjun Wu explains how it is architected to power analytical workflows on continuous data flows, and the challenges of making it responsive and scalable. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

SQL

SQL Data Lake High Quality Data Data Pipeline

Version Your Data Lakehouse Like Your Software With Nessie

Data Engineering Podcast

MARCH 10, 2024

In this episode Alex Merced explains how the branching and merging functionality in Nessie allows you to use the same versioning semantics for your data lakehouse that you are used to from Git. Data lakes are notoriously complex. Visit [dataengineeringpodcast.com/data-council]([link] and use code **dataengpod20** to register today!

Data Lake

Data Lake High Quality Data Data Pipeline Architecture

What is Data Accuracy? Definition, Examples and KPIs

Monte Carlo

JULY 11, 2023

If the data they use to inform these decisions isn’t accurate—that is, if it doesn’t reflect reality—then the decisions they make can result in lost revenue, eroded stakeholder trust, and wasted data engineering resources. In other words, bad data is bad for business.”

Data Cleanse

Data Cleanse Datasets Data Governance Government

Build vs Buy Data Pipeline Guide

Monte Carlo

APRIL 24, 2023

The ingestion process can be broadly categorized into 3 paradigms: Batch: import data in bulk from varied sources Streaming: ingest event style data Incremental (or mini batch): in between batch and streaming Source : Fundamentals of Data Engineering by Joe Reis and Matt Housley. Upstream data evolution breaks pipelines.

Data Pipeline

Data Pipeline Building Data Ingestion BI

Data Sharing Across Business And Platform Boundaries

Data Engineering Podcast

FEBRUARY 11, 2024

In this episode Andrew Jefferson explains the complexities of building a robust system for data sharing, the techno-social considerations, and how the Bobsled platform that he is building aims to simplify the process. Dagster offers a new approach to building and running data platforms and data pipelines.

Data Lake

Data Lake High Quality Data Government Data Pipeline

Troubleshooting Kafka In Production

Data Engineering Podcast

DECEMBER 24, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Kafka

Kafka Data Lake High Quality Data SQL

Data Observability for Analytics and ML teams

Towards Data Science

APRIL 6, 2023

Principles, practices, and examples for ensuring high quality data flows Source: DreamStudio (generated by author) Nearly 100% of companies today rely on data to power business opportunities and 76% use data as an integral part of forming a business strategy. Or even just a little bit.

Unstructured Data

Unstructured Data Metadata Data Coding

Implementing Data Contracts in the Data Warehouse

Monte Carlo

JANUARY 25, 2023

Data professionals are producers and consumers The first key difference between production service contracts and contracts within the data warehouse is that data producers and consumers are often both in data roles (analysts, data scientists, data engineers, etc). What happens when we detect bad data?

Data Warehouse

Data Warehouse Data High Quality Data Metadata

Build A Data Lake For Your Security Logs With Scanner

Data Engineering Podcast

JANUARY 28, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

Data Lake

Data Lake Building High Quality Data AWS

Data Engineering Weekly #161

5 Skills Data Engineers Should Master to Keep Pace with GenAI

Webinars

Trending Sources

The Symbiotic Relationship Between AI and Data Engineering

Webinars

How to become Azure Data Engineer I Edureka

Intrinsic Data Quality: 6 Essential Tactics Every Data Engineer Needs to Know

Forge Your Career Path with Best Data Engineering Certifications

7 Essential Data Cleaning Best Practices

5 Hard Truths About Generative AI for Technology Leaders

How Fox Facilitates Data Trust with Governance and Monte Carlo

DataOps vs. MLOps: Similarities, Differences, and How to Choose

Data Quality Platform: Benefits, Key Features, and How to Choose

Data Observability Tools: Types, Capabilities, and Notable Solutions

[O’Reilly Book] Chapter 1: Why Data Quality Deserves Attention Now

The Significance of O’Reilly’s Data Quality Fundamentals

AI Implementation: The Roadmap to Leveraging AI in Your Organization

Data Quality Engineer: Skills, Salary, & Tools Required

The Data Janitor Letters - September 2021

Data Integrity vs. Data Validity: Key Differences with a Zoo Analogy

Data Quality Testing: 7 Essential Tests

Data Validation Testing: Techniques, Examples, & Tools

Data Integrity vs. Data Quality: 4 Key Differences You Can’t Confuse

How Tenable Built a Reliable Data Platform at Terabyte Scale with Monte Carlo

Making Email Better With AI At Shortwave

An Open Letter to Data Ninjas - Yes, You Need To Implement Data Contract System

Using Trino And Iceberg As The Foundation Of Your Data Lakehouse

Build Your Second Brain One Piece At A Time

How Dr. Squatch Keeps Data Clean & Fresh with Monte Carlo

Ship Smarter Not Harder With Declarative And Collaborative Data Orchestration On Dagster+

Designing A Non-Relational Database Engine

Centralize Your Data Processes With a DataOps Process Hub

Celebrating the New Pioneers of Data Reliability

When And How To Conduct An AI Program

Find Out About The Technology Behind The Latest PFAD In Analytical Database Development

Adding Anomaly Detection And Observability To Your dbt Projects Is Elementary

Reconciling The Data In Your Databases With Datafold

Tackling Real Time Streaming Data With SQL Using RisingWave

Version Your Data Lakehouse Like Your Software With Nessie

What is Data Accuracy? Definition, Examples and KPIs

Build vs Buy Data Pipeline Guide

Data Sharing Across Business And Platform Boundaries

Troubleshooting Kafka In Production

Data Observability for Analytics and ML teams

Implementing Data Contracts in the Data Warehouse

Build A Data Lake For Your Security Logs With Scanner

Stay Connected