Maxime Beauchemin

JANUARY 20, 2017

I joined Facebook in 2011 as a business intelligence engineer. By the time I left in 2013, I was a data engineer. We were data engineers! Data Engineering? Data science as a discipline was going through its adolescence of self-affirming and defining itself. We were pioneers.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130

Edureka

FEBRUARY 7, 2023

An Azure Data Engineer is responsible for designing, implementing, and maintaining data management and data processing systems on the Microsoft Azure cloud platform. They work with large and complex data sets and are responsible for ensuring that data is stored, processed, and secured efficiently and effectively.

Data Engineering 52

Data Engineering Data Engineer Engineering Programming Language 52

ProjectPro

FEBRUARY 21, 2023

With so many data engineering certifications available , choosing the right one can be a daunting task. There are over 133K data engineer job openings in the US, but how will you stand out in such a crowded job market? The answer is- by earning professional data engineering certifications! AWS or Azure?

Certification 52

Certification Data Engineering Data Engineer Engineering 52

Monte Carlo

JANUARY 10, 2024

On the other hand, “Can the marketing team easily segment the customer data for targeted communications?” usability) would be about extrinsic data quality. A data observability platform, like Monte Carlo’s, can help. It employs machine learning to empower data engineering teams to resolve data issues more rapidly.

Data Cleanse 52

Data Cleanse Data Engineering Data Engineer Engineering 52

Data Engineering Podcast

OCTOBER 1, 2023

Summary Artificial intelligence applications require substantial high quality data, which is provided through ETL pipelines. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. Datafold : ![Datafold]([link]

Building 162

Building BI SQL Machine Learning 162

Monte Carlo

JANUARY 3, 2024

But RAG development comes with a learning curve, even for your most talented data engineers. They need to know prompt engineering , vector databases and embedding vectors , data modeling, data orchestration , data pipelines and all for RAG. Your data engineering team is the backbone for ensuring data health.

Technology 113

Technology Database Data Governance Data Engineering 113

Monte Carlo

DECEMBER 22, 2023

So, as the year comes to a close, we rounded up our top 5 blogs about generative AI in 2023, to help you catch up before we venture into the AI madness of 2024 These 5 articles aim to give some clarity on the current state of generative AI, how to define use-cases and build for value, and finally, what’s coming next for data and AI teams.

High Quality Data 52

High Quality Data Data Science Data Engineering Data Engineer 52

Monte Carlo

MAY 3, 2024

But even though the data landscape is evolving, many enterprise data organizations are still managing data quality the “old” way: with simple data quality monitoring. The basics haven’t changed: high-quality data is still critical to successful business operations.

High Quality Data 52

High Quality Data Data Architecture Media 52

Monte Carlo

MARCH 7, 2024

Here’s the short answer: no, prompt engineering isn’t all hype. And as we move into our undoubtedly AI-laden future, prompt engineering will be a necessary skill for every data engineer to develop. But, unlike the click-bait influencers might suggest, prompt engineering isn’t going to be your super power either.

Engineering 59

Engineering High Quality Data Database Architecture 59

Monte Carlo

DECEMBER 22, 2023

So, as the year comes to a close, we rounded up our top 5 blogs about generative AI in 2023, to help you catch up before we venture into the AI madness of 2024 These 5 articles aim to give some clarity on the current state of generative AI, how to define use-cases and build for value, and finally, what’s coming next for data and AI teams.

High Quality Data 52

High Quality Data Data Science Data Engineering Data Engineer 52

Data Engineering Podcast

JUNE 29, 2020

This was a great conversation about the complexities of working in a niche domain of data analysis and how to build a pipeline of high quality data from collection to analysis. If you hand a book to a new data engineer, what wisdom would you add to it?

Data Collection 100

Data Collection Management High Quality Data Metadata 100

Monte Carlo

JANUARY 16, 2024

Table of Contents Solve data silos starting at the people-level Keep data governance approachable Oliver Gomes’ data governance best practices Manage and promote the value of high-quality data How will Generative AI impact data quality at Fox? The complexity of a modern data pipeline.

Government 52

Government High Quality Data Data Governance Entertainment 52

Monte Carlo

APRIL 1, 2024

But, for data engineers, there’s something else that comes pretty close to the top of that list: clean data. Data cleaning is an essential step to ensure your data is safe from the adage “garbage in, garbage out.” Implement Routine Data Audits Build a data cleaning cadence into your data teams’ schedule.

High Quality Data 52

High Quality Data Datasets Data Data Pipeline 52

Data Engineering Podcast

JULY 16, 2021

This is particularly important in large and complex organizations where domain knowledge and context is paramount and there may not be access to engineers for codifying that expertise. There are a huge number of tools and recommended architectures for every variety of data need.

High Quality Data 100

High Quality Data Data Engineering Data Engineer Coding 100

Databand.ai

JULY 17, 2023

By adopting a set of best practices inspired by Agile methodologies, DevOps principles, and statistical process control techniques, DataOps helps organizations deliver high-quality data insights more efficiently. Learn more in our detailed guide to DataOps tools (coming soon) What Is MLOps?

Data Pipeline 52

Data Pipeline Machine Learning High Quality Data BI 52

Data Engineering Podcast

MAY 18, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management This episode is supported by Code Comments, an original podcast from Red Hat. Data lakes are notoriously complex. Data lakes are notoriously complex. My thanks to the team at Code Comments for their support.

Building 278

Building Data Lake High Quality Data Business Intelligence 278

Monte Carlo

OCTOBER 4, 2022

It’s the first book of its kind to help data engineers and analysts understand the critical factors underpinning poor data quality; apply cutting edge technologies to their existing data stacks; and build resilient, observable systems to prevent data downtime from happening in the first place.

High Quality Data 52

High Quality Data Data Pipeline Media Data 52

Databand.ai

JULY 11, 2023

There are several reasons why organizations need a data quality platform to ensure the accuracy and reliability of their data. With a data quality platform in place, decision-makers can trust the data they use, reducing the risk of costly mistakes and missed opportunities.

Data Cleanse 52

Data Cleanse Telecommunication High Quality Data BI 52

KDnuggets

MARCH 23, 2023

This article highlights the significance of ensuring high-quality data and presents six key dimensions for measuring it. These dimensions include Completeness, Consistency, Integrity, Timelessness, Uniqueness, and Validity.

High Quality Data 101

High Quality Data Data IT Data Engineering 101

Monte Carlo

JULY 27, 2023

These specialists are also commonly referred to as data reliability engineers. To be successful in their role, data quality engineers will need to gather data quality requirements (mentioned in 65% of job postings) from relevant stakeholders.

Engineering 52

Engineering Healthcare Scala Data Warehouse 52

Data Engineering Weekly

NOVEMBER 11, 2022

I took a break from data engineering and focused on system monitoring and observability for some time. Little did I know I would be building data pipelines to measure system efficiency instead of measuring business efficiency. As the famous saying goes; Once a data engineer, always a data engineer.

Systems 40

Systems High Quality Data Consulting Metadata 40

Pipeline Data Engineering

OCTOBER 15, 2021

Data engineering salon. News and interesting reads about the world of data. The First Rule of Machine Learning: Start without Machine Learning Eugene Yan, Applied Scientist, Amazon Having robust data pipelines and high-quality data labels also suggests you’re ready for machine learning.

High Quality Data 52

High Quality Data Machine Learning Kafka Data Pipeline 52

Workfall

JULY 4, 2023

Reading Time: 8 minutes In the world of data engineering, a mighty tool called DBT (Data Build Tool) comes to the rescue of modern data workflows. Imagine a team of skilled data engineers on an exciting quest to transform raw data into a treasure trove of insights.

Data Warehouse 69

Data Warehouse SQL PostgreSQL Database 69

Databand.ai

JUNE 14, 2023

The primary goal of data quality testing is to ensure that an organization’s datasets are accurate and reliable enough to support well-informed decisions. Data engineers often utilize specialized tools to perform comprehensive tests on their datasets while tracking metrics related to these aspects throughout the entire lifecycle.

Amazon Web Services 52

Amazon Web Services Datasets High Quality Data ETL Tools 52

Monte Carlo

AUGUST 17, 2023

Their proprietary training system collects data on every session at a ball-by-ball level giving players the insights they need to take their training, and their game, to the next level. TOCA’s business also needs insights from high-quality data to perform at its peak. Our implementation of Monte Carlo was very timely.

High Quality Data 52

High Quality Data Data Data Engineering Data Engineer 52

Data Engineering Weekly

MAY 19, 2024

The article about data asset pricing is one of the comprehensive thoughts I came across about pricing models, establishing two basic factors. Data value depends on the users and the use cases Data quality is multi-dimensional, and high-quality data costs more.

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Databand.ai

JULY 5, 2023

What Are Data Observability Tools? Data observability tools are software solutions that oversee, analyze, and improve the performance of data pipelines. These tools offer data engineers insight into the health of their data infrastructure, by giving visibility into crucial metrics like latency, throughput, and error rates.

Data Pipeline 52

Data Pipeline Data Lake Data Warehouse Datasets 52

Monte Carlo

DECEMBER 19, 2022

Data quality testing is the process of validating that key characteristics of a dataset match what is anticipated prior to its consumption. According to Gartner , bad data costs organizations on average an estimated $12.9 Download our Data Quality Testing 101 eBook So, what is data quality testing anyway?

High Quality Data 52

High Quality Data Data Bytes SQL 52

Monte Carlo

SEPTEMBER 22, 2022

Most employees have access to domain specific dashboards and will also engage the data team when their questions are deeper than a dashboard can answer. “We We have a sole data engineer and the rest of our team is pretty analytics focused. You can think of it as nine analytics folks and our singular hero on data engineering.

High Quality Data 52

High Quality Data Data Data Engineering Data Engineer 52

Monte Carlo

AUGUST 31, 2023

Not long after data warehouses moved to the cloud, so too did data lakes (a place to transform and store unstructured data), giving data teams even greater flexibility when it comes to managing their data assets. As data becomes more and more foundational to business, data teams will only grow.

Data Lake 52

Data Lake Data Pipeline Unstructured Data Data Warehouse 52

Data Engineering Podcast

FEBRUARY 18, 2024

In this episode Dain Sundstrom, CTO of Starburst, explains how the combination of the Trino query engine and the Iceberg table format offer the ease of use and execution speed of data warehouses with the infinite storage and scalability of data lakes. Data lakes are notoriously complex. Your first 30 days are free!

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Data Engineering Podcast

JUNE 9, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management This episode is supported by Code Comments, an original podcast from Red Hat. Data lakes are notoriously complex. Unfortunately, reasoning about stream processing engines is complex and lacks sufficient tooling.

Process 130

Process Data Lake High Quality Data Machine Learning 130

Monte Carlo

FEBRUARY 21, 2023

Despite its prevalence, data can be messy, siloed, ungovernable, and inaccessible—especially to the non-technical employees who rely on it. Enter data fabric: a data management architecture designed to serve the needs of the business, not just those of data engineers.

Data Architecture 52

Data Architecture Architecture Metadata Unstructured Data 52

Monte Carlo

FEBRUARY 21, 2023

Despite its prevalence, data can be messy, siloed, ungovernable, and inaccessible—especially to the non-technical employees who rely on it. Enter data fabric: a data management architecture designed to serve the needs of the business, not just those of data engineers.

Data Architecture 52

Data Architecture Architecture Metadata Unstructured Data 52

Data Engineering Podcast

JUNE 2, 2024

In this episode she shares the practical steps to implementing a data governance practice in your organization, and the pitfalls to avoid. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Starburst : ![Starburst

Data Governance 147

Data Governance Government Data Lake High Quality Data 147

DataKitchen

NOVEMBER 4, 2021

It often takes months to progress from a data lake to the final delivery of insights. One data engineer called it the “last mile problem.” . In our many conversations about data analytics, data engineers, analysts and scientists have verbalized the difficulty of creating analytics in the modern enterprise.

Process 98

Process Data Process Pharmaceutical Data Lake 98

Monte Carlo

JUNE 12, 2023

Like many companies, Tenable’s data team structure has shifted over time, but they currently have a centralized data platform team that includes data engineering, data science, and a data ingestion team.

Kafka 52

Kafka Algorithm Data Pipeline High Quality Data 52