Data Engineering Digest

Building DoorDash’s Product Knowledge Graph with Large Language Models

DoorDash Engineering

APRIL 23, 2024

When a merchant comes onboard at DoorDash, we add their internal SKU data — raw merchant data — to our retail catalog. SKU data from different merchants come in varying formats and quality; they may, for example, have missing or incorrect attribute values. Examples include OpenAI’s GPT-4, Google’s Bard, and Meta’s Llama.

Building

Building Retail Manufacturing Unstructured Data

A New Horizon for Data Reliability With Monte Carlo and Snowflake

Monte Carlo

JANUARY 29, 2024

It’s one thing to get your data into a modern data cloud. Monte Carlo is thrilled to be part of the Snowflake Horizon partner ecosystem as we leverage many of the pre-built features Snowflake provides in order to help organizations reduce their data downtime and improve data quality at scale.

Metadata

Metadata High Quality Data Data Pipeline Machine Learning

Just Launched: Data Products

Monte Carlo

MAY 16, 2024

At what level does it make sense to deploy your data monitoring coverage? For most traditional data quality tools the answer has been by table. Data teams are onboarding datasets quickly and with the power of modern platforms, that process has only been getting faster and volumes only getting bigger.

Datasets

Datasets Data Finance AWS

Webinars

Generative AI Deep Dive: Advancing from Proof of Concept to Production

Understanding User Needs and Satisfying Them

Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You Need to Know

Leading the Development of Profitable and Sustainable Products

How To Get Promoted In Product Management

MORE WEBINARS

Data Catalog vs. Data Dictionary: 3 Essential Differences

Monte Carlo

MAY 20, 2024

Much like a confused tourist trying to differentiate between an alligator and a crocodile from a safe distance, you might squint at a data catalog vs. data dictionary and wonder, “Aren’t they just two boring lists of data stuff?” It is primarily concerned with the accuracy and proper usage of data at a micro level.

Datasets

Datasets Government Data Business Analyst

Metadata Management and Data Governance with Cloudera SDX

Cloudera

JANUARY 26, 2024

In this article, we will walk you through the process of implementing fine grained access control for the data governance framework within the Cloudera platform. In a good data governance strategy, it is important to define roles that allow the business to limit the level of access that users can have to their strategic data assets.

Metadata

Metadata Data Governance Government Management

Improving the code quality of your dbt models with unit tests and TDD

Towards Data Science

JUNE 2, 2023

How to improve the code quality of your dbt models with unit tests and TDD All you need to know to start unit testing your dbt SQL models Photo by Christin Hume on Unsplash If you are a data or analytics engineer, you are probably comfortable writing SQL models and testing for data quality with dbt tests.

Coding

Coding SQL Software Engineer Software Engineering

What is Angularjs? Applications and Advantages

Knowledge Hut

MAY 1, 2024

Instead of having to load and fetch separate pages from the memory as and when requested by the user every time they visit the site, a single page website would mean less overload on the memory and improved quality of user-experience. This way, the developer can concentrate on building quality applications without worrying about integration.

Java

Java Coding Python Designing

Metadata Management & Data Governance with Cloudera SDX

Cloudera

MARCH 4, 2024

In this article, we will walk you through the process of implementing fine grained access control for the data governance framework within the Cloudera platform. In a good data governance strategy, it is important to define roles that allow the business to limit the level of access that users can have to their strategic data assets.

Metadata

Metadata Data Governance Government Management

How DoorDash Migrated from StatsD to Prometheus

DoorDash Engineering

AUGUST 1, 2023

Just when we most needed observability data, the system would leave us in the lurch. Challenges Faced With StatsD StatsD was a great asset for our early observability needs, but we began encountering constraints such as losing metrics during surge events, difficulties with naming/standardized tags, and a lack of reporting tools.

AWS

AWS Transportation Programming Language Government

Announcing New Innovations for Snowflake Horizon

Snowflake

NOVEMBER 2, 2023

Snowflake’s single, cross-cloud governance model has always been a powerful differentiator, enabling customers to manage their increasingly complex data ecosystems with simplicity and ease. As a result, Snowflake is enhancing its governance capabilities that thousands of customers already rely on through Snowflake Horizon.

Metadata

Metadata Government AWS Medical

From Big Data to Better Data: Ensuring Data Quality with Verity

Lyft Engineering

OCTOBER 3, 2023

High-quality data is necessary for the success of every data-driven company. It is now the norm for tech companies to have a well-developed data platform. This makes it easy for engineers to generate, transform, store, and analyze data at the petabyte scale. Yet this has come at the cost of quality.

Big Data

Big Data Metadata Data Warehouse Data

The very strange way of doing Data Quality at Airbnb

François Nguyen

JANUARY 23, 2021

or why you should have a look at Data Observability ! This article is the second part on how Airbnb is managing data quality : “Part 2 — A New Gold Standard” The first part can be found here and it was just good principles about roles and responsabilities. Welcome to this strange way of managing data quality !

Finance

Finance Certification Data Pipeline Data

The Complete Front-End Developer Roadmap 2024

Knowledge Hut

DECEMBER 29, 2023

This markup language is made up of tags that label or indicate what the content should appear as. The HTML opening tag (<html>) and the HTML closing tag (</html>) indicate the start and end of a HTML structure. The “button” tag is used to create a button with the name “Click Me!”

Portfolio

Portfolio Amazon Web Services Programming Language Coding

The Past, Present, and Future of Data Quality Management: Understanding Testing, Monitoring, and Data Observability in 2024

Monte Carlo

MAY 22, 2024

Data quality monitoring. Data testing. Data observability. Like everything in data engineering, data quality management is evolving at lightning speed. Table of Contents Understanding the modern data quality problem So, what’s the difference between data monitoring, testing, and observability?

Management

Management Data Datasets Machine Learning

How to identify your business-critical data

Towards Data Science

JUNE 16, 2023

How to Identify Your Business-Critical Data Practical steps to identifying business-critical data models and dashboards and drive confidence in your data Source: synq.io This article has been co-written with Lindsay Murphy Not all data is created equal. There’s a good reason for this. Source: synq.io

BI

BI Data ETL Tools Machine Learning

4 Native Snowflake Data Quality Checks & Features You Should Know

Monte Carlo

APRIL 21, 2022

Adopting a cloud data warehouse like Snowflake is an important investment for any organization that wants to get the most value out of their data. When data quality is neglected, data teams end up spending valuable time responding to broken dashboards and unreliable reports. Data can be stale or duplicative.

Metadata

Metadata Bytes Government Data

How Pie Insurance Created a Self-Serve Incident Triaging & Resolution Workflow with Monte Carlo and Slack

Monte Carlo

NOVEMBER 30, 2023

With more than 30 years of experience managing data at organizations like Ticketmaster and Eventbrite, Ed Presz considers himself a veteran of the data space. But when he left Eventbrite to lead the data team at Pie Insurance , that meant learning a new industry with unique challenges. There are compliance risks.

Insurance

Insurance Data Engineering Data Engineer SQL

How to Build a Chatbot Using Retrieval Augmented Generation (RAG)

Rockset

MAY 21, 2024

How RAG Works RAG extends the capabilities of an LLM beyond its initial training data by providing more accurate and up-to-date responses. When a prompt is given to the LLM, RAG first uses the prompt to pull relevant information from an external data source.

Building

Building Utilities Datasets Database

Data Governance Series: Part 2 – The Importance of Data Taxonomy

The Modern Data Company

MARCH 7, 2023

Data governance can be a powerful agent in scaling the use and distribution of trusted data throughout the company. If you missed it, make sure to catch up on Part 1 – Data Timeliness. What Is Data Taxonomy? Data that is properly classified, catalogued, and tagged is usually well-governed data.

Data Governance

Data Governance Government Metadata Datasets

Migrating Netflix to GraphQL Safely

Netflix Tech

JUNE 14, 2023

Testing Strategies: A Summary Two key factors determined our testing strategies: Functional vs. non-functional requirements Idempotency If we were testing functional requirements like data accuracy, and if the request was idempotent , we relied on Replay Testing. In such cases, we were not testing for response data but overall behavior.

Utilities

Utilities Java Systems AWS

Extracting skills from content to fuel the LinkedIn Skills Graph

LinkedIn Engineering

DECEMBER 13, 2023

There are also LinkedIn Learning courses where only a subset of skills are tagged directly, with a number of relevant skills being mentioned solely in the description or in the course itself (i.e., data analytics” and “data analysis” are the same type of skill), which we represent in our skills taxonomy.

Recruitment

Recruitment Utilities Designing Java

Yelp Content As Embeddings

Yelp Engineering

APRIL 19, 2023

Yelp aims to offer easily accessible high-quality content. We need to tag, organize and rank online content to attain this goal. For this purpose, Yelp engineers have started using general embeddings on different data. It improves usability and efficiency for all kinds of model development.

Deep Learning

Deep Learning Accessible Accessibility Engineering

How Pie Insurance Created a Self-Serve Incident Triaging & Resolution Workflow with Monte Carlo and Slack

Monte Carlo

NOVEMBER 30, 2023

With more than 30 years of experience managing data at organizations like Ticketmaster and Eventbrite, Ed Presz considers himself a veteran of the data space. But when he left Eventbrite to lead the data team at Pie Insurance , that meant learning a new industry with unique challenges. There are compliance risks.

Insurance

Insurance Data Engineering Data Engineer SQL

Data Triage: 4 Best Practices For Stronger Data Incident Management

Monte Carlo

JANUARY 9, 2024

Senior ops engineer Sean McAfee can recall one of the incidents that served as a catalyst for standardizing the data triage and incident management strategy for the data team at Seasoned , a job board and hiring app specifically designed for the service industry. Table of Contents Systemic challenges to effective data triaging 1.

Management

Management Data Datasets Data Pipeline

Movie Recommendation System: Definition, Strategies, Usecase

Knowledge Hut

FEBRUARY 1, 2024

Today, we’ll talk about how Machine Learning (ML) can be used to build a movie recommendation system - from researching data sets & understanding user preferences all the way through training models & deploying them in applications. Let’s begin with all the aspects of movie recommendation using ML.

Systems

Systems Entertainment Algorithm Datasets

Building and maintaining the skills taxonomy that powers LinkedIn's Skills Graph

LinkedIn Engineering

MARCH 21, 2023

However, this type of ambiguous relation would violate our structured skills quality guardrails and would not be included. This dual approach helps grow the taxonomy at scale while ensuring the skills data meets our required quality and standards. This can introduce different types of noise and varying data quality.

Building

Building Recruitment Machine Learning Deep Learning

Lineage + Hamilton in 10 minutes

Towards Data Science

MAY 26, 2023

It is great for data & Machine Learning (ML) work. In this post, we’re going to walk you through Hamilton’s lineage capabilities that can help you quickly answer common questions that arise while working with data & ML, so that you can work more efficiently and collaborate more effectively with colleagues.

Metadata

Metadata Data Science Coding Machine Learning

Data Labeling in Machine Learning: Process, Types, and Best Practices

AltexSoft

DECEMBER 21, 2021

Computer systems have limited capabilities without human guidance, and data labeling is the way to teach them to become “smart.” ” In this article, you will find out what data labeling is, how it works, which data labeling types exist, and what best practices to follow to make this process smooth as glass.

Machine Learning

Machine Learning Process Raw Data Datasets

How we cut our tests by 80% while increasing data quality: the power of aggregating test failures in dbt

dbt Developer Hub

JANUARY 23, 2023

Testing the quality of data in your warehouse is an important aspect in any mature data pipeline. One of the biggest blockers for developing a successful data quality pipeline is aggregating test failures and successes in an informational and actionable way. However, ensuring actionability can be challenging.

Metadata

Metadata High Quality Data SQL Data Integration

A Complete Guide to Scale Your Data Pipelines and Data Products with Contract Testing and Dbt

Towards Data Science

OCTOBER 25, 2023

Not too long ago, almost all data architectures and data team structures followed a centralized approach. As a data or analytics engineer, you knew where to find all the transformation logic and models because they were all in the same codebase. There was only one data team, two at most.

Data Pipeline

Data Pipeline SQL Data Architecture Data

Getting ready for AI in government: a guide for data-literate leaders by Jessica McEvoy

Scott Logic

OCTOBER 30, 2023

At the time there were thousands of untagged pages of content that we estimated would take years for civil servants across government to review and tag. The project team built a supervised machine learning model that could recognise patterns in the pages, read the content, and suggest suitable tags.

Government

Government Hospitality Database-centric Machine Learning

Writing Effective User Stories in JIRA

Knowledge Hut

OCTOBER 24, 2023

The acceptance criteria also help the development team implement the business rules, and functionality and will be the single point of reference for the Quality Assurance Team. Often, user stories are accompanied by a process diagram, UI wireframe or a mockup, data dictionary, etc., Boost your career with PMP certs.

Project

Project Systems Designing Management

Upgrade your Modern Data Stack

Christophe Blefari

SEPTEMBER 28, 2023

Make your data stack take-off ( credits ) Hello, another edition of Data News. This week, we're going to take a step back and look at the current state of data platforms. What are the current trends and why are people fighting around the concept of the modern data stack. Is the modern data stack dying?

Cloud Storage

Cloud Storage Big Data Hadoop SQL

Top Data Quality Alert Strategies From 3 Real Data Teams

Monte Carlo

MARCH 28, 2024

When it comes to sweet treats, sunshine, and errant data quality alerts, the answer is – unfortunately – yes. Alerts for data quality incidents, like anomalies in the freshness, volume, or schema of your tables, are essential for maintaining the health and accuracy of the data flowing through your pipelines.

Data

Data Data Engineering Data Engineer Engineering

Supporting content decision makers with machine learning

Netflix Tech

DECEMBER 10, 2020

Executives consider many factors including narrative quality, relation to the current societal context or zeitgeist, creative talent relationships, and audience composition and size, to name a few. genre, runtime, series or film) as well as tags or text summaries curated by domain experts describing thematic/plot elements.

Machine Learning

Machine Learning Entertainment Metadata Consulting

5 Data Quality Tools—What They Are & When You Need Them

Monte Carlo

MARCH 13, 2024

It’s rare to find a data leader, engineer, scientist, or analyst — or business stakeholder — who doesn’t believe data quality is important. When your company relies on data to power products or decision-making processes, everyone agrees that data needs to be accurate and trustworthy. And there’s a huge difference.

Data Governance

Data Governance Government Metadata Data

Natural Language Processing: A Guide to NLP Use Cases, Approaches, and Tools

AltexSoft

AUGUST 25, 2021

Specifics of data used in NLP. Both in daily life and in business, we deal with massive volumes of unstructured text data : emails, legal documents, product reviews, tweets, etc. Another way to handle unstructured text data using NLP is information extraction (IE). Rule-based NLP — great for data preprocessing.

Process

Process Deep Learning Datasets Machine Learning

7 Best Practices to Use While Annotating Images

AltexSoft

AUGUST 3, 2021

No matter how big or small your machine learning (ML) project might be, the overall output depends on the quality of data used to train the ML models. Data annotation plays a pivotal role in the process. That said, data annotation is key in training ML models if you want to achieve high-quality outputs.

Datasets

Datasets High Quality Data Metadata Raw Data

Top Data Catalog Tools

Monte Carlo

FEBRUARY 26, 2024

A data catalog is a constantly updated inventory of the universe of data assets within an organization. It uses metadata to create a picture of the data, as well as the relationships between data assets of diverse sources, and the processing that takes place as data moves through systems.

Metadata

Metadata Government Data Data Governance

Costwiz: Saving cost for LinkedIn enterprise on Azure

LinkedIn Engineering

JULY 27, 2023

Costwiz provides a unified experience that helps leaders drive more accurate forecasting of Azure budgets at LinkedIn with resource ownership detection, accountability, expedited remedies, and holistic data visibility (via custom dashboards). based on particular workflow requirements.

Metadata

Metadata Utilities Cloud Data Lake

The Best Way to Learn React in 2024 [Complete Roadmap]

Knowledge Hut

DECEMBER 22, 2023

Pros: Remarkably simple to use It has good documentation It supports virtual DOM Manipulation and 2-way Data binding Unit testing can be efficiently done Good community s upport Cons: It is not suitable for large - scale projects Limited resources Not immensely popular among developers Official Documentation Link.

Coding

Coding Python Programming Language Certification

Data Engineering Weekly #133

Data Engineering Weekly

JUNE 4, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Do you need data/ metrics at all? Sign up free to test out the tool today.

Data Engineering

Data Engineering Data Engineer Engineering Medical

Data Lake Explained: A Comprehensive Guide to Its Architecture and Use Cases

AltexSoft

AUGUST 29, 2023

In 2010, a transformative concept took root in the realm of data storage and analytics — a data lake. The term was coined by James Dixon , Back-End Java, Data, and Business Intelligence Engineer, and it started a new era in how organizations could store, manage, and analyze their data. What is a data lake?

Data Lake

Data Lake Architecture IT Amazon Web Services

Monte Carlo and dbt Labs Announce Partnership to Help Analytics Engineering Teams Achieve More Reliable Data

Monte Carlo

AUGUST 16, 2022

When it comes to trusting your data, Monte Carlo, the creator of the data observability category, and dbt Labs , creators of dbt, are better together. What went wrong with my data?!” If you’ve been on the receiving end of a broken data pipeline, these questions probably look familiar to you. Why didn’t my job run?”

Engineering

Engineering Data Lake Retail BI

Building DoorDash’s Product Knowledge Graph with Large Language Models

A New Horizon for Data Reliability With Monte Carlo and Snowflake

Webinars

Trending Sources

Just Launched: Data Products

Webinars

Data Catalog vs. Data Dictionary: 3 Essential Differences

Metadata Management and Data Governance with Cloudera SDX

Improving the code quality of your dbt models with unit tests and TDD

What is Angularjs? Applications and Advantages

Metadata Management & Data Governance with Cloudera SDX

How DoorDash Migrated from StatsD to Prometheus

Announcing New Innovations for Snowflake Horizon

From Big Data to Better Data: Ensuring Data Quality with Verity

The very strange way of doing Data Quality at Airbnb

The Complete Front-End Developer Roadmap 2024

The Past, Present, and Future of Data Quality Management: Understanding Testing, Monitoring, and Data Observability in 2024

How to identify your business-critical data

4 Native Snowflake Data Quality Checks & Features You Should Know

How Pie Insurance Created a Self-Serve Incident Triaging & Resolution Workflow with Monte Carlo and Slack

How to Build a Chatbot Using Retrieval Augmented Generation (RAG)

Data Governance Series: Part 2 – The Importance of Data Taxonomy

Migrating Netflix to GraphQL Safely

Extracting skills from content to fuel the LinkedIn Skills Graph

Yelp Content As Embeddings

How Pie Insurance Created a Self-Serve Incident Triaging & Resolution Workflow with Monte Carlo and Slack

Data Triage: 4 Best Practices For Stronger Data Incident Management

Movie Recommendation System: Definition, Strategies, Usecase

Building and maintaining the skills taxonomy that powers LinkedIn's Skills Graph

Lineage + Hamilton in 10 minutes

Data Labeling in Machine Learning: Process, Types, and Best Practices

How we cut our tests by 80% while increasing data quality: the power of aggregating test failures in dbt

A Complete Guide to Scale Your Data Pipelines and Data Products with Contract Testing and Dbt

Getting ready for AI in government: a guide for data-literate leaders by Jessica McEvoy

Writing Effective User Stories in JIRA

Upgrade your Modern Data Stack

Top Data Quality Alert Strategies From 3 Real Data Teams

Supporting content decision makers with machine learning

5 Data Quality Tools—What They Are & When You Need Them

Natural Language Processing: A Guide to NLP Use Cases, Approaches, and Tools

7 Best Practices to Use While Annotating Images

Top Data Catalog Tools

Costwiz: Saving cost for LinkedIn enterprise on Azure

The Best Way to Learn React in 2024 [Complete Roadmap]

Data Engineering Weekly #133

Data Lake Explained: A Comprehensive Guide to Its Architecture and Use Cases

Monte Carlo and dbt Labs Announce Partnership to Help Analytics Engineering Teams Achieve More Reliable Data

Stay Connected