Lyft Engineering

OCTOBER 3, 2023

We will then introduce our in-house product, Verity, and showcase how it serves as a central platform for ensuring data quality in our Hive Data Warehouse. In future posts we will discuss how Verity addresses data quality elsewhere in our data platform. What and Where is Data Quality?

Big Data 84

Big Data Metadata Data Warehouse Data 84

Airbnb Tech

NOVEMBER 28, 2023

Starting in 2020, the Midas process, along with the work to re-architect our most critical data models, has brought a dramatic increase in data quality and timeliness to Airbnb’s most critical data. While this made sense for our most critical data, pursuing such rigorous standards at scale presented challenges.

Data Warehouse 98

Data Warehouse Metadata Data Certification 98

Data Engineering Podcast

SEPTEMBER 11, 2022

In this episode he shares the design of the project and how it fits into your development practices. Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities.

Systems 100

Systems Metadata Building MongoDB 100

Data Engineering Podcast

JANUARY 7, 2022

In this episode Metaplane founder Kevin Hu shares his working definition of the term and explains the work that he and his team are doing to cut down on the time to adoption for this new set of practices. Prophecy provides an easy-to-use visual interface to design & deploy data pipelines on Apache Spark & Apache Airflow.

BI 100

BI Data Warehouse Metadata SQL 100

ProjectPro

AUGUST 11, 2021

“Data Lake vs Data Warehouse = Load First, Think Later vs Think First, Load Later” The terms data lake and data warehouse are frequently stumbled upon when it comes to storing large volumes of data. Data Warehouse Architecture What is a Data lake?

Data Lake 52

Data Lake Data Warehouse Cloud Hadoop 52

Netflix Tech

DECEMBER 3, 2022

The most commonly used one is dataflow project , which helps folks in managing their data pipeline repositories through creation, testing, deployment and few other activities. It lets you create YAML formatted mock data files based on selected tables, columns and a few rows of data from the Netflix data warehouse.

Data Pipeline 143

Data Pipeline Scala Metadata Food 143

Towards Data Science

NOVEMBER 4, 2023

Back in October, I wrote about the rise of the Data Engineer, the role, its challenges, responsibilities, daily routine and how to become successful in this field. The data engineering landscape is constantly changing but major trends seem to remain the same. Data warehouse exmaple. Consider this example below.

Data Engineering 82

Data Engineering Data Engineer Engineering BI 82

Knowledge Hut

MARCH 7, 2024

This implies two things, one, the data coming from one source is out of date when compared to another source. Two, it creates a commonality of data definitions, concepts, metadata and the like. However, regulating access is one of the primary challenges for companies who frequently work with large sets of data.

Big Data 98

Big Data Bytes Data Governance Raw Data 98

Cloudera

FEBRUARY 8, 2024

It has been designed and developed as an open community standard to ensure compatibility across languages and implementations. It brings the reliability and simplicity of SQL tables to big data while enabling engines like Hive, Impala, Spark, Trino, Flink, and Presto to work with the same tables at the same time.

Metadata 90

Metadata Data Warehouse BI AWS 90

Ascend.io

APRIL 22, 2024

This initiative is more than just an upgrade; it’s a reimagining of what a Data Automation Platform can be: dynamic, extensible, and highly intelligent. A unified platform that combines a powerful metadata core, an extensible plugin architecture, DataAware automation, and multiple AI Assistants.

Project 52

Project Metadata Data Pipeline Data Engineering 52

Data Engineering Podcast

MARCH 27, 2022

You have full control over your data and their plugin system lets you integrate with all of your other data tools, including data warehouses and SaaS platforms. What is your working definition of "data governance" and how does that influence your product focus and priorities?

Data Governance 130

Data Governance Government Cloud Building 130

Monte Carlo

AUGUST 30, 2023

Your test passes when there are no rows returned, which indicates your data meets your defined conditions. Often, teams run custom data tests as part of a deployment pipeline, or scheduled on production systems via job schedulers like Apache Airflow, dbt Cloud, or via in-built schedulers in your data warehouse solution.

SQL 52

SQL Datasets Database High Quality Data 52

Data Engineering Podcast

MAY 18, 2021

RudderStack’s smart customer data pipeline is warehouse-first. It builds your customer data warehouse and your identity graph on your data warehouse, with support for Snowflake, Google BigQuery, Amazon Redshift, and more. Interview Introduction How did you get involved in the area of data management?

Metadata 100

Metadata Kafka Data Warehouse Hadoop 100

François Nguyen

MARCH 22, 2021

TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. With this 3rd platform generation, you have more real time data analytics and a cost reduction because it is easier to manage this infrastructure in the cloud thanks to managed services. What you have to code is this workflow !

Technology 130

Technology Architecture Google Cloud Metadata 130

RandomTrees

FEBRUARY 6, 2024

Transformation: Shaping Data for the Future: LLMs facilitate standardizing date formats with precision and translation of complex organizational structures into logical database designs, streamline the definition of business rules, automate data cleansing, and propose the inclusion of external data for a more complete analytical view.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

Monte Carlo

FEBRUARY 9, 2023

Over the past several years, data warehouses have evolved dramatically, but that doesn’t mean the fundamentals underpinning sound data architecture needs to be thrown out the window. What is a Data Vault model? Pie Insurance , a leading small business insurtech, leverages a data vault 2.0 methodology.

Architecture 52

Architecture Raw Data Metadata Data Warehouse 52

Data Engineering Podcast

SEPTEMBER 19, 2021

He also discusses the technical underpinnings of what he is building and how using the data warehouse as a shared resource drastically shortens the time required to see value. That leaves DataOps reactive to data quality issues and can make your consumers lose confidence in your data.

Machine Learning 100

Machine Learning Data Warehouse Banking Metadata 100

Data Engineering Podcast

NOVEMBER 20, 2021

In the same way that application performance monitoring ensures reliable software and keeps application downtime at bay, Monte Carlo solves the costly problem of broken data pipelines. Start trusting your data with Monte Carlo today! Hightouch is the easiest way to sync data into the platforms that your business teams rely on.

Data Warehouse 100

Data Warehouse Data Lake BI Business Intelligence 100

Data Engineering Podcast

MAY 3, 2021

They explain how to think about your data systems in a holistic and maintainable fashion, the security challenges that threaten to derail your efforts, and the power of using metadata as the foundation of everything that you do. RudderStack’s smart customer data pipeline is warehouse-first.

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100

DoorDash Engineering

APRIL 12, 2023

The Metrics Layer, also known as a Semantic Layer, is a critical component of the modern data stack that has recently received significant industry attention offers a powerful solution to the challenge of standardizing metric definitions. We will also dive deep into our design and implementation processes and the lessons we learnt.

SQL 82

SQL Metadata Raw Data Government 82

Cloudera

NOVEMBER 5, 2020

Many organizations struggle to meet growing and variable data warehouse demands. This is exactly what Cloudera Data Platform (CDP) provides to the Cloudera Data Warehouse. CDP is a data platform that is optimized for both business units and central IT. . Cloudera Data Warehouse Security.

Cloud 72

Cloud Data Lake Data Warehouse Metadata 72

Monte Carlo

JANUARY 23, 2024

Every team has their own analytic domain they own, forcing decentralized team structures around broadly standardized definitions of what “good” data looks like. The data warehouse is the mirror of the organization in many ways. Data debt or data pipeline chaos. What are data engineering skills?

Data Engineering 98

Data Engineering Data Engineer Engineering Data 98

Ascend.io

FEBRUARY 28, 2024

Engineers ensure the availability of clean, structured data, a necessity for AI systems to learn from patterns, make accurate predictions, and automate decision-making processes. Through the design and maintenance of efficient data pipelines , data engineers facilitate the seamless flow and accessibility of data for AI processing.

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

Maxime Beauchemin

JANUARY 20, 2017

Data modeling is changing Typical data modeling techniques — like the star schema  — which defined our approach to data modeling for the analytics workloads typically associated with data warehouses, are less relevant than they once were.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130

Cloudera

JANUARY 21, 2021

While cloud-native, point-solution data warehouse services may serve your immediate business needs, there are dangers to the corporation as a whole when you do your own IT this way. Cloudera Data Warehouse (CDW) is here to save the day! CDW is an integrated data warehouse service within Cloudera Data Platform (CDP).

IT 93

IT Data Lake Data Warehouse Cloud Storage 93

Data Engineering Podcast

NOVEMBER 9, 2020

In this episode Grant Seward explains how he built Tree Schema to be an easy to use and cost effective option for organizations to build their data catalogs. He also shares the internal architecture, how he approached the design to make it accessible and easy to use, and how it autodiscovers the schemas and metadata for your source systems.

Building 100

Building PostgreSQL BI Metadata 100

ProjectPro

FEBRUARY 8, 2023

It offers users a data integration tool that organizes data from many sources, formats it, and stores it in a single repository, such as data lakes, data warehouses, etc., Glue uses ETL jobs for extracting data from various AWS cloud services and integrating it into data warehouses and lakes.

AWS 98

AWS Scala Metadata Data Lake 98

AltexSoft

AUGUST 22, 2022

In this post, we’ll attempt to explain the idea behind a data fabric, its architectural building blocks, the benefits it brings, and ways to approach its implementation. What is data fabric? to provide a unified view of all enterprise data. Data fabric architecture example. Data fabric architecture building blocks.

Architecture 52

Architecture Metadata Data Lake Machine Learning 52

dbt Developer Hub

JULY 26, 2022

Ultimately, this looks like people being able to interact with trusted datasets in the tools that they are comfortable with (and eventually new tools designed specifically around metrics). Best of all, definition changes would get reflected in downstream tools, so you no longer need to manually search and update every downstream dependency.

Metadata 52

Metadata BI Datasets SQL 52

Knowledge Hut

JULY 24, 2023

The ability of a DBMS to change its schema definition at one level without affecting the schema definition at the next level is called data independence. But why do we need data integrity in a DBMS? In addition to data entered by users, database systems typically store large amounts of data.

Database Design 52

Database Design Relational Database Metadata Database 52

Data Engineering Weekly

JANUARY 15, 2023

SiliconANGLE theCUBE: Analyst Predictions 2023 - The Future of Data Management By far one of the best analyses of trends in Data Management. 2023 predictions from the panel are; Unified metadata becomes kingmaker. The blog narrates the causal machine-learning approach in designing promotional artwork for their TV shows.

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

AltexSoft

NOVEMBER 23, 2021

Before we get into more detail, let’s determine how data virtualization is different from another, more common data integration technique — data consolidation. Data virtualization vs data consolidation. The example of a typical two-tier architecture with a data lake and data warehouses and several ETL processes.

Process 69

Process Data Lake Metadata Data Warehouse 69

Cloudera

SEPTEMBER 15, 2022

The gathering in 2022 marked the sixteenth year for top data and analytics professionals to come to the MIT campus to explore current and future trends. A key area of focus for the symposium this year was the design and deployment of modern data platforms. What is a data fabric?

Data Lake 81

Data Lake Analytics Application Cloud Storage Architecture 81

Grouparoo

OCTOBER 6, 2021

This provided a nice overview of the breadth of topics that are relevant to data engineering including data warehouses/lakes, pipelines, metadata, security, compliance, quality, and working with other teams. For example, grouping the ones about metadata, discoverability, and column naming might have made a lot of sense.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

AltexSoft

MARCH 14, 2023

As the volume and complexity of data continue to grow, organizations seek faster, more efficient, and cost-effective ways to manage and analyze data. In recent years, cloud-based data warehouses have revolutionized data processing with their advanced massively parallel processing (MPP) capabilities and SQL support.

IT 59

IT Data Warehouse Data Governance Data Lake 59

Data Engineering Weekly

NOVEMBER 6, 2022

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Thanks to David Jayatillake for setting up data-folks.masto.host.

Data Engineering 40

Data Engineering Data Engineer Engineering Machine Learning 40

Netflix Tech

MARCH 25, 2019

Finally, imagine yourself in the role of a data platform reliability engineer tasked with providing advanced lead time to data pipeline (ETL) owners by proactively identifying issues upstream to their ETL jobs. Design a flexible data model ? —?Represent Enable seamless integration?—? push or pull.

Building 87

Building Metadata Transportation Data Ingestion 87