Lyft Engineering

OCTOBER 3, 2023

In this post we will define data quality at a high-level and explore our motivation to achieve better data quality. We will then introduce our in-house product, Verity, and showcase how it serves as a central platform for ensuring data quality in our Hive Data Warehouse. What and Where is Data Quality?

Big Data 84

Big Data Metadata Data Warehouse Data 84

Cloudera

DECEMBER 10, 2020

Why worry about costs with cloud-native data warehousing? Have you been burned by the unexpected costs of a cloud data warehouse? If not, before adopting a cloud data warehouse, consider the true costs of a cloud-native data warehouse. These costs impede the adoption of cloud-native data warehouses.

Data Warehouse 98

Data Warehouse Cloud Storage Metadata Data 98

Data Engineering Weekly

MARCH 10, 2024

Pradheep Arjunan - Shared insights on AZ's journey from on-prem to the cloud data warehouses. Google: Croissant- a metadata format for ML-ready datasets Google Research introduced Croissant, a new metadata format designed to make datasets ML-ready by standardizing the format, facilitating easier use in machine learning projects.

Data Engineering 52

Data Engineering Data Engineer Engineering Datasets 52

Data Engineering Weekly

MAY 16, 2023

In the first part of this series, we talked about design patterns for data creation and the pros & cons of each system from the data contract perspective. In the second part, we will focus on architectural patterns to implement data quality from a data contract perspective. Why is Data Quality Expensive?

Engineering 110

Engineering Kafka Data Pipeline Data Warehouse 110

Databand.ai

JULY 19, 2023

Learn more in our detailed guide to data lineage visualization (coming soon) Integration with Multiple Data Sources Data lineage tools are designed to integrate with a wide range of data sources, including databases, data warehouses, and cloud-based data platforms.

Pipeline-centric 52

Pipeline-centric Data Governance Metadata Government 52

Cloudera

JUNE 18, 2022

These lakes power mission critical large scale data analytics, business intelligence (BI), and machine learning use cases, including enterprise data warehouses. In recent years, the term “data lakehouse” was coined to describe this architectural pattern of tabular analytics over data in the data lake.

Data Lake 90

Data Lake Data Warehouse BI SQL 90

Cloudera

JANUARY 19, 2022

Cloudera Data Platform (CDP) scored among the top 10 vendors on all four Analytical Use Cases — Data Warehouse, Logical Data Warehouse, Data Lake and Operational Intelligence in the Critical Capabilities for Cloud Database Management Systems for Analytics Use Cases.

Database 87

Database Cloud Data Warehouse Data Lake 87

Data Engineering Weekly

FEBRUARY 18, 2024

[link] Joe Reis: Everything Ends - My Journey With the Modern Data Stack Joe writes another excellent retrospect for Modern Data Stack, walking down memory lane of the early and golden days of the Modern Data Stack. We are so over the Big Data Era to Modern Data Stack.

Data Engineering 62

Data Engineering Data Engineer Engineering Data 62

Maxime Beauchemin

JANUARY 20, 2017

The fact that ETL tools evolved to expose graphical interfaces seems like a detour in the history of data processing, and would certainly make for an interesting blog post of its own. Sure, there’s a need to abstract the complexity of data processing, computation and storage.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130

DataKitchen

AUGUST 3, 2021

The data mesh design pattern breaks giant, monolithic enterprise data architectures into subsystems or domains, each managed by a dedicated team. But first, let’s define the data mesh design pattern. The past decades of enterprise data platform architectures can be summarized in 69 words.

Pharmaceutical 98

Pharmaceutical Data Lake Data Architecture Scala 98

Workfall

AUGUST 14, 2023

Two powerful tools that have emerged to simplify the management of data pipelines are DBT (Data Build Tool) and Airflow. These tools, when combined, form a dynamic duo that streamlines and scales data engineering workflows. So, let’s dive into the world of DBT and Airflow and discover how they revolutionize data pipelines.

Data Pipeline 62

Data Pipeline Raw Data Data Database 62

AltexSoft

FEBRUARY 11, 2023

In the article, we explore the role of a data architect, discuss the responsibilities and required skills, and share what kind of companies may need such a specialist. What is a data architect? Data architecture is the organization and design of how data is collected, transformed, integrated, stored, and used by a company.

Data Architect 87

Data Architect Certification Generalist Big Data 87

Monte Carlo

MARCH 3, 2021

Our SOC 2 designation means that Monte Carlo has designed a set of internal controls, systems, policies, and procedures that meet industry best practices for protecting our customer data. Specifically, with Monte Carlo, data never leaves your environment — full stop. Until then, here’s wishing you no data downtime!

Metadata 52

Metadata Data Lake Business Intelligence Certification 52

Hepta Analytics

FEBRUARY 14, 2022

DE Zoomcamp 2.2.1 – Introduction to Workflow Orchestration Following last weeks blog , we move to data ingestion. We already had a script that downloaded a csv file, processed the data and pushed the data to postgres database. This week, we got to think about our data ingestion design.

Data Ingestion 52

Data Ingestion Data Engineering Data Engineer Engineering 52

Cloudera

SEPTEMBER 17, 2020

To address this, they focused on creating an experimentation-oriented culture, enabled thanks to a cloud-native platform supporting the full data lifecycle. This platform, including an ad-hoc capable data warehouse service with built-in, easy-to-use visualization, made it easy for anyone to jump in and start experimenting.

Unstructured Data 80

Unstructured Data Data Warehouse Pharmaceutical MySQL 80

RandomTrees

SEPTEMBER 6, 2020

Snowflake Overview A data warehouse is a critical part of any business organization. Lot of cloud-based data warehouses are available in the market today, out of which let us focus on Snowflake. Snowflake is an analytical data warehouse that is provided as Software-as-a-Service (SaaS).

Cloud Storage 52

Cloud Storage Data Ingestion Data Cleanse Data Warehouse 52

Data Engineering Weekly

NOVEMBER 6, 2022

A data contract is a continuous and collaborative system because the business context and requirements won’t be static. I plan to write a series of blogs on Schemata and Data Contract in the coming weeks. Martin kindly stepped in for me to give the update for my promised blog posts.

Data Engineering 40

Data Engineering Data Engineer Engineering Machine Learning 40

Snowflake

APRIL 17, 2023

What started as a cloud data warehouse evolved into the Data Cloud, which includes the content—data, services, and applications—that customers and partners bring to the cloud. Snowflake’s Global Data Clean Rooms allow customers to “share without showing,” a concept I’ve discussed in this blog post.

Retail 69

Retail Metadata Cloud Data Warehouse 69

Databand.ai

JUNE 28, 2023

Most were designed for the best-case scenario. This might include processes like data extraction from different sources, data cleansing, data transformation (like aggregation), and loading the data into a database or a data warehouse. is a unified data observability platform built for data engineers.

Data Pipeline 98

Data Pipeline Data Engineering Data Engineer Engineering 98

Rockset

AUGUST 2, 2023

The Windward Maritime AI platform Lastly, Windward wanted to move their entire platform from batch-based data infrastructure to streaming. In this blog, we’ll describe the new data platform for Windward and how it is API first, enables rapid product iteration and is architected for real-time, streaming data.

Database-centric 52

Database-centric PostgreSQL Transportation Insurance 52

Cloudera

SEPTEMBER 15, 2022

The gathering in 2022 marked the sixteenth year for top data and analytics professionals to come to the MIT campus to explore current and future trends. A key area of focus for the symposium this year was the design and deployment of modern data platforms. What is a data fabric?

Data Lake 79

Data Lake Analytics Application Cloud Storage Architecture 79

Cloudera

FEBRUARY 11, 2019

The January 2019 “Magic Quadrant for Data Management Solutions for Analytics” provides valuable insights into the status, direction, and players in the DMSA market. In this blog, we share our takeaways as they relate to the DMSA market trends. 3. Expansion beyond core data management. Cloudera’s Positioning.

Data Management 42

Data Management Management Metadata Government 42

Netflix Tech

DECEMBER 21, 2020

By Anupom Syam Background At Netflix, our current data warehouse contains hundreds of Petabytes of data stored in AWS S3 , and each day we ingest and create additional Petabytes. Some of the optimizations are prerequisites for a high-performance data warehouse.

Data Warehouse 81

Data Warehouse Metadata Algorithm Data 81

Cloudera

AUGUST 30, 2018

Cloud service architecture – storing and accessing data and services over the Internet instead of your data center or local computer’s hard drive. Machine learning driven business – A focus on the design of systems that can learn from and make decisions and predictions based on data. Tough decisions.

Unstructured Data 40

Unstructured Data Machine Learning Data Warehouse BI 40

Cloudera

APRIL 3, 2023

In this blog, we will share with you in detail how Cloudera integrates core compute engines including Apache Hive and Apache Impala in Cloudera Data Warehouse with Iceberg. We will publish follow up blogs for other data services. Iceberg basics Iceberg is an open table format designed for large analytic workloads.

Data Warehouse 79

Data Warehouse Metadata Java Data 79

Snowflake

NOVEMBER 2, 2023

Over the years, the technology landscape for data management has given rise to various architecture patterns, each thoughtfully designed to cater to specific use cases and requirements. Customers that require a hybrid of these to support many different tools and languages have built a data lakehouse.

Data Lake 89

Data Lake Data Warehouse Cloud Unstructured Data 89

Cloudera

MAY 22, 2018

With CDSW, organizations can research and experiment faster, deploy models easily and with confidence, as well as rely on the wider Cloudera platform to reduce the risks and costs of data science projects. save the built model container, along with metadata like who built or deployed it. appeared first on Cloudera Blog.

Data Science 41

Data Science Machine Learning Metadata Big Data 41

Cloudera

FEBRUARY 8, 2024

Overview This blog post describes support for materialized views for the Iceberg table format. It has been designed and developed as an open community standard to ensure compatibility across languages and implementations. The snapshotId of the source tables involved in the materialized view are also maintained in the metadata.

Metadata 88

Metadata Data Warehouse BI AWS 88

Cloudera

NOVEMBER 6, 2017

Governance is handled evenly across all environments and applications, both for technical and business metadata. Self-service becomes easy with a unified view of who has rights to what types of data. Check back for Part 3 of 3, coming soon… The post Cloudera in the Cloud (Part 2) appeared first on Cloudera Blog.

Cloud 40

Cloud Amazon Web Services AWS Metadata 40

Cloudera

JULY 13, 2023

Iceberg is an emerging open-table format designed for large analytic workloads. Several compute engines such as Impala, Hive, Spark, and Trino have supported querying data in Iceberg table format by adopting this Java Library provided by the Apache Iceberg project. However, Iceberg Java API calls are not always cheap.

Metadata 69

Metadata Java PostgreSQL Data Warehouse 69

Airbnb Tech

NOVEMBER 28, 2023

Starting in 2020, the Midas process, along with the work to re-architect our most critical data models, has brought a dramatic increase in data quality and timeliness to Airbnb’s most critical data. While this made sense for our most critical data, pursuing such rigorous standards at scale presented challenges.

Data Warehouse 98

Data Warehouse Metadata Data Certification 98

Data Engineering Podcast

NOVEMBER 20, 2022

Summary The majority of blog posts and presentations about data engineering and analytics assume that the consumers of those efforts are internal business users accessing an environment controlled by the business. Atlan is the metadata hub for your data ecosystem.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Databand.ai

AUGUST 30, 2023

DataOps , short for data operations, is an emerging discipline that focuses on improving the collaboration, integration, and automation of data processes across an organization. By improving data quality, organizations can generate more accurate and reliable insights from their data analytics processes.

Data Cleanse 52

Data Cleanse Data Pipeline Data Ingestion Data Validation 52

Monte Carlo

FEBRUARY 9, 2023

Over the past several years, data warehouses have evolved dramatically, but that doesn’t mean the fundamentals underpinning sound data architecture needs to be thrown out the window. While data vault has many benefits, it is a sophisticated and complex methodology that can present challenges to data quality.

Architecture 52

Architecture Raw Data Metadata Data Warehouse 52

Cloudera

SEPTEMBER 9, 2018

We are proud to announce the general availability of Cloudera Altus Data Warehouse , the only cloud data warehousing service that brings the warehouse to the data. Modern data warehousing for the cloud. Modern data warehousing for the cloud. Using Cloudera Altus for your cloud data warehouse.

Data Warehouse 40

Data Warehouse Metadata Cloud Storage Cloud 40

Netflix Tech

DECEMBER 3, 2022

The most commonly used one is dataflow project , which helps folks in managing their data pipeline repositories through creation, testing, deployment and few other activities. It lets you create YAML formatted mock data files based on selected tables, columns and a few rows of data from the Netflix data warehouse.

Data Pipeline 143

Data Pipeline Scala Metadata Food 143

Cloudera

AUGUST 30, 2022

Data Lakehouse: Data lakehouses integrate and unify the capabilities of data warehouses and data lakes, aiming to support artificial intelligence, business intelligence, machine learning, and data engineering use cases on a single platform. Towards Data Science ). Forrester ).

Data Architecture 76

Data Architecture Architecture Data Lake NoSQL 76