Data Engineering Weekly

MARCH 10, 2024

Pradheep Arjunan - Shared insights on AZ's journey from on-prem to the cloud data warehouses. Google: Croissant- a metadata format for ML-ready datasets Google Research introduced Croissant, a new metadata format designed to make datasets ML-ready by standardizing the format, facilitating easier use in machine learning projects.

Data Engineering 52

Data Engineering Data Engineer Engineering Datasets 52

Data Engineering Weekly

MAY 16, 2023

In the second part, we will focus on architectural patterns to implement data quality from a data contract perspective. Why is Data Quality Expensive? I won’t bore you with the importance of data quality in the blog. Event Routers can add additional metadata to the envelope of the event. How to Fix It?

Engineering 110

Engineering Kafka Data Pipeline Data Warehouse 110

Cloudera

DECEMBER 10, 2020

Why worry about costs with cloud-native data warehousing? Have you been burned by the unexpected costs of a cloud data warehouse? If not, before adopting a cloud data warehouse, consider the true costs of a cloud-native data warehouse. These costs impede the adoption of cloud-native data warehouses.

Data Warehouse 98

Data Warehouse Cloud Storage Metadata Data 98

dbt Developer Hub

OCTOBER 30, 2023

In this blog post, I’ll attempt to provide this guidance by answering these FAQs: What is dbt clone ? that leverages native zero-copy clone functionality on supported warehouses to copy entire schemas for free, almost instantly. On the surface, cloning and deferral seem similar – they’re both ways to save costs in the data warehouse.

BI 52

BI Datasets Metadata Data Warehouse 52

Cloudera

MARCH 17, 2023

I love the enthusiasm surrounding ChatGPT and the eagerness to learn about modern data architectures such as data lakehouses, data meshes, and data fabrics. ChatGPT is an excellent resource for gaining high-level insights and building awareness of any technology. I thought this was a fairly comprehensive list.

Education 61

Education Unstructured Data Data Lake Data Warehouse 61

Cloudera

JANUARY 19, 2022

Cloudera Data Platform (CDP) scored among the top 10 vendors on all four Analytical Use Cases — Data Warehouse, Logical Data Warehouse, Data Lake and Operational Intelligence in the Critical Capabilities for Cloud Database Management Systems for Analytics Use Cases.

Database 87

Database Cloud Data Warehouse Data Lake 87

Maxime Beauchemin

JANUARY 20, 2017

They’re highly analytical, and are interested in data visualization. Unlike data scientists — and inspired by our more mature parent, software engineering  — data engineers build tools, infrastructure, frameworks, and services. Sure, there’s a need to abstract the complexity of data processing, computation and storage.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130

Cloudera

SEPTEMBER 29, 2022

This team or domain expert will be responsible for the data produced by the team. The data itself is then treated as a product. The data product is not just the data itself, but a bunch of metadata that surrounds it — the simple stuff like schema is a given. What is a data mesh contract?

Architecture 73

Architecture Data Architecture Metadata Data Warehouse 73

Data Engineering Weekly

FEBRUARY 18, 2024

Benn Stancil: The problem was the product - How the modern data stack got lost Generated via DallE-3 The week is full of retrospect for the Modern Data Stack :-) The author shares a unique point of view on how the lack of collaboration among the MDS companies leads to its loss of popularity. Elevate your data skills! It is true.

Data Engineering 62

Data Engineering Data Engineer Engineering Data 62

Data Engineering Weekly

OCTOBER 30, 2022

We saw in the Data Catalog poll how far it has to go to be helpful and active within a data workflow. The author narrates the data contract, why it is the foundation for building data mesh, and how it glue data catalogs, glossary and data quality systems.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Ingestion 52

Data Engineering Weekly

JULY 2, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork, and SQL grunt work out of building complete customer profiles, so you can quickly ship actionable, enriched data to every downstream team. Find the right, trusted data by simply asking a question.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

DataKitchen

AUGUST 3, 2021

The past decades of enterprise data platform architectures can be summarized in 69 words. First-generation – expensive, proprietary enterprise data warehouse and business intelligence platforms maintained by a specialized team drowning in technical debt. Data professionals are not perfectly interchangeable.

Pharmaceutical 98

Pharmaceutical Data Lake Data Architecture Scala 98

Workfall

AUGUST 14, 2023

These pipelines ensure that data flows smoothly from various sources to its intended destinations, enabling businesses to make informed decisions and gain valuable insights. Two powerful tools that have emerged to simplify the management of data pipelines are DBT (Data Build Tool) and Airflow. What is Airflow?

Data Pipeline 62

Data Pipeline Raw Data Data Database 62

Hepta Analytics

FEBRUARY 14, 2022

DE Zoomcamp 2.2.1 – Introduction to Workflow Orchestration Following last weeks blog , we move to data ingestion. We already had a script that downloaded a csv file, processed the data and pushed the data to postgres database. This week, we got to think about our data ingestion design.

Data Ingestion 52

Data Ingestion Data Engineering Data Engineer Engineering 52

Cloudera

DECEMBER 21, 2023

In the “assumptions” field, we see how the SQL AI Assistant looked over our data model; compared to what we’re looking for, it was able to find the right tables, columns, and joins needed to provide a query that will give us the list we’re looking for. And as a bonus, we even get the query written for us, saving us even more time!

SQL 66

SQL Data Warehouse Business Analyst Data Schemas 66

Monte Carlo

MARCH 3, 2021

After all, any solution dedicated to keeping data pipelines reliable should take its own medicine when it comes to building the actual product. Instead, we only extract query logs, metadata, and aggregated statistics about data usage to ensure that your most critical data assets are as trustworthy and reliable as possible.

Metadata 52

Metadata Data Lake Business Intelligence Certification 52

AltexSoft

FEBRUARY 11, 2023

It serves as a foundation for the entire data management strategy and consists of multiple components including data pipelines; , on-premises and cloud storage facilities – data lakes , data warehouses , data hubs ;, data streaming and Big Data analytics solutions ( Hadoop , Spark , Kafka , etc.);

Data Architect 87

Data Architect Certification Generalist Big Data 87

Cloudera

SEPTEMBER 17, 2020

To address this, they focused on creating an experimentation-oriented culture, enabled thanks to a cloud-native platform supporting the full data lifecycle. This platform, including an ad-hoc capable data warehouse service with built-in, easy-to-use visualization, made it easy for anyone to jump in and start experimenting.

Unstructured Data 80

Unstructured Data Data Warehouse Pharmaceutical MySQL 80

Snowflake

APRIL 17, 2023

What started as a cloud data warehouse evolved into the Data Cloud, which includes the content—data, services, and applications—that customers and partners bring to the cloud. But we’re also seeing more interest in acquiring external data and finding ways to share data with others.

Retail 69

Retail Metadata Cloud Data Warehouse 69

Rockset

AUGUST 2, 2023

To support this, Windward’s data platform needed to enable rapid iteration so they could quickly innovate and build more AI capabilities. The Windward Maritime AI platform Lastly, Windward wanted to move their entire platform from batch-based data infrastructure to streaming.

Database-centric 52

Database-centric PostgreSQL Transportation Insurance 52

Cloudera

FEBRUARY 16, 2022

Weak model lineage can result in reduced model performance, a lack of confidence in model predictions and potentially violation of company, industry or legal regulations on how data is used. . Within the CML data service, model lineage is managed and tracked at a project level by the SDX. Figure 03: lineage.yaml.

Machine Learning 81

Machine Learning Algorithm Government Metadata 81

Data Engineering Weekly

NOVEMBER 6, 2022

If you’re a data professional, please join at data-folks.masto.host. I’m at ananth@data-folks.masto.host. I’m following most of the data professionals, so you can easily build your network from my following list. I plan to write a series of blogs on Schemata and Data Contract in the coming weeks.

Data Engineering 40

Data Engineering Data Engineer Engineering Machine Learning 40

ProjectPro

JANUARY 24, 2023

It is a popular ETL tool well-suited for big data environments and extensively used by data engineers today to build and maintain data pipelines with minimal effort. Furthermore, AWS Glue DataBrew allows you to visually clean and normalize data without any code.

AWS 52

AWS Data Lake Scala ETL Tools 52

Monte Carlo

SEPTEMBER 12, 2022

But with the rise of tools such as Segment, Fivetran, Meltano, and Airbyte, it’s become relatively easy for teams to bring all of their data from external sources into a centralized place like a data warehouse. He continues to share his takes on the Preset blog , and is excited to see what happens in the years to come.

BI 52

BI Data Governance ETL Tools Data Warehouse 52

dbt Developer Hub

AUGUST 23, 2022

Those who have been building data warehouses for a long time have undoubtedly encountered the challenge of building surrogate keys on their data models. This configuration allows you to define any metadata dictionary that you want. This can cause adverse performance during table builds. Let’s dive in.

Datasets 52

Datasets Metadata SQL Data Warehouse 52

Cloudera

MAY 12, 2021

Use cases could include but are not limited to: workload analysis and replication, migrating or bursting to cloud, data warehouse optimization, and more. The post Announcing the 2021 Data Impact Awards appeared first on Cloudera Blog. SECURITY AND GOVERNANCE LEADERSHIP. INDUSTRY TRANSFORMATION.

Food 66

Food Healthcare Government Medical 66

Snowflake

NOVEMBER 2, 2023

Over the years, the technology landscape for data management has given rise to various architecture patterns, each thoughtfully designed to cater to specific use cases and requirements. These patterns include both centralized storage patterns like data warehouse , data lake and data lakehouse , and distributed patterns such as data mesh.

Data Lake 89

Data Lake Data Warehouse Cloud Unstructured Data 89

Cloudera

MAY 22, 2018

Access any data anywhere – from cloud object storage to data warehouses, CDSW provides connectivity not only to CDH but the systems your data science teams rely on for analysis. build and execute the training run in an isolated container. The post Now Available: Cloudera Data Science Workbench Release 1.4

Data Science 41

Data Science Machine Learning Metadata Big Data 41

Cloudera

FEBRUARY 8, 2024

Overview This blog post describes support for materialized views for the Iceberg table format. It brings the reliability and simplicity of SQL tables to big data while enabling engines like Hive, Impala, Spark, Trino, Flink, and Presto to work with the same tables at the same time. Starting from the CDW Public Cloud DWX-1.6.1

Metadata 88

Metadata Data Warehouse BI AWS 88

Cloudera

NOVEMBER 6, 2017

The newer, second option is to leverage Cloudera Altus “data engineering-as-a-service,” which simplifies the experience by eliminating even the initial steps of installing software or configuring clusters. Instead, users can immediately begin to build data pipelines and run jobs for their applications.

Cloud 40

Cloud Amazon Web Services AWS Metadata 40

Data Engineering Podcast

NOVEMBER 20, 2022

Summary The majority of blog posts and presentations about data engineering and analytics assume that the consumers of those efforts are internal business users accessing an environment controlled by the business. Atlan is the metadata hub for your data ecosystem.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Cloudera

FEBRUARY 8, 2021

This is part 2 in this blog series. You can read part 1, here: Digital Transformation is a Data Journey From Edge to Insight. The first blog introduced a mock connected vehicle manufacturing company, The Electric Car Company (ECC), to illustrate the manufacturing data path through the data lifecycle.

Data Pipeline 114

Data Pipeline Building Manufacturing Data Warehouse 114

dbt Developer Hub

JANUARY 8, 2024

OK,” you think further, “I’ll just run a dbt build -s +my_changed_model to make sure I have everything I need built into my dev schema and I can test my changes”. Luckily, dbt’s defer functionality allow you to only build what you care about when you need it, and nothing more. You run the command. You wait some more. That’s no good!

Coding 59

Coding Metadata Cloud Healthcare 59

Cloudera

SEPTEMBER 9, 2018

We are proud to announce the general availability of Cloudera Altus Data Warehouse , the only cloud data warehousing service that brings the warehouse to the data. Modern data warehousing for the cloud. Modern data warehousing for the cloud. Using Cloudera Altus for your cloud data warehouse.

Data Warehouse 40

Data Warehouse Metadata Cloud Storage Cloud 40

Netflix Tech

DECEMBER 3, 2022

The most commonly used one is dataflow project , which helps folks in managing their data pipeline repositories through creation, testing, deployment and few other activities. It lets you create YAML formatted mock data files based on selected tables, columns and a few rows of data from the Netflix data warehouse.

Data Pipeline 143

Data Pipeline Scala Metadata Food 143

Netflix Tech

MARCH 25, 2019

Building and Scaling Data Lineage at Netflix to Improve Data Infrastructure Reliability, and Efficiency By: Di Lin , Girish Lingappa , Jitender Aswani Imagine yourself in the role of a data-inspired decision maker staring at a metric on a dashboard about to make a critical business decision but pausing to ask a question?—?“Can

Building 87

Building Metadata Transportation Data Ingestion 87

Monte Carlo

FEBRUARY 9, 2023

Over the past several years, data warehouses have evolved dramatically, but that doesn’t mean the fundamentals underpinning sound data architecture needs to be thrown out the window. While data vault has many benefits, it is a sophisticated and complex methodology that can present challenges to data quality.

Architecture 52

Architecture Raw Data Metadata Data Warehouse 52