phData: Data Engineering

SEPTEMBER 19, 2023

By employing robust data modeling techniques, businesses can unlock the true value of their data lake and transform it into a strategic asset. With many data modeling methodologies and processes available, choosing the right approach can be daunting. Want to learn more about data governance?

Data Lake 52

Data Lake Process Metadata Data Warehouse 52

Lyft Engineering

OCTOBER 3, 2023

In this post we will define data quality at a high-level and explore our motivation to achieve better data quality. We will then introduce our in-house product, Verity, and showcase how it serves as a central platform for ensuring data quality in our Hive Data Warehouse. What and Where is Data Quality?

Big Data 84

Big Data Metadata Data Warehouse Data 84

Workfall

JUNE 12, 2023

So, embrace the power of Change Data Capture, and embark on a captivating journey where the magic of real-time data awaits. In this blog, we will cover: What Is CDC and Its Benefits? CDC also plays a crucial role in data integration and ETL processes. Where Is CDC Used and Who Uses It?

Telecommunication 80

Telecommunication Metadata Healthcare Finance 80

Snowflake

NOVEMBER 30, 2023

At TCS , we help companies shift their enterprise data warehouse (EDW) platforms to the cloud as well as offering IT services. We’re extremely familiar with just how tricky a cloud migration can be, especially when it involves moving historical business data. Use separate data warehouses for cost-effective data loading.

Data Warehouse 97

Data Warehouse Banking Data Cloud 97

Data Engineering Podcast

MARCH 22, 2021

Datafold also helps automate regression testing of ETL code with its Data Diff feature that instantly shows how a change in ETL or BI code affects the produced data, both on a statistical level and down to individual rows and values. RudderStack’s smart customer data pipeline is warehouse-first.

Data Warehouse 100

Data Warehouse Metadata Data Lake Hadoop 100

Cloudera

AUGUST 31, 2021

Cloudera and Accenture demonstrate strength in their relationship with an accelerator called the Smart Data Transition Toolkit for migration of legacy data warehouses into Cloudera Data Platform. Accenture’s Smart Data Transition Toolkit . Are you looking for your data warehouse to support the hybrid multi-cloud?

Data Warehouse 80

Data Warehouse Database-centric Metadata Cloud 80

Cloudera

DECEMBER 10, 2020

Why worry about costs with cloud-native data warehousing? Have you been burned by the unexpected costs of a cloud data warehouse? If not, before adopting a cloud data warehouse, consider the true costs of a cloud-native data warehouse. These costs impede the adoption of cloud-native data warehouses.

Data Warehouse 98

Data Warehouse Cloud Storage Metadata Data 98

Cloudera

JULY 14, 2023

Depending on the size and usage patterns of the data, several different strategies could be pursued to achieve a successful migration. In this blog, I will describe a few strategies one could undertake for various use cases. This will be discussed in a later blog. Relatively fast as the underlying data files are kept in place.

Metadata 69

Metadata Data Warehouse Big Data Ecosystem Java 69

Cloudera

JANUARY 19, 2024

As described in our recent blog post , an SQL AI Assistant has been integrated into Hue with the capability to leverage the power of large language models (LLMs) for a number of SQL tasks. This is a real game-changer for data analysts on all levels and will make SQL development faster, easier, and less error-prone.

SQL 71

SQL Data Warehouse AWS Architecture 71

Cloudera

JUNE 18, 2022

These lakes power mission critical large scale data analytics, business intelligence (BI), and machine learning use cases, including enterprise data warehouses. In recent years, the term “data lakehouse” was coined to describe this architectural pattern of tabular analytics over data in the data lake.

Data Lake 89

Data Lake Data Warehouse BI SQL 89

Databand.ai

JULY 19, 2023

Learn more in our detailed guide to data lineage visualization (coming soon) Integration with Multiple Data Sources Data lineage tools are designed to integrate with a wide range of data sources, including databases, data warehouses, and cloud-based data platforms.

Pipeline-centric 52

Pipeline-centric Data Governance Metadata Government 52

dbt Developer Hub

OCTOBER 30, 2023

In this blog post, I’ll attempt to provide this guidance by answering these FAQs: What is dbt clone ? that leverages native zero-copy clone functionality on supported warehouses to copy entire schemas for free, almost instantly. On the surface, cloning and deferral seem similar – they’re both ways to save costs in the data warehouse.

BI 52

BI Datasets Metadata Data Warehouse 52

Data Engineering Weekly

MARCH 10, 2024

Pradheep Arjunan - Shared insights on AZ's journey from on-prem to the cloud data warehouses. Google: Croissant- a metadata format for ML-ready datasets Google Research introduced Croissant, a new metadata format designed to make datasets ML-ready by standardizing the format, facilitating easier use in machine learning projects.

Data Engineering 52

Data Engineering Data Engineer Engineering Datasets 52

Cloudera

MARCH 17, 2023

I took the free version of ChatGPT on a test drive (in March 2023) and asked some simple questions on data lakehouse and its components. Hopefully this blog will give ChatGPT an opportunity to learn and correct itself while counting towards my 2023 contribution to social good. I thought this was a fairly comprehensive list.

Education 60

Education Unstructured Data Data Lake Data Warehouse 60

Data Engineering Weekly

MAY 16, 2023

In the second part, we will focus on architectural patterns to implement data quality from a data contract perspective. Why is Data Quality Expensive? I won’t bore you with the importance of data quality in the blog. Event Routers can add additional metadata to the envelope of the event. How to Fix It?

Engineering 110

Engineering Kafka Data Pipeline Data Warehouse 110

Cloudera

SEPTEMBER 29, 2022

This team or domain expert will be responsible for the data produced by the team. The data itself is then treated as a product. The data product is not just the data itself, but a bunch of metadata that surrounds it — the simple stuff like schema is a given. What is a data mesh contract?

Architecture 73

Architecture Data Architecture Metadata Data Warehouse 73

Cloudera

FEBRUARY 14, 2024

Introduction Apache Iceberg has recently grown in popularity because it adds data warehouse-like capabilities to your data lake making it easier to analyze all your data — structured and unstructured. Expiring snapshots is a relatively cheap operation and uses metadata to determine newly unreachable files.

Bytes 56

Bytes Metadata Data Lake SQL 56

Maxime Beauchemin

JANUARY 20, 2017

The fact that ETL tools evolved to expose graphical interfaces seems like a detour in the history of data processing, and would certainly make for an interesting blog post of its own. Sure, there’s a need to abstract the complexity of data processing, computation and storage.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130

Cloudera

JANUARY 19, 2022

Cloudera Data Platform (CDP) scored among the top 10 vendors on all four Analytical Use Cases — Data Warehouse, Logical Data Warehouse, Data Lake and Operational Intelligence in the Critical Capabilities for Cloud Database Management Systems for Analytics Use Cases.

Database 86

Database Cloud Data Warehouse Data Lake 86

Cloudera

NOVEMBER 12, 2020

Your sunk costs are minimal and if a workload or project you are supporting becomes irrelevant, you can quickly spin down your cloud data warehouses and not be “stuck” with unused infrastructure. Cloud deployments for suitable workloads gives you the agility to keep pace with rapidly changing business and data needs.

Cloud 101

Cloud Data Warehouse Banking Data 101

phData: Data Engineering

OCTOBER 12, 2022

In this blog post, we’ll dive into six of the most powerful dbt Cloud features and many others that you probably don’t know about. Even if there is a failure, or if your data is stale, dbt Cloud provides a link to see the details of the tests and the sources that might be stale. Our team of data experts are happy to assist.

Cloud 52

Cloud Metadata SQL Data Warehouse 52

Data Engineering Weekly

OCTOBER 30, 2022

I found the blog helpful in understanding the generative model’s historical development and the path forward. link] Sponsored- [New eBook] The Ultimate Data Observability Platform Evaluation Guide Considering investing in a data quality solution? The author explains how to dump the history of blockchains into S3.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Ingestion 52

Data Engineering Weekly

FEBRUARY 18, 2024

[link] Joe Reis: Everything Ends - My Journey With the Modern Data Stack Joe writes another excellent retrospect for Modern Data Stack, walking down memory lane of the early and golden days of the Modern Data Stack. We are so over the Big Data Era to Modern Data Stack.

Data Engineering 62

Data Engineering Data Engineer Engineering Data 62

Cloudera

OCTOBER 11, 2022

But it isn’t just aggregating data for models. Data needs to be prepared and analyzed. Different data types need different types of analytics – real-time, streaming, operational, data warehouses. And that data is likely in clouds, in data centers and at the edge.

Data Science 89

Data Science Aggregated Data Data Consulting 89

DataKitchen

AUGUST 3, 2021

The past decades of enterprise data platform architectures can be summarized in 69 words. First-generation – expensive, proprietary enterprise data warehouse and business intelligence platforms maintained by a specialized team drowning in technical debt. The post What is a Data Mesh? first appeared on DataKitchen.

Pharmaceutical 98

Pharmaceutical Data Lake Data Architecture Scala 98

Cloudera

DECEMBER 21, 2023

In the “assumptions” field, we see how the SQL AI Assistant looked over our data model; compared to what we’re looking for, it was able to find the right tables, columns, and joins needed to provide a query that will give us the list we’re looking for. And as a bonus, we even get the query written for us, saving us even more time!

SQL 66

SQL Data Warehouse Business Analyst Data Schemas 66

Data Engineering Weekly

JULY 2, 2023

link] Apache Doris: Building a Data Warehouse for Traditional Industry The blog narates the incremental update and the overall update strategy of a typical legacy datawarehouses. Find the right, trusted data by simply asking a question. Similarly, the blog narrates how to use vector searching in Postgres using PGVector.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Advancing Analytics: Data Engineering

SEPTEMBER 7, 2021

"By its definition, a data lake accepts any data, without oversight or governance. Without descriptive metadata and a mechanism to maintain it, the data lake risks turning into a data swamp." - Gartner 2014 - [link] So here is where these next few blog articles come in. Here is where the Delta Lake comes in.

Data Lake 52

Data Lake Datasets Data Warehouse SQL 52

Cloudera

SEPTEMBER 17, 2020

To address this, they focused on creating an experimentation-oriented culture, enabled thanks to a cloud-native platform supporting the full data lifecycle. This platform, including an ad-hoc capable data warehouse service with built-in, easy-to-use visualization, made it easy for anyone to jump in and start experimenting.

Unstructured Data 80

Unstructured Data Pharmaceutical Data Warehouse MySQL 80

Hepta Analytics

FEBRUARY 14, 2022

DE Zoomcamp 2.2.1 – Introduction to Workflow Orchestration Following last weeks blog , we move to data ingestion. We already had a script that downloaded a csv file, processed the data and pushed the data to postgres database. This week, we got to think about our data ingestion design.

Data Ingestion 52

Data Ingestion Data Engineering Data Engineer Engineering 52

Monte Carlo

MARCH 3, 2021

Instead, we only extract query logs, metadata, and aggregated statistics about data usage to ensure that your most critical data assets are as trustworthy and reliable as possible. We designed our product without having to store or access individual records, PII, or any other sensitive information. What’s next?

Metadata 52

Metadata Data Lake Business Intelligence Certification 52

Workfall

AUGUST 14, 2023

Two powerful tools that have emerged to simplify the management of data pipelines are DBT (Data Build Tool) and Airflow. These tools, when combined, form a dynamic duo that streamlines and scales data engineering workflows. So, let’s dive into the world of DBT and Airflow and discover how they revolutionize data pipelines.

Data Pipeline 62

Data Pipeline Raw Data Data Database 62

Cloudera

NOVEMBER 23, 2020

Upgrading components of Cloudera Data Warehouse (CDW) is as easy as one decision and a few clicks. While the Database (DB) Catalog and Virtual Warehouses (VWs) need to be compatible, the CDW upgrade framework understands the interoperability constraints between them. . contains new API for faster metadata.

Metadata 65

Metadata Data Warehouse Database Accessible 65

AltexSoft

FEBRUARY 11, 2023

It serves as a foundation for the entire data management strategy and consists of multiple components including data pipelines; , on-premises and cloud storage facilities – data lakes , data warehouses , data hubs ;, data streaming and Big Data analytics solutions ( Hadoop , Spark , Kafka , etc.);

Data Architect 87

Data Architect Certification Generalist Big Data 87

Databand.ai

JUNE 28, 2023

This might include processes like data extraction from different sources, data cleansing, data transformation (like aggregation), and loading the data into a database or a data warehouse. Data storage and delivery : Observability continues into the storage and delivery phase.

Data Pipeline 98

Data Pipeline Data Engineering Data Engineer Engineering 98

Data Engineering Weekly

NOVEMBER 6, 2022

A data contract is a continuous and collaborative system because the business context and requirements won’t be static. I plan to write a series of blogs on Schemata and Data Contract in the coming weeks. Martin kindly stepped in for me to give the update for my promised blog posts.

Data Engineering 40

Data Engineering Data Engineer Engineering Machine Learning 40

Snowflake

APRIL 17, 2023

What started as a cloud data warehouse evolved into the Data Cloud, which includes the content—data, services, and applications—that customers and partners bring to the cloud. Snowflake’s Global Data Clean Rooms allow customers to “share without showing,” a concept I’ve discussed in this blog post.

Retail 69

Retail Metadata Cloud Data Warehouse 69