Christophe Blefari

JUNE 16, 2023

Why data consumers do not trust your reporting — It is a good illustration of the data journey manifesto. Stakeholders often notice data issues before the data team does. Data warehouses are mutable, this is one of the many root causes proposed by Lucas. Data Documentation 101: Why?

Programming Language 130

Programming Language SQL PostgreSQL Data 130

Data Engineering Podcast

DECEMBER 16, 2019

The solution to discoverability and tracking of data lineage is to incorporate a metadata repository into your data platform. The metadata repository serves as a data catalog and a means of reporting on the health and status of your datasets when it is properly integrated into the rest of your tools.

Metadata 100

Metadata PostgreSQL Datasets Data Warehouse 100

Cloudera

AUGUST 31, 2021

Cloudera and Accenture demonstrate strength in their relationship with an accelerator called the Smart Data Transition Toolkit for migration of legacy data warehouses into Cloudera Data Platform. Accenture’s Smart Data Transition Toolkit . Are you looking for your data warehouse to support the hybrid multi-cloud?

Data Warehouse 81

Data Warehouse Database-centric Metadata Cloud 81

Cloudera

JULY 14, 2023

While these instructions are carried out for Cloudera Data Platform (CDP), Cloudera Data Engineering, and Cloudera Data Warehouse, one can extrapolate them easily to other services and other use cases as well. Query engines (Impala, Hive, Spark) might mitigate some of these problems by using Iceberg’s metadata files.

Metadata 70

Metadata Data Warehouse Big Data Ecosystem Java 70

Cloudera

DECEMBER 10, 2020

Why worry about costs with cloud-native data warehousing? Have you been burned by the unexpected costs of a cloud data warehouse? If not, before adopting a cloud data warehouse, consider the true costs of a cloud-native data warehouse. These costs impede the adoption of cloud-native data warehouses.

Data Warehouse 98

Data Warehouse Cloud Storage Metadata Data 98

Monte Carlo

AUGUST 22, 2023

For example, let’s say a resource monitor has alerted the team that the Snowflake data warehouse, “TRANSFORMING_METRX,” has exceeded its threshold. You can view all the query runtimes for that warehouse during the time period when the alert was triggered. This could drive unwanted costs. Now it’s time to figure out why.

Metadata 52

Metadata Data Pipeline Data Warehouse Utilities 52

Cloudera

MARCH 17, 2023

When implementing a data lakehouse, the table format is a critical piece because it acts as an abstraction layer, making it easy to access all the structured, unstructured data in the lakehouse by any engine or tool, concurrently. Also, the data lake layer is not limited to cloud object stores.

Education 62

Education Unstructured Data Data Lake Data Warehouse 62

Databand.ai

JUNE 28, 2023

This might include processes like data extraction from different sources, data cleansing, data transformation (like aggregation), and loading the data into a database or a data warehouse. Data storage and delivery : Observability continues into the storage and delivery phase. Was this article helpful?

Data Pipeline 98

Data Pipeline Data Engineering Data Engineer Engineering 98

Monte Carlo

APRIL 24, 2023

Traditionally, after being stored in a data lake, raw data was then often moved to various destinations like a data warehouse for further processing, analysis, and consumption. Databricks Data Catalog and AWS Lake Formation are examples in this vein. See our post: Data Lakes vs. Data Warehouses.

Data Lake 52

Data Lake Google Cloud Data Warehouse AWS 52

Data Engineering Podcast

SEPTEMBER 23, 2018

This includes modeling the lifecycle of your information as a pipeline from the raw, messy, loosely structured records in your data lake, through a series of transformations and ultimately to your data warehouse. What is your opinion on the relative merits of a data warehouse vs a data lake and are they mutually exclusive?

Data Lake 100

Data Lake Data Warehouse Data Architecture Architecture 100

Cloudera

FEBRUARY 14, 2024

Introduction Apache Iceberg has recently grown in popularity because it adds data warehouse-like capabilities to your data lake making it easier to analyze all your data — structured and unstructured. Expiring snapshots is a relatively cheap operation and uses metadata to determine newly unreachable files.

Bytes 57

Bytes Metadata Data Lake SQL 57

Data Engineering Weekly

JULY 2, 2023

link] Apache Doris: Building a Data Warehouse for Traditional Industry The blog narates the incremental update and the overall update strategy of a typical legacy datawarehouses. Atlan AI leverages metadata that Atlan captures across the data stack to make AI part of your data stack.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

dbt Developer Hub

JANUARY 24, 2023

Ensures developers can work quickly by shortening the CI feedback loop Reduce costs in your data warehouse by running only what's been modified I think we can all agree that increasing developer productivity while simultaneously reducing costs is an outcome that every company strives for. A query method on the metadata property.

Metadata 52

Metadata Python Cloud Data Warehouse 52

Snowflake

APRIL 17, 2023

What started as a cloud data warehouse evolved into the Data Cloud, which includes the content—data, services, and applications—that customers and partners bring to the cloud. Snowflake makes it easy to build and deliver data and applications to customers across clouds. Oh, and a net promoter score of 72.

Retail 69

Retail Metadata Cloud Data Warehouse 69

Netflix Tech

OCTOBER 27, 2021

The GUI would be targeted at professional data scientists. Instead of a fancy GUI for demos and presentations, we wanted a serious productivity tool with carefully thought-out user workflows that would fit seamlessly into our toolchain of data science. The GUI should integrate well with other GUIs.

Python 89

Python Machine Learning Data Science Metadata 89

Cloudera

FEBRUARY 8, 2024

Cloudera Data Warehouse (CDW) running Hive has previously supported creating materialized views against Hive ACID source tables. release and the matching CDW Private Cloud Data Services release, Hive also supports creating, using, and rebuilding materialized views for Iceberg table format.

Metadata 89

Metadata Data Warehouse BI AWS 89

Data Engineering Podcast

JUNE 26, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Datasets 130

Datasets Unstructured Data Metadata MongoDB 130

Cloudera

JULY 13, 2023

In this blog, we will discuss performance improvement that Cloudera has contributed to the Apache Iceberg project in regards to Iceberg metadata reads, and we’ll showcase the performance benefit using Apache Impala as the query engine. Impala can access Hive table metadata fast because HMS is backed by RDBMS, such as mysql or postgresql.

Metadata 70

Metadata Java PostgreSQL Data Warehouse 70

Data Engineering Podcast

NOVEMBER 20, 2022

Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold. RudderStack helps you build a customer data platform on your warehouse or data lake.

Data Lake 130

Data Lake Data Ingestion MongoDB Scala 130

Data Engineering Podcast

MAY 18, 2021

RudderStack’s smart customer data pipeline is warehouse-first. It builds your customer data warehouse and your identity graph on your data warehouse, with support for Snowflake, Google BigQuery, Amazon Redshift, and more. RudderStack’s smart customer data pipeline is warehouse-first.

Metadata 100

Metadata Kafka Data Warehouse Hadoop 100

RandomTrees

APRIL 18, 2024

By leveraging Python’s capabilities, we can create scripts that analyze Informatica workflows, extract relevant metadata, and generate equivalent Talend jobs. Analysing Informatica Workflows : The first step in the conversion process is to analyze Informatica workflows to extract essential metadata and transformation logic.

Coding 52

Coding Retail Metadata Python 52

Snowflake

FEBRUARY 29, 2024

This grid, along with the metadata that represents the details of how the grid is constructed and used, is referred to as a spatial index. Caption: A demo Streamlit app visualizes H3 hexagons at different resolutions for the same data. There are 16 layers, or levels, of resolution in H3.

Insurance 85

Insurance SQL BI Consulting 85

AltexSoft

NOVEMBER 7, 2022

Metadata database. A metadata database stores information about user permissions, past and current DAG and task runs, DAG configurations, and more. By default, Airflow handles metadata with SQLite which is meant for development only. The Good and the Bad of Power BI Data Visualization.

PostgreSQL 52

PostgreSQL Metadata Python MySQL 52

Datakin

OCTOBER 14, 2021

dbt is an amazing way to transform data within a data warehouse. Run models & capture lineage metadata When working with Datakin (or any other OpenLineage backend) it’s important to generate the dbt docs first. Our schema has changed, and we want Datakin to have the latest metadata about tables and columns. %

Google Cloud 52

Google Cloud Datasets Bytes Metadata 52

Monte Carlo

AUGUST 7, 2023

Prevention: Finally, data observability also provides mechanisms to prevent data issues from happening in the first place, like placing circuit breaking in pipelines and creating visibility around the impact code changes would make on data, among other proactive measures of preventing bad data from entering your pipelines in the first place.

Coding 52

Coding Data Software Engineer Software Engineering 52

AltexSoft

OCTOBER 8, 2021

Data integration defines the process of collecting data from a number of disparate source systems and presenting it in a unified form within a centralized location like a data warehouse. So, why is data integration such a big deal? Oracle Data Integrator, IBM InfoSphere, Snaplogic, Xplenty, and.

Data Integration 52

Data Integration Hadoop Data Warehouse Data Lake 52

Data Engineering Podcast

JANUARY 28, 2018

Contact Info Dat datproject.org Email @dat_project on Twitter Dat Chat Danielle Email @daniellecrobins Joe Email @joeahand on Twitter Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? So this is really useful for the big data sets, you know, I think the whole thing.

Data 100

Data Project Electronics Data Management 100

Ascend.io

APRIL 7, 2023

Source: Data Mesh Principles and Logical Architecture by Zhamak Dehghani What is a Data Fabric? Data fabric is a centralized platform architecture originating from a curated metadata layer that sits on top of an organization’s data infrastructure. Increasing speed.

Metadata 52

Metadata Datasets Data Governance Government 52

Data Engineering Weekly

JULY 9, 2023

The platform approach to enable the citizen machine learning engineers is a great perspective while building both the Data & ML platform. Architectural patterns like Lambda Architecture and Kappa Architecture emerged to bridge the gap between real-time and batch data processing. Write SQL queries without learning SQL?

Data Engineering 52

Data Engineering Data Engineer Engineering Lambda Architecture 52

Monte Carlo

APRIL 20, 2023

I can surface ownership metadata and alert the relevant owners to make sure the appropriate changes are made so these breakages never happen. And over time, we’ve invested effort into cleaning up our lineages, simplifying our logic,” said SeatGeek Director of Data Engineering, Brian London. Data lineage can help!

Data Warehouse 52

Data Warehouse BI Government Data 52

AltexSoft

MARCH 30, 2023

What is Databricks Databricks is an analytics platform with a unified set of tools for data engineering, data management , data science, and machine learning. It combines the best elements of a data warehouse, a centralized repository for structured data, and a data lake used to host large amounts of raw data.

Scala 64

Scala Data Lake BI Google Cloud 64