Data Engineering Weekly

DECEMBER 29, 2022

Data Catalog as a passive web portal to display metadata requires significant rethinking to adopt modern data workflow, not just adding “modern” in its prefix. I know that is an expensive statement to make😊 To be fair, I’m a big fan of data catalogs, or metadata management , to be precise.

Metadata 62

Metadata Data Warehouse ETL Tools Data Workflow 62

François Nguyen

MARCH 22, 2021

TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. With this 3rd platform generation, you have more real time data analytics and a cost reduction because it is easier to manage this infrastructure in the cloud thanks to managed services. What you have to code is this workflow !

Architecture 130

Architecture Technology Google Cloud Metadata 130

Data Engineering Podcast

DECEMBER 4, 2022

Summary Business intelligence is the foremost application of data in organizations of all sizes. The typical conception of how it is accessed is through a web or desktop application running on a powerful laptop. Zing Data is building a mobile native platform for business intelligence.

Business Intelligence 100

Business Intelligence Metadata BI MongoDB 100

Knowledge Hut

MARCH 7, 2024

This implies two things, one, the data coming from one source is out of date when compared to another source. Two, it creates a commonality of data definitions, concepts, metadata and the like. However, regulating access is one of the primary challenges for companies who frequently work with large sets of data.

Big Data 98

Big Data Bytes Data Governance Raw Data 98

ThoughtSpot

MAY 31, 2023

This includes services that: Manage and monitor the tenant-specific resources—this does not include access to tenant data Maintains indexed data to serve as your application home page. This multi-tenant service isolates the tenant metadata index, authorizing and filtering the search answer requests from every tenant.

Data Security 86

Data Security Metadata Transportation Data Warehouse 86

phData: Data Engineering

SEPTEMBER 19, 2023

Data Governance and Security By defining data models, organizations can establish policies, access controls, and security measures to protect sensitive data. Data models can also facilitate compliance with regulations and ensure proper data handling and protection. Want to learn more about data governance?

Data Lake 52

Data Lake Process Metadata Data Warehouse 52

Cloudera

DECEMBER 10, 2020

Why worry about costs with cloud-native data warehousing? Have you been burned by the unexpected costs of a cloud data warehouse? If not, before adopting a cloud data warehouse, consider the true costs of a cloud-native data warehouse. These costs impede the adoption of cloud-native data warehouses.

Data Warehouse 98

Data Warehouse Cloud Storage Metadata Data 98

Data Engineering Podcast

JULY 31, 2022

Summary Data lineage is the roadmap for your data platform, providing visibility into all of the dependencies for any report, machine learning model, or data warehouse table that you are working with. Atlan is the metadata hub for your data ecosystem. Can you describe what Manta is and the story behind it?

IT 100

IT Metadata MongoDB MySQL 100

Towards Data Science

FEBRUARY 6, 2024

As you do not want to start your development with uncertainty, you decide to go for the operational raw data directly. Accessing Operational Data I used to connect to views in transactional databases or APIs offered by operational systems to request the raw data. Metadata update Data products need to be understandable.

Systems 83

Systems Raw Data Metadata Data Cleanse 83

Data Engineering Podcast

MARCH 27, 2022

You have full control over your data and their plugin system lets you integrate with all of your other data tools, including data warehouses and SaaS platforms. Acryl]([link] The modern data stack needs a reimagined metadata management platform.

Data Governance 130

Data Governance Government Cloud Building 130

Data Engineering Weekly

MARCH 24, 2024

Dive into Spyne's experience with: - Their search for query acceleration with pre-aggregations and caching - Developing new functionality with Open AI - Optimizing query cost with their data warehouse [link] Suresh Hasuni: Cost Optimization Strategies for Scalable Data Lakehouse Cost is the major concern as the adoption of data lakes increases.

Data Engineering 62

Data Engineering Data Engineer Engineering Metadata 62

Christophe Blefari

FEBRUARY 3, 2023

The article shows our Netflix art creators are using past data to create new artworks. ebay, Variable Hub a data access layer for risk decisioning — Looks like a feature store but for risk topics. The idea is to create a unified layer that stores all the data needed to take decisions. It makes sense.

BI 130

BI Google Cloud SQL Machine Learning 130

Monte Carlo

APRIL 24, 2023

Traditionally, after being stored in a data lake, raw data was then often moved to various destinations like a data warehouse for further processing, analysis, and consumption. Databricks Data Catalog and AWS Lake Formation are examples in this vein. See our post: Data Lakes vs. Data Warehouses.

Data Lake 52

Data Lake Google Cloud Data Warehouse AWS 52

Data Engineering Podcast

FEBRUARY 20, 2022

RudderStack’s smart customer data pipeline is warehouse-first. It builds your customer data warehouse and your identity graph on your data warehouse, with support for Snowflake, Google BigQuery, Amazon Redshift, and more. What are some of the ways that you make your data accessible to AI/ML engineers?

Systems 100

Systems Software Engineer Software Engineering Data Warehouse 100

AltexSoft

AUGUST 29, 2023

Instead of relying on traditional hierarchical structures and predefined schemas, as in the case of data warehouses, a data lake utilizes a flat architecture. This structure is made efficient by data engineering practices that include object storage. Data warehouse vs. data lake in a nutshell.

Data Lake 52

Data Lake Architecture IT Amazon Web Services 52

Cloudera

AUGUST 31, 2021

Cloudera and Accenture demonstrate strength in their relationship with an accelerator called the Smart Data Transition Toolkit for migration of legacy data warehouses into Cloudera Data Platform. Accenture’s Smart Data Transition Toolkit . Are you looking for your data warehouse to support the hybrid multi-cloud?

Data Warehouse 80

Data Warehouse Database-centric Metadata Cloud 80

Monte Carlo

JANUARY 29, 2024

Snowflake recently announced the launch of Snowflake Horizon —a unified collection of compliance, security, privacy, interoperability, and access capabilities—that make it easy for organizations to govern and take action on their data and apps across clouds, teams, partners, and customers. Volume: Are there too many or too few rows?

Metadata 64

Metadata High Quality Data Data Pipeline Machine Learning 64

Towards Data Science

DECEMBER 1, 2023

This tool automates ELT (Extract, Load, Transform) process, integrating your data from the source system of Google Calendar to our Snowflake data warehouse. Storage —  Snowflake Snowflake, a cloud-based data warehouse tailored for analytical needs, will serve as our data storage solution.

Data Engineering 73

Data Engineering Data Engineer Project Engineering 73

Monte Carlo

MAY 18, 2023

Depending on your requirements (more on that in the next section), it is prudent to evaluate open source data lineage tools alongside the several other types of platforms that provide data lineage capabilities. You may even already have access to one of them. Why do you need data lineage? “I

Metadata 64

Metadata Data Lake Government Data Governance 64

Monte Carlo

MARCH 4, 2024

To do this, data observability solutions like Monte Carlo will piece together information from query logs, data aggregates and system metadata. All of this is to say that how table formats are structured, and specifically the metadata, impacts how they can be monitored and observed.

Metadata 52

Metadata Data Lake Data Warehouse Data Engineering 52

Cloudera

JANUARY 19, 2024

Below are the models tested by Cloudera: Data shared with the LLM models It is important to understand that by using the SQL AI Assistant you are sending your own prompts and also significant additional information as input to the LLM. Azure subscription First, get Azure access. Log in to the Cloudera Data Warehouse service as DWAdmin.

SQL 71

SQL Data Warehouse AWS Architecture 71

Data Engineering Weekly

JUNE 29, 2023

The quest to simplify data access is there forever, but with the advancement in LLM, I think it will become a reality. Databricks and Snowflake are better places to index the data and its metadata to enable natural language query capabilities. Google Cloud is always a pioneer in the data intelligence layer.

Data Engineering 76

Data Engineering Data Engineer Google Cloud Engineering 76

Monte Carlo

FEBRUARY 12, 2024

To differentiate and expand the usefulness of these models, organizations must augment them with first-party data – typically via a process called RAG (retrieval augmented generation). Today, this first-party data mostly lives in two types of data repositories.

Database 72

Database Unstructured Data Data Pipeline Metadata 72

Snowflake

APRIL 5, 2023

Two different data modeling approaches—dimensional data modeling and Data Vault—each have their own pros and cons. Modernizing a data warehouse with Snowflake Data Cloud is a smart investment that can provide significant benefits to businesses of all sizes, today more than ever as data models become ever more complex.

Data Warehouse 52

Data Warehouse Healthcare Unstructured Data Metadata 52

Data Engineering Podcast

MARCH 22, 2021

Datafold also helps automate regression testing of ETL code with its Data Diff feature that instantly shows how a change in ETL or BI code affects the produced data, both on a statistical level and down to individual rows and values. RudderStack’s smart customer data pipeline is warehouse-first.

Data Warehouse 100

Data Warehouse Metadata Data Lake Hadoop 100

ProjectPro

FEBRUARY 16, 2023

With so much riding on the efficiency of ETL processes for data engineering teams, it is essential to take a deep dive into the complex world of ETL on AWS to take your data management to the next level. ETL has typically been carried out utilizing data warehouses and on-premise ETL tools.

AWS 52

AWS Data Management ETL Tools Management 52

Ascend.io

AUGUST 31, 2023

Secondly , the rise of data lakes that catalyzed the transition from ELT to ELT and paved the way for niche paradigms such as Reverse ETL and Zero-ETL. Still, these methods have been overshadowed by EtLT — the predominant approach reshaping today’s data landscape.

Data Lake 52

Data Lake ETL Tools Data Warehouse Data Pipeline 52

Netflix Tech

MARCH 7, 2024

Once we have discovered the Parquet files to be processed, MetaflowDataFrame takes over: it downloads data using Metaflow’s high-throughput S3 client directly to the process’ memory, which often outperforms reading of local files. Thanks to Arrow, data can be accessed through these libraries in a zero-copy fashion.

Systems 90

Systems Media Machine Learning Data Warehouse 90

Ascend.io

FEBRUARY 28, 2024

Engineers ensure the availability of clean, structured data, a necessity for AI systems to learn from patterns, make accurate predictions, and automate decision-making processes. Through the design and maintenance of efficient data pipelines , data engineers facilitate the seamless flow and accessibility of data for AI processing.

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

Data Engineering Weekly

MARCH 10, 2024

Pradheep Arjunan - Shared insights on AZ's journey from on-prem to the cloud data warehouses. Google: Croissant- a metadata format for ML-ready datasets Google Research introduced Croissant, a new metadata format designed to make datasets ML-ready by standardizing the format, facilitating easier use in machine learning projects.

Data Engineering 52

Data Engineering Data Engineer Engineering Datasets 52

Cloudera

NOVEMBER 12, 2020

Your sunk costs are minimal and if a workload or project you are supporting becomes irrelevant, you can quickly spin down your cloud data warehouses and not be “stuck” with unused infrastructure. Cloud deployments for suitable workloads gives you the agility to keep pace with rapidly changing business and data needs.

Cloud 102

Cloud Data Warehouse Banking Data 102

Cloudera

MARCH 17, 2023

The one key component that is missing is a common, shared table format, that can be used by all analytic services accessing the lakehouse data. The table format provides the necessary structure for the unstructured data that is missing in a data lake, using a schema or metadata definition, to bring it closer to a data warehouse.

Education 61

Education Unstructured Data Data Lake Data Warehouse 61

Data Engineering Podcast

MAY 3, 2021

They explain how to think about your data systems in a holistic and maintainable fashion, the security challenges that threaten to derail your efforts, and the power of using metadata as the foundation of everything that you do. RudderStack’s smart customer data pipeline is warehouse-first.

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100

Cloudera

JANUARY 19, 2022

Cloudera Data Platform (CDP) scored among the top 10 vendors on all four Analytical Use Cases — Data Warehouse, Logical Data Warehouse, Data Lake and Operational Intelligence in the Critical Capabilities for Cloud Database Management Systems for Analytics Use Cases.

Database 87

Database Cloud Data Warehouse Data Lake 87

Cloudera

JULY 14, 2023

While these instructions are carried out for Cloudera Data Platform (CDP), Cloudera Data Engineering, and Cloudera Data Warehouse, one can extrapolate them easily to other services and other use cases as well. Query engines (Impala, Hive, Spark) might mitigate some of these problems by using Iceberg’s metadata files.

Metadata 69

Metadata Data Warehouse Big Data Ecosystem Java 69

Data Engineering Weekly

JANUARY 6, 2023

Author: Chris Ricommni Recap Github: [link] [Don't forget to star it] I’m excited to release Recap , a dead simple data catalog for engineers written in Python. Recap makes it easy for engineers to build infrastructure and tools that need metadata. Unlike traditional data catalogs, Recap is designed to power software.

Metadata 40

Metadata ETL Tools MySQL Data Lake 40

Cloudera

SEPTEMBER 29, 2022

This team or domain expert will be responsible for the data produced by the team. The data itself is then treated as a product. The data product is not just the data itself, but a bunch of metadata that surrounds it — the simple stuff like schema is a given. What is a data mesh contract?

Architecture 73

Architecture Data Architecture Metadata Data Warehouse 73