Monte Carlo

APRIL 16, 2022

The data warehouse is the foundation of the modern data stack, so it caught our attention when we saw Convoy head of data Chad Sanderson declare, “ the data warehouse is broken ” on LinkedIn. Treating data like an API. Immutable data warehouses have challenges too.

Data Warehouse 52

Data Warehouse Architecture Data Government 52

Data Engineering Podcast

AUGUST 13, 2022

Select Star’s data discovery platform solves that out of the box, with an automated catalog that includes lineage from where the data originated, all the way to which dashboards rely on it and who is viewing them every day. In fact, while only 3.5% That’s where our friends at Ascend.io In fact, while only 3.5%

Metadata 100

Metadata MongoDB Scala MySQL 100

Data Engineering Podcast

FEBRUARY 19, 2023

Summary Cloud data warehouses have unlocked a massive amount of innovation and investment in data applications, but they are still inherently limiting. Because of their complete ownership of your data they constrain the possibilities of what data you can store and how it can be used.

IT 147

IT Data Lake Metadata Data Warehouse 147

Cloudera

NOVEMBER 12, 2020

Cloud has given us hope, with public clouds at our disposal we now have virtually infinite resources, but they come at a different cost – using the cloud means we may be creating yet another series of silos, which also creates unmeasurable new risks in security and traceability of our data. A solution.

Cloud 101

Cloud Data Warehouse Banking Data 101

Christophe Blefari

JANUARY 20, 2024

Data engineering inherits from years of data practices in US big companies. Hadoop initially led the way with Big Data and distributed computing on-premise to finally land on Modern Data Stack — in the cloud — with a data warehouse at the center. This is often linked to real-time.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

phData: Data Engineering

OCTOBER 12, 2022

So you’ve heard all the talk around dbt , but now you’re working to determine if you should go with dbt Core or dbt Cloud and you’re wanting to know what advantages dbt Cloud has over the free dbt Core offering. However, dbt Cloud offers you much more than that.

Cloud 52

Cloud Metadata SQL Data Warehouse 52

dbt Developer Hub

JULY 2, 2023

The Data Vault data model is designed using a few well-applied principles, and in practice, pools source data so it is available for use by all downstream consumers. This promotes a scalable data warehouse through reusability and modularity. Should you consider Data Vault for your data warehouse?

Cloud 52

Cloud Data Warehouse Data BI 52

Cloudera

JANUARY 19, 2022

Gartner® recognized Cloudera in three recent reports – Magic Quadrant for Cloud Database Management Systems (DBMS), Critical Capabilities for Cloud Database Management Systems for Analytical Use Cases and Critical Capabilities for Cloud Database Management Systems for Operational Use Cases.

Database 86

Database Cloud Data Warehouse Data Lake 86

François Nguyen

MARCH 22, 2021

TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. How can we interoperate between the data domains ? How do we govern all these data products and domains ? It will be illustrated with our technical choices and the services we are using in the Google Cloud Platform.

Architecture 130

Architecture Technology Google Cloud Metadata 130

Data Engineering Podcast

DECEMBER 19, 2021

Summary Building a well managed data ecosystem for your organization requires a holistic view of all of the producers, consumers, and processors of information. The team at Metaphor are building a fully connected metadata layer to provide both technical and social intelligence about your data. No more scripts, just SQL.

Systems 100

Systems Building Metadata Data Warehouse 100

Snowflake

NOVEMBER 30, 2023

At TCS , we help companies shift their enterprise data warehouse (EDW) platforms to the cloud as well as offering IT services. We’re extremely familiar with just how tricky a cloud migration can be, especially when it involves moving historical business data.

Data Warehouse 96

Data Warehouse Banking Data Cloud 96

Monte Carlo

JANUARY 5, 2024

You know what they always say: data lakehouse architecture is like an onion. …ok, Data lakehouse architecture combines the benefits of data warehouses and data lakes, bringing together the structure and performance of a data warehouse with the flexibility of a data lake. Metadata layer 4.

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

Monte Carlo

JANUARY 5, 2024

You know what they always say: data lakehouse architecture is like an onion. …ok, Data lakehouse architecture combines the benefits of data warehouses and data lakes, bringing together the structure and performance of a data warehouse with the flexibility of a data lake. Metadata layer 4.

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

Data Engineering Podcast

JUNE 19, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Metadata 130

Metadata Unstructured Data MongoDB MySQL 130

Cloudera

AUGUST 31, 2021

Cloudera and Accenture demonstrate strength in their relationship with an accelerator called the Smart Data Transition Toolkit for migration of legacy data warehouses into Cloudera Data Platform. Accenture’s Smart Data Transition Toolkit . Are you looking for your data warehouse to support the hybrid multi-cloud?

Data Warehouse 80

Data Warehouse Database-centric Metadata Cloud 80

Cloudera

DECEMBER 10, 2020

Why worry about costs with cloud-native data warehousing? Have you been burned by the unexpected costs of a cloud data warehouse? If so, you know about the failed economics of some cloud-native solutions on the market today. These costs impede the adoption of cloud-native data warehouses.

Data Warehouse 98

Data Warehouse Cloud Storage Metadata Data 98

Data Engineering Weekly

DECEMBER 29, 2022

Data Catalog as a passive web portal to display metadata requires significant rethinking to adopt modern data workflow, not just adding “modern” in its prefix. I know that is an expensive statement to make😊 To be fair, I’m a big fan of data catalogs, or metadata management , to be precise.

Metadata 62

Metadata Data Warehouse ETL Tools Data Workflow 62

ThoughtSpot

MAY 31, 2023

Architecture Let's start with the big picture and tackle how we adjusted our cloud architecture with additional internal and external interfaces to integrate LLM. This multi-tenant service isolates the tenant metadata index, authorizing and filtering the search answer requests from every tenant.

Data Security 86

Data Security Metadata Transportation Data Warehouse 86

phData: Data Engineering

SEPTEMBER 19, 2023

Want to learn more about data governance? Check out our Data Governance on Snowflake blog! Metadata Management Data modeling methodologies help in managing metadata within the data lake. Metadata describes the characteristics, attributes, and context of the data.

Data Lake 52

Data Lake Process Metadata Data Warehouse 52

Data Engineering Podcast

JULY 3, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Architecture 100

Architecture Metadata MongoDB MySQL 100

Data Engineering Podcast

JULY 17, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Data Management 130

Data Management Management Metadata MongoDB 130

Monte Carlo

APRIL 24, 2023

Traditionally, after being stored in a data lake, raw data was then often moved to various destinations like a data warehouse for further processing, analysis, and consumption. Databricks Data Catalog and AWS Lake Formation are examples in this vein. See our post: Data Lakes vs. Data Warehouses.

Data Lake 52

Data Lake Google Cloud Data Warehouse AWS 52

Data Engineering Podcast

JUNE 12, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking all of that information into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Go to dataengineeringpodcast.com/atlan today to learn more about how you can take advantage of active metadata and escape the chaos.

Metadata 100

Metadata Unstructured Data Business Intelligence MongoDB 100

Data Engineering Podcast

JANUARY 7, 2022

He discusses the factors that influenced his decision to start with the data warehouse, the potential shortcomings of that approach, and where he plans to go from there. This is a great exploration of what it means to treat your data platform as a living system and apply state of the art engineering to it.

BI 100

BI Data Warehouse Metadata SQL 100

Christophe Blefari

JUNE 16, 2023

Why data consumers do not trust your reporting — It is a good illustration of the data journey manifesto. Stakeholders often notice data issues before the data team does. Data warehouses are mutable, this is one of the many root causes proposed by Lucas. Data Documentation 101: Why?

Programming Language 130

Programming Language SQL PostgreSQL Data 130

Data Engineering Podcast

FEBRUARY 20, 2022

RudderStack’s smart customer data pipeline is warehouse-first. It builds your customer data warehouse and your identity graph on your data warehouse, with support for Snowflake, Google BigQuery, Amazon Redshift, and more. RudderStack’s smart customer data pipeline is warehouse-first.

Systems 100

Systems Software Engineer Software Engineering Data Warehouse 100

Ascend.io

AUGUST 31, 2023

In the dynamic world of data, many professionals are still fixated on traditional patterns of data warehousing and ETL, even while their organizations are migrating to the cloud and adopting cloud-native data services. Source : A stream of sensor data represented as a directed acyclic graph.

Data Lake 52

Data Lake ETL Tools Data Warehouse Data Pipeline 52

Monte Carlo

MARCH 12, 2024

Google Cloud’s Dataplex is a data fabric tool that enables organizations to discover, manage, monitor, and govern their data across all of their data systems, including their data lakes, data warehouses, data lakehouses, and data marts. Dataplex works with your metadata.

Google Cloud 52

Google Cloud Metadata SQL Data Lake 52

Towards Data Science

DECEMBER 1, 2023

This tool automates ELT (Extract, Load, Transform) process, integrating your data from the source system of Google Calendar to our Snowflake data warehouse. Storage —  Snowflake Snowflake, a cloud-based data warehouse tailored for analytical needs, will serve as our data storage solution.

Data Engineering 74

Data Engineering Data Engineer Project Engineering 74

Data Engineering Weekly

JUNE 29, 2023

The quest to simplify data access is there forever, but with the advancement in LLM, I think it will become a reality. Databricks and Snowflake are better places to index the data and its metadata to enable natural language query capabilities. Snowflake is a DataLake Platform Snowflake is moving beyond a SQL data warehouse.

Data Engineering 76

Data Engineering Data Engineer Google Cloud Engineering 76

Monte Carlo

MARCH 4, 2024

To do this, data observability solutions like Monte Carlo will piece together information from query logs, data aggregates and system metadata. All of this is to say that how table formats are structured, and specifically the metadata, impacts how they can be monitored and observed.

Metadata 52

Metadata Data Lake Data Warehouse Data Engineering 52

AltexSoft

AUGUST 29, 2023

Instead of relying on traditional hierarchical structures and predefined schemas, as in the case of data warehouses, a data lake utilizes a flat architecture. This structure is made efficient by data engineering practices that include object storage. Data warehouse vs. data lake in a nutshell.

Data Lake 52

Data Lake Architecture IT Amazon Web Services 52

Data Engineering Podcast

JULY 5, 2021

Deploying, managing, and scaling that orchestration can consume a large fraction of a data team’s energy so it is important to pick something that provides the power and flexibility that you need. RudderStack’s smart customer data pipeline is warehouse-first. Today, there is a better way.

Systems 100

Systems Management Data Warehouse Programming Language 100

Data Engineering Podcast

JUNE 5, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking all of that information into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Go to dataengineeringpodcast.com/atlan today to learn more about how you can take advantage of active metadata and escape the chaos.

Data Security 100

Data Security Metadata MongoDB Scala 100

Cloudera

SEPTEMBER 29, 2022

Data teams have the impossible task of delivering everything (data and workloads) everywhere (on premise and in all clouds) all at once (with little to no latency). First, we describe how data mesh and data fabric could be related. Data mesh defined. The data itself is then treated as a product.

Architecture 73

Architecture Data Architecture Metadata Data Warehouse 73

Monte Carlo

JANUARY 29, 2024

It’s one thing to get your data into a modern data cloud. Monte Carlo is thrilled to be part of the Snowflake Horizon partner ecosystem as we leverage many of the pre-built features Snowflake provides in order to help organizations reduce their data downtime and improve data quality at scale.

Metadata 64

Metadata High Quality Data Data Pipeline Machine Learning 64

Christophe Blefari

FEBRUARY 3, 2023

Optimizing dbt development with Snowflake clones — dbt development in large data warehouse can become expensive if you ask every dbt developer to dbt run the whole SQL tree. How we deployed a simple wildlife monitoring system on Google Cloud — Artefact engineering a serverless platform on GCP to do wildlife monitoring.

BI 130

BI Google Cloud SQL Machine Learning 130