Monte Carlo

APRIL 4, 2023

Data mesh vs data warehouse is an interesting framing because it is not necessarily a binary choice depending on what exactly you mean by data warehouse (more on that later). Despite their differences, however, both approaches require high-quality, reliable data in order to function. What is a Data Mesh?

Data Warehouse 52

Data Warehouse Data Governance Data Architecture 52

ProjectPro

JANUARY 27, 2023

Whether you are a data engineer, BI engineer, data analyst, or an ETL developer, understanding various ETL use cases and applications can help you make the most of your data by unleashing the power and capabilities of ETL in your organization. You have probably heard the saying, "data is the new oil".

BI 52

BI ETL Tools Retail Healthcare 52

Knowledge Hut

APRIL 19, 2023

It is the process of extracting data from various sources, transforming it into a format suitable for analysis, and loading it into a target database or data warehouse. ETL is used to integrate data from different sources and formats into a single target for analysis. What is an ETL Pipeline?

Data Warehouse 52

Data Warehouse Business Intelligence ETL Tools Data Pipeline 52

Monte Carlo

FEBRUARY 6, 2023

So, you’re planning a cloud data warehouse migration. But be warned, a warehouse migration isn’t for the faint of heart. As you probably already know if you’re reading this, a data warehouse migration is the process of moving data from one warehouse to another. A worthy quest to be sure.

Data Warehouse 52

Data Warehouse AWS Data Validation Data 52

Monte Carlo

JANUARY 25, 2023

In this article, Chad Sanderson , Head of Product, Data Platform , at Convoy and creator of Data Quality Camp , introduces a new application of data contracts: in your data warehouse. In the last couple of posts , I’ve focused on implementing data contracts in production services.

Data Warehouse 52

Data Warehouse Data High Quality Data Metadata 52

Data Engineering Podcast

AUGUST 31, 2020

Summary Data warehouse technology has been around for decades and has gone through several generational shifts in that time. The current trends in data warehousing are oriented around cloud native architectures that take advantage of dynamic scaling and the separation of compute and storage.

Data Warehouse 100

Data Warehouse Cloud Building Data Lake 100

Cloudera

JULY 16, 2021

Did you know Cloudera customers, such as SMG and Geisinger , offloaded their legacy DW environment to Cloudera Data Warehouse (CDW) to take advantage of CDW’s modern architecture and best-in-class performance? The Data Warehouse on Cloudera Data Platform provides easy to use self-service and advanced analytics use cases at scale.

Data Warehouse 115

Data Warehouse SQL PostgreSQL MySQL 115

Monte Carlo

OCTOBER 17, 2022

Platform and data warehouse migrations aren’t something you do everyday or even every few years, but they’re becoming much more frequent as organizations seek to modernize their data infrastructure with the new capabilities being offered by Snowflake, Databricks, Google, AWS, and others. Editor’s note: We agree.

Data Warehouse 59

Data Warehouse AWS MySQL Data 59

Monte Carlo

AUGUST 25, 2023

Different vendors offering data warehouses, data lakes, and now data lakehouses all offer their own distinct advantages and disadvantages for data teams to consider. So let’s get to the bottom of the big question: what kind of data storage layer will provide the strongest foundation for your data platform?

Data Lake 52

Data Lake Data Warehouse Unstructured Data Raw Data 52

Data Engineering Podcast

DECEMBER 8, 2019

Summary Data warehouses have gone through many transformations, from standard relational databases on powerful hardware, to column oriented storage engines, to the current generation of cloud-native analytical engines. If you are evaluating your options for building or migrating a data platform, then this is definitely worth a listen.

Data Warehouse 100

Data Warehouse Cloud AWS Relational Database 100

Monte Carlo

MAY 17, 2023

Because the technology is so extensible, there have been a wide array of suggestions–some more grounded than others–for how it can be used. To help clarify, we reviewed hundreds of data observability deployments to identify 61 real data observability use cases and benefits. Reduce The Amount Of Data Incidents 2.

Data 52

Data Data Pipeline Data Engineering Data Engineer 52

Cloudera

JANUARY 15, 2021

Cloud data warehouses allow users to run analytic workloads with greater agility, better isolation and scale, and lower administrative overhead than ever before. The results demonstrate superior price performance of Cloudera Data Warehouse on the full set of 99 queries from the TPC-DS benchmark. Introduction.

Data Warehouse 91

Data Warehouse Cloud Consulting SQL 91

Rockset

MAY 8, 2023

Many organizations that we’ve spoken to are in the exploration phase of using vector search for AI-powered personalization, recommendations, semantic search and anomaly detection. Different user interactions and data points that are used to form vector embeddings at Pinterest.

Metadata 52

Metadata Algorithm Datasets Google Cloud 52

Data Engineering Podcast

OCTOBER 14, 2019

Summary Managing a data warehouse can be challenging, especially when trying to maintain a common set of patterns. He started Datacoral with the goal to make SQL the universal data programming language. Are you working on data, analytics, or AI using platforms such as Presto, Spark, or Tensorflow?

Data Warehouse 100

Data Warehouse PostgreSQL AWS Programming Language 100

Data Engineering Podcast

MAY 1, 2022

Datafold shows how a change in SQL code affects your data, both on a statistical level and down to individual rows and values before it gets merged to production. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows.

Data Warehouse 100

Data Warehouse Data Integration Cloud Google Cloud 100

Monte Carlo

MAY 17, 2023

Because the technology is so extensible, there have been a wide array of suggestions–some more grounded than others–for how it can be used. To help clarify, we reviewed hundreds of data observability deployments to identify 61 real data observability use cases and benefits. Reduce the amount of data incidents 2.

Data Pipeline 52

Data Pipeline Data Data Engineering Data Engineer 52

ProjectPro

DECEMBER 7, 2021

This blog will give you an in-depth knowledge of what is a data pipeline and also explore other aspects such as data pipeline architecture, data pipeline tools, use cases, and so much more. Traditional organizations fail to analyze all the generated data due to a lack of automation.

Data Pipeline 52

Data Pipeline Architecture Kafka AWS 52

Data Engineering Podcast

JULY 8, 2019

Summary The market for data warehouse platforms is large and varied, with options for every use case. Use the code BNLLC to get an additional 10% off any pass when you register. What are the primary use cases that Clickhouse is targeting? How is data replication and consistency managed?

Data Warehouse 100

Data Warehouse MySQL Data Lake Hadoop 100

Monte Carlo

APRIL 16, 2022

The data warehouse is the foundation of the modern data stack, so it caught our attention when we saw Convoy head of data Chad Sanderson declare, “ the data warehouse is broken ” on LinkedIn. Of course, Chad isn’t referring to the technology, but how it’s being used. Treating data like an API.

Data Warehouse 52

Data Warehouse Architecture Data Government 52

Hepta Analytics

FEBRUARY 24, 2022

We will take the already ingested data and create an external table from it and optimize the performance of queries through partitioning and clustering. Then automate the whole process using airflow. There are two systems types when dealing with data: Online Transaction Processing (OLTP) and Online Analytical Processing (OLAP).

Data Warehouse 52

Data Warehouse Data Engineering Data Engineer Engineering 52

Cloudera

FEBRUARY 9, 2021

Today’s customers have a growing need for a faster end to end data ingestion to meet the expected speed of insights and overall business demand. This ‘need for speed’ drives a rethink on building a more modern data warehouse solution, one that balances speed with platform cost management, performance, and reliability.

Data Warehouse 67

Data Warehouse Cloud Kafka Cloud Storage 67

dbt Developer Hub

NOVEMBER 22, 2022

Once your data warehouse is built out, the vast majority of your data will have come from other SaaS tools, internal databases, or customer data platforms (CDPs). Spreadsheets are the Swiss army knife of data processing. With a visual interface to modify data types, it’s easy for anyone to use.

Data Warehouse 52

Data Warehouse ETL Tools Google Cloud Cloud Storage 52

dbt Developer Hub

NOVEMBER 14, 2021

I’ve used the dateadd SQL function thousands of times. This article will go over how the DATEADD function works, the nuances of using it across the major cloud warehouses, and how to standardize the syntax variances using dbt macro. What is the DATEADD SQL Function? ​ dateadd ( month , 1 , '2021-08-12' }} New to dbt?

Data Warehouse 52

Data Warehouse SQL Software Engineer Software Engineering 52

ProjectPro

AUGUST 11, 2021

“Data Lake vs Data Warehouse = Load First, Think Later vs Think First, Load Later” The terms data lake and data warehouse are frequently stumbled upon when it comes to storing large volumes of data. Data Warehouse Architecture What is a Data lake?

Data Lake 52

Data Lake Data Warehouse Cloud Hadoop 52

Cloudera

SEPTEMBER 24, 2020

Some of the most powerful results come from combining complementary superpowers, and the “dynamic duo” of Apache Hive LLAP and Apache Impala, both included in Cloudera Data Warehouse , is further evidence of this. Both Impala and Hive can operate at an unprecedented and massive scale, with many petabytes of data. So, why choose?

Data Warehouse 69

Data Warehouse SQL Engineering Metadata 69

Monte Carlo

JANUARY 25, 2023

When it comes to the question of building or buying your data stack, there’s never a one-size-fits-all solution for every data team—or every component of your data stack. Data storage and compute are very much the foundation of your data platform. So, let’s take a look at each in a bit more detail.

Data Warehouse 40

Data Warehouse Building Data Lake Data Storage 40

Data Engineering Weekly

JULY 5, 2023

It ensures sustainable and scalable data operations, fostering a balanced business growth path in the data-driven era. Coinbase writes one case about cost management for Databricks and how they use the open-source overwatch tool to manage Databrick’s cost. link] Matt Palmer: What's the hype behind DuckDB?

Management 52

Management Data Warehouse Data Engineering Data Engineer 52

Seattle Data Guy

NOVEMBER 28, 2023

Maybe we wouldn’t even need to build a data warehouse. But as anyone who has worked with data from more than one source knows, that’s rarely the case. appeared first on Seattle Data Guy.

IT 130

IT Data Warehouse Building Data 130

Cloudera

SEPTEMBER 10, 2018

Among the many reasons that a majority of large enterprises have adopted Cloudera Data Warehouse as their modern analytic platform of choice is the incredible ecosystem of partners that have emerged over recent years. Informatica’s Big Data Manager and Qlik’s acquisition of Podium Data are just 2 examples.

Data Warehouse 43

Data Warehouse BI Unstructured Data Data 43

Hevo

MARCH 30, 2023

The field of data analytics has also been touched by the winds of change. The need for up-to-date analytics has increased due to the increased urgency for business-leading data. We all are in the post-pandemic era now. Everything in the business sector has changed, including how they operate and employ tools and methods.

Data Analytics 52

Data Analytics Data Data Warehouse 52

Data Engineering Podcast

APRIL 9, 2023

When it was difficult to wire together the event collection, data modeling, reporting, and activation it made sense to buy monolithic products that handled every stage of the customer data lifecycle. Now that the data warehouse has taken center stage a new approach of composable customer data platforms is emerging.

Data Lake 147

Data Lake Data Warehouse Machine Learning Data 147

Data Engineering Podcast

APRIL 23, 2023

Applications powered by real-time data were the exclusive domain of large and/or sophisticated tech companies for several years due to the inherent complexities involved. What are some of the remaining blockers for teams who want to start using real-time data? RudderStack also supports real-time use cases.

Data Lake 277

Data Lake Kafka Machine Learning Data Warehouse 277

Data Engineering Podcast

JUNE 4, 2023

In this episode Tevje Olin explains how the platform is implemented, the features that it provides to reduce the amount of effort required to keep your pipelines running, and how you can start using it in your own team. How does the unified experience of Agile Data Engine change the way that teams think about the lifecycle of their data?

Data Engineering 130

Data Engineering Data Engineer Engineering Data Lake 130

Data Engineering Podcast

JUNE 25, 2023

In this episode Toby Mao explains how it works, the importance of automatic column-level lineage tracking, and how you can start using it today. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake.

Data Engineering 225

Data Engineering Data Engineer Python Engineering 225

ProjectPro

MARCH 15, 2016

It is important to study and understand several Hadoop use cases for these simple reasons – Hadoop is still a complex technology with several limitations and complications. All Data is not Big Data and might not require a Hadoop solution.

Hadoop 40

Hadoop Retail Healthcare Banking 40

Monte Carlo

OCTOBER 17, 2022

As companies ingest more data and expand access to it , your beautifully curated warehouse (or lakehouse, we don’t need to discriminate) can slowly turn into a data swamp as a result of this unfortunate reality of human behavior. To use another adage… having too many cooks in the kitchen spoils the broth.

Data Warehouse 52

Data Warehouse Government Data Governance Data 52

Snowflake

NOVEMBER 30, 2023

At TCS , we help companies shift their enterprise data warehouse (EDW) platforms to the cloud as well as offering IT services. We’re extremely familiar with just how tricky a cloud migration can be, especially when it involves moving historical business data. Use compressed files when possible.

Data Warehouse 93

Data Warehouse Banking Data Cloud 93