Monte Carlo

FEBRUARY 6, 2023

So, you’re planning a cloud data warehouse migration. But be warned, a warehouse migration isn’t for the faint of heart. As you probably already know if you’re reading this, a data warehouse migration is the process of moving data from one warehouse to another. A worthy quest to be sure.

Data Warehouse 52

Data Warehouse AWS Data Validation Data 52

Monte Carlo

JUNE 14, 2023

In this post, we will help you quickly level up your overall knowledge of data pipeline architecture by reviewing: Table of Contents What is data pipeline architecture? Why is data pipeline architecture important? What is data pipeline architecture? Why is data pipeline architecture important?

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

Ascend.io

MAY 31, 2023

In the modern world of data engineering, two concepts often find themselves in a semantic tug-of-war: data pipeline and ETL. Fast forward to the present day, and we now have data pipelines. Data Ingestion Data ingestion is the first step of both ETL and data pipelines.

Data Pipeline 52

Data Pipeline ETL Tools Pipeline-centric Data Warehouse 52

Data Engineering Podcast

FEBRUARY 8, 2021

In this episode Zeeshan Qureshi and Michelle Ark share their experiences using DBT to manage the data warehouse for Shopify. Modern Data teams are dealing with a lot of complexity in their data pipelines and analytical code. What kinds of data sources are you working with?

Data Warehouse 100

Data Warehouse Building BI SQL 100

Data Engineering Podcast

JUNE 1, 2021

Summary Every part of the business relies on data, yet only a small team has the context and expertise to build and maintain workflows and data pipelines to transform, clean, and integrate it. RudderStack’s smart customer data pipeline is warehouse-first.

Data Pipeline 100

Data Pipeline Data Warehouse Data Workflow Data 100

Data Engineering Podcast

APRIL 19, 2021

Summary Most of the time when you think about a data pipeline or ETL job what comes to mind is a purely mechanistic progression of functions that move data from point A to point B. Modern Data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Data Pipeline 100

Data Pipeline Machine Learning Data Warehouse Datasets 100

Cloudera

JUNE 2, 2021

Data pipelines are in high demand in today’s data-driven organizations. As critical elements in supplying trusted, curated, and usable data for end-to-end analytic and machine learning workflows, the role of data pipelines is becoming indispensable.

Data Pipeline 91

Data Pipeline Data Warehouse Machine Learning Data Architect 91

Monte Carlo

AUGUST 25, 2023

At the same time, 81% of IT leaders say their C-suite has mandated no additional spending or a reduction of cloud costs. Data teams need to balance the need for robust, powerful data platforms with increasing scrutiny on costs. But, the options for data storage are evolving quickly. Let’s dive in.

Data Lake 52

Data Lake Data Warehouse Unstructured Data Raw Data 52

Grouparoo

OCTOBER 26, 2021

As a result, data has to be moved between the source and destination systems and this is usually done with the aid of data pipelines. What is a Data Pipeline? A data pipeline is a set of processes that enable the movement and transformation of data from different sources to destinations.

Data Pipeline 52

Data Pipeline ETL Tools ETL System Data Warehouse 52

Data Engineering Podcast

JANUARY 18, 2021

Summary The data warehouse has become the central component of the modern data stack. This is an interesting conversation about the importance of the data warehouse and how it can be used beyond just internal analytics. And don’t forget to thank them for their continued support of this show!

Data Warehouse 100

Data Warehouse BI Data Data Pipeline 100

Monte Carlo

OCTOBER 17, 2022

Platform and data warehouse migrations aren’t something you do everyday or even every few years, but they’re becoming much more frequent as organizations seek to modernize their data infrastructure with the new capabilities being offered by Snowflake, Databricks, Google, AWS, and others. Editor’s note: We agree.

Data Warehouse 59

Data Warehouse AWS MySQL Data 59

Striim

FEBRUARY 2, 2024

Introducing Striim Cloud for Application Integration: A fully managed, simple, and scalable SaaS service for application connectors. With this new application integration service, users can stream real-time CRM, ERP, Billing, and Payment data from their cloud applications to data warehouses in minutes with zero coding.

Cloud 52

Cloud Google Cloud Data Integration Data Pipeline 52

ProjectPro

JUNE 16, 2022

An AWS data pipeline helps businesses move and unify their data to support several data-driven initiatives. It enables flow from a data lake to an analytics database or an application to a data warehouse. This blog will teach you about AWS Data Pipeline, its architecture, components, and benefits.

Data Pipeline 59

Data Pipeline AWS Amazon Web Services Data Consolidation 59

Christophe Blefari

MARCH 15, 2024

Postgres creator launches DBOS, a transactional serverless computing platform — Mike sees DBOS like a cloud-native OS that runs on-top of the database in order to rethink application development and deployment. Coding data pipelines is faster than renting connector catalogs — This is something I've always believed.

Metadata 272

Metadata Datasets Data Data Warehouse 272

Data Engineering Podcast

OCTOBER 14, 2019

Summary Managing a data warehouse can be challenging, especially when trying to maintain a common set of patterns. They provide an AWS-native, serverless, data infrastructure that installs in your VPC. Datacoral helps data engineers build and manage the flow of data pipelines without having to manage any infrastructure.

Data Warehouse 100

Data Warehouse PostgreSQL AWS Programming Language 100

Cloudera

JUNE 17, 2022

In the second blog of the Universal Data Distribution blog series , we explored how Cloudera DataFlow for the Public Cloud (CDF-PC) can help you implement use cases like data lakehouse and data warehouse ingest, cybersecurity, and log optimization, as well as IoT and streaming data collection.

Data Pipeline 92

Data Pipeline Building Kafka Java 92

ProjectPro

AUGUST 11, 2021

“Data Lake vs Data Warehouse = Load First, Think Later vs Think First, Load Later” The terms data lake and data warehouse are frequently stumbled upon when it comes to storing large volumes of data. Data Warehouse Architecture What is a Data lake?

Data Lake 52

Data Lake Data Warehouse Cloud Hadoop 52

Ascend.io

APRIL 28, 2023

Data pipelines are having a moment — at least, that is, within the data world. That’s because as more and more businesses are adopting a data-driven mindset, the movement of data into and within organizations has never been a bigger priority.

Data Pipeline 52

Data Pipeline Data Data Warehouse Data Engineering 52

U-Next

SEPTEMBER 7, 2022

The terms “ Data Warehouse ” and “ Data Lake ” may have confused you, and you have some questions. There are times when the data is structured , but it is often messy since it is ingested directly from the data source. What is Data Warehouse? . Data Warehouse in DBMS: .

Data Lake 52

Data Lake Data Warehouse Unstructured Data Amazon Web Services 52

Workfall

AUGUST 14, 2023

Reading Time: 7 minutes In today’s data-driven world, efficient data pipelines have become the backbone of successful organizations. These pipelines ensure that data flows smoothly from various sources to its intended destinations, enabling businesses to make informed decisions and gain valuable insights.

Data Pipeline 62

Data Pipeline Data Raw Data Database 62

Data Engineering Podcast

FEBRUARY 4, 2024

Dagster offers a new approach to building and running data platforms and data pipelines. It is an open-source, cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability, a declarative programming model, and best-in-class testability.

SQL 173

SQL Data Lake High Quality Data Data Pipeline 173

DataKitchen

JULY 27, 2023

The Ten Standard Tools To Develop Data Pipelines In Microsoft Azure. While working in Azure with our customers, we have noticed several standard Azure tools people use to develop data pipelines and ETL or ELT processes. We counted ten ‘standard’ ways to transform and set up batch data pipelines in Microsoft Azure.

Data Pipeline 52

Data Pipeline BI Machine Learning Data Preparation 52

ProjectPro

DECEMBER 7, 2021

Data pipelines are a significant part of the big data domain, and every professional working or willing to work in this field must have extensive knowledge of them. Table of Contents What is a Data Pipeline? The Importance of a Data Pipeline What is an ETL Data Pipeline?

Data Pipeline 52

Data Pipeline Architecture Kafka AWS 52

Monte Carlo

APRIL 24, 2023

In an evolving data landscape, the explosion of new tooling solutions—from cloud-based transforms to data observability —has made the question of “build versus buy” increasingly important for data leaders. There are two primary types of raw data. Missed Nishith’s 5 considerations?

Data Pipeline 52

Data Pipeline Building Data Ingestion BI 52

Data Engineering Podcast

JANUARY 21, 2024

A substantial amount of the data that is being managed in these systems is related to customers and their interactions with an organization. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

Data Lake 147

Data Lake High Quality Data NoSQL Data Warehouse 147

Christophe Blefari

JANUARY 20, 2024

Data engineering inherits from years of data practices in US big companies. Hadoop initially led the way with Big Data and distributed computing on-premise to finally land on Modern Data Stack — in the cloud — with a data warehouse at the center. This is often linked to real-time.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Cloudera

NOVEMBER 12, 2020

Cloud has given us hope, with public clouds at our disposal we now have virtually infinite resources, but they come at a different cost – using the cloud means we may be creating yet another series of silos, which also creates unmeasurable new risks in security and traceability of our data. A solution.

Cloud 103

Cloud Data Warehouse Banking Data 103

Cloudera

SEPTEMBER 17, 2020

To tackle these challenges, we’re thrilled to announce CDP Data Engineering (DE) , the only cloud-native service purpose-built for enterprise data engineering teams. Native Apache Airflow and robust APIs for orchestrating and automating job scheduling and delivering complex data pipelines anywhere.

Data Pipeline 103

Data Pipeline Data Engineering Data Engineer Engineering 103

Cloudera

SEPTEMBER 9, 2018

We are proud to announce the general availability of Cloudera Altus Data Warehouse , the only cloud data warehousing service that brings the warehouse to the data. Modern data warehousing for the cloud. Cloudera Altus Data Warehouse is designed with agile data teams in mind.

Data Warehouse 40

Data Warehouse Metadata Cloud Storage Cloud 40

Data Engineering Podcast

APRIL 9, 2023

Summary The customer data platform is a category of services that was developed early in the evolution of the current era of cloud services for data processing. Now that the data warehouse has taken center stage a new approach of composable customer data platforms is emerging.

Data Lake 147

Data Lake Data Warehouse Machine Learning Data 147

DareData

JULY 29, 2020

With these points in mind, I argue that the biggest hurdle to the widespread adoption of these advanced techniques in the healthcare industry is not intrinsic to the industry itself, or in any way related to its practitioners or patients, but simply the current lack of high-quality data pipelines. What makes a good Data Pipeline?

Healthcare 52

Healthcare Data Pipeline Medical Pipeline-centric 52

Data Engineering Podcast

DECEMBER 28, 2022

Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days or even weeks. RudderStack helps you build a customer data platform on your warehouse or data lake.

Data Lake 130

Data Lake Data Warehouse Data Pipeline MongoDB 130

Data Engineering Weekly

MAY 26, 2024

[link] Tweeq: Tweeq Data Platform: Journey and Lessons Learned: Clickhouse, dbt, Dagster, and Superset Tweeq writes about its journey of building a data platform with cloud-agnostic open-source solutions and some integration challenges. It is refreshing to see an open stack after the Hadoop era.

Data Engineering 57

Data Engineering Data Engineer Engineering Google Cloud 57

Data Engineering Podcast

DECEMBER 25, 2022

Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days or even weeks. RudderStack helps you build a customer data platform on your warehouse or data lake.

Machine Learning 130

Machine Learning Systems Data Lake Data Warehouse 130

Christophe Blefari

APRIL 19, 2024

Fast News ⚡️ Theseus against really big data ( credits ) Principal Engineer — Although staffs and principals have been on the career ladder for a long time, there are very few articles on what it takes to become one of the greats. Up to 30TBs > Cloud warehouse or Spark Over 30TBs > Go Theseus.

MySQL 130

MySQL Data Datasets SQL 130

Cloudera

DECEMBER 15, 2022

Today, we are thrilled to share some new advancements in Cloudera’s integration of Apache Iceberg in CDP to help accelerate your multi-cloud open data lakehouse implementation. Multi-cloud deployment with CDP public cloud. Multi-cloud capability is now available for Apache Iceberg in CDP. Advanced capabilitie.

Cloud 78

Cloud Metadata Google Cloud Data Warehouse 78

Data Engineering Podcast

SEPTEMBER 29, 2021

There have been many attempts to harness all of the data that you generate for gaining useful insights about yourself, but they are generally difficult to set up and manage or require software development experience. Start trusting your data with Monte Carlo today! Start trusting your data with Monte Carlo today!

Cloud 100

Cloud Data Lake Business Intelligence Data 100