Precisely

DECEMBER 28, 2023

Eventually, your streaming data pipeline delivers the information to its final destination. In many cases, that will be a cloud-based analytics platform like Snowflake , Databricks , or Amazon Redshift for real-time insights. Then it transforms the format and/or content of that data before sending it on its way.

Data Pipeline 91

Data Pipeline Building Kafka Big Data 91

Data Engineering Podcast

MARCH 22, 2021

Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. It builds your customer data warehouse and your identity graph on your data warehouse, with support for Snowflake, Google BigQuery, Amazon Redshift, and more. What are the barriers to entry?

Data Warehouse 100

Data Warehouse Metadata Data Lake Hadoop 100

Cloudera

JUNE 2, 2022

allowing developers to connect to any data source anywhere with any structure, process it, and deliver to any destination. I want integration between all my systems. That’s why we love that Cloudera uses NiFi and the way it integrates between all systems. Why does every organization need it when using a modern data stack?

Systems 99

Systems Data Lake Google Cloud Data Collection 99

Data Engineering Podcast

NOVEMBER 13, 2022

Prefect specializes in glueing together the disparate pieces of a pipeline, and integrating with modern distributed compute libraries to bring power where you need it, when you need it. Trusted by thousands of organizations and supported by over 20,000 community members, Prefect powers over 100MM business critical tasks a month.

Building 130

Building Metadata MongoDB MySQL 130

phData: Data Engineering

APRIL 4, 2023

With expansive storage capacity and support for multiple data formats, the data lake is a popular destination for teams doing analysis on massive data sets or running extensive data science projects that fuel their business. This nicely complements modern data architectures around a cloud data warehouse like Snowflake.

Data Lake 52

Data Lake Amazon Web Services Data Cleanse Data Warehouse 52

Monte Carlo

JANUARY 30, 2024

This case study is based on information shared in recent Snowflake webinars and Summit presentations. With over 1,000 flights per day servicing over 100 network destinations, JetBlue needs to ensure the data informing its operations and business is accurate, on-time, and reliable. Monte Carlo’s automated monitoring fit the bill.

Data 64

Data SQL Metadata Data Engineering 64

Data Engineering Podcast

OCTOBER 30, 2022

Prefect specializes in glueing together the disparate pieces of a pipeline, and integrating with modern distributed compute libraries to bring power where you need it, when you need it. What are the options for integrating Sisense with an organization’s data platform? Can you describe how the Sisense platform is implemented?

Business Intelligence 130

Business Intelligence Metadata MongoDB MySQL 130

Data Engineering Podcast

DECEMBER 4, 2022

Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools.

Data Lake 100

Data Lake MongoDB MySQL Data Warehouse 100

Data Engineering Podcast

FEBRUARY 22, 2021

Summary Data integration is a critical piece of every data pipeline, yet it is still far from being a solved problem. There are a number of managed platforms available, but the list of options for an open source system that supports a large variety of sources and destinations is still embarrasingly short.

Data Integration 100

Data Integration Data Warehouse Data Pipeline BI 100

Cloudyard

DECEMBER 13, 2022

Read Time: 4 Minute, 23 Second During this post we will discuss how AWS S3 service and Snowflake integration can be used as Data Lake in current organizations. How customer has migrated On Premises EDW to Snowflake to leverage snowflake Data Lake capabilities.

Data Lake 52

Data Lake AWS Metadata Data 52

Databand.ai

JULY 19, 2023

They trace and document the life cycle of data, from its origin to its various transformations and final destination. This allows businesses to identify any transformations or processes that may be compromising the data’s integrity. It provides context for data, making it easier to understand and manage.

Pipeline-centric 52

Pipeline-centric Data Governance Metadata Government 52

Data Engineering Podcast

SEPTEMBER 25, 2022

Summary Data integration from source systems to their downstream destinations is the foundational step for any data product. In this episode Mark Van de Wiel explains how they integrated CDC functionality into their existing product, discusses the nuances of different approaches to change data capture from various sources.

Food 100

Food MongoDB Scala MySQL 100

Data Engineering Podcast

DECEMBER 11, 2022

Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools.

Database 100

Database MySQL Data Lake MongoDB 100

Data Engineering Podcast

OCTOBER 9, 2022

In order to make the data lakehouse available to a wider audience the team at Iomete built an all-in-one service that handles management and integration of the various technologies so that you can worry about answering important business questions.

Metadata 100

Metadata AWS MongoDB MySQL 100

Snowflake

FEBRUARY 14, 2023

Here’s how a few of our retail customers are using Snowflake to optimize fulfillment and delivery, leading to a great customer experience. Overnight, the restaurant chain had to cut costs and radically transform from an in-person destination to a digital-first business. The company needed a solution that could scale with its growth.

Retail 52

Retail Portfolio Cloud Accessible 52

Data Engineering Podcast

MAY 10, 2021

It builds your customer data warehouse and your identity graph on your data warehouse, with support for Snowflake, Google BigQuery, Amazon Redshift, and more. Their SDKs and plugins make event streaming easy, and their integrations with cloud applications like Salesforce and ZenDesk help you go beyond event streaming. Sound exciting?

PostgreSQL 100

PostgreSQL Data Warehouse Data Pipeline Kafka 100

Data Engineering Podcast

JULY 10, 2022

Summary Building a data platform is a journey, not a destination. Beyond the work of assembling a set of technologies and building integrations across them, there is also the work of growing and organizing a team that can support and benefit from that platform.

Metadata 100

Metadata MongoDB MySQL Data Pipeline 100

Data Engineering Weekly

MARCH 19, 2023

Debezium Kafka Snowflake Airflow & dbt If anyone is confused with the modern data landscape, the blog is an excellent reminder that you only need a handful of systems to process your data. The blog discusses how these ML models integrate with the application to serve users. from the browser or mobile app). Moderation is essential.

Data Engineering 57

Data Engineering Data Engineer Engineering Media 57

Snowflake

AUGUST 16, 2023

At Snowflake Summit, we announced a wave of product innovations: Snowpark ML Modeling API, Snowflake Native App Framework, Dynamic Tables and more. Applications Snowflake Native App Framework now available in AWS – public preview Snowflake Native Apps are an entirely new way to put data to work. Learn more here.

Scala 52

Scala Transportation Kafka Data Lake 52

Striim

FEBRUARY 10, 2023

Striim is a data integration platform that connects data from different applications and services to deliver real-time data analytics. is one of America’s largest retailers, delivering quality fashion to customers in more than 100 international destinations through the leading e-commerce site macys.com. Macy’s, Inc.

Data Analytics 52

Data Analytics Telecommunication Insurance Google Cloud 52

Meltano

MARCH 30, 2023

This article will introduce you to isolated test environments and their benefits, and show you how to implement an isolated test environment with Meltano and Snowflake. Continuous integration (CI): Isolated testing can be incorporated into the CI process (build stage) to ensure code is ready for immediate deployment.

Metadata 59

Metadata Database Python Coding 59

Grouparoo

JULY 18, 2021

Grouparoo Cloud helps make deployments quick and easy, optimizes Infrastructure for your Grouparoo instance, and provides Github integration for Continuous Deployment and Testing. This includes aspects such as config-as-code checked into git, tests and continuous integration, code reviews, staging environments, and production deployments.

Data Warehouse 52

Data Warehouse Data Integration Cloud Coding 52

Monte Carlo

APRIL 24, 2023

Traditionally, after being stored in a data lake, raw data was then often moved to various destinations like a data warehouse for further processing, analysis, and consumption. With our new partnership and updated integration, Monte Carlo provides full, end-to-end coverage across data lake and lakehouse environments powered by Databricks.

Data Lake 52

Data Lake Google Cloud Data Warehouse AWS 52

Workfall

AUGUST 14, 2023

These pipelines ensure that data flows smoothly from various sources to its intended destinations, enabling businesses to make informed decisions and gain valuable insights. DBT and Airflow Integration Best Practices Scalability and Flexibility Hands-On Conclusion Understanding DBT and Airflow What is DBT? What is Airflow?

Data Pipeline 62

Data Pipeline Raw Data Data Database 62

Data Engineering Podcast

MARCH 8, 2021

This introduces challenges such as keeping information synchronized, managing schema evolution, building transformations to match the expectations of the destination systems. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows.

IT 100

IT Data Warehouse MongoDB Kafka 100

Knowledge Hut

APRIL 25, 2023

Data integration: Data engineers should be able to integrate data from various sources like databases, APIs, or file systems, using tools like Apache NiFi, Fivetran, or Talend. Key features: Robust data visualization capabilities Seamless integration with Microsoft tools Easy-to-use interface 2. Let’s take a look: 1.

Data Engineering 52

Data Engineering Data Engineer Engineering Google Cloud 52

Monte Carlo

FEBRUARY 15, 2023

And as pipelines become increasingly complex with the adoption of distributed platform architectures, you can bet that data quality issues are destined to multiply right along with them. The back-end architecture of our field-level lineage solution, built on top of Snowflake and Elasticsearch. COLUMN LEVEL LINEAGE ARCHITECTURE.

Data Validation 52

Data Validation Data Data Engineering Data Engineer 52

Monte Carlo

FEBRUARY 15, 2023

And as pipelines become increasingly complex with the adoption of distributed platform architectures, you can bet that data quality issues are destined to multiply right along with them. The back-end architecture of our field-level lineage solution, built on top of Snowflake and Elasticsearch. COLUMN LEVEL LINEAGE ARCHITECTURE.

Data Validation 52

Data Validation Data Data Engineering Data Engineer 52

Databand.ai

JULY 19, 2023

Despite these challenges, proper data acquisition is essential to ensure the data’s integrity and usefulness. Data Loading Data loading is where the transformed data is loaded into a data warehouse or any other desired destination for further analysis or reporting. Data loading is the culmination of the data ingestion process.

Data Ingestion 52

Data Ingestion Process Data Cleanse Data Governance 52

FreshBI

JANUARY 31, 2022

When we ask our analysts to extract data from various sources, find a way to integrate them, keep these integrations up to date and deliver intelligence solutions, we risk the chance to act on insights that could have been delivered when the time was right.

BI 52

BI Data Engineering Data Engineer Engineering 52

ProjectPro

DECEMBER 7, 2021

To understand the working of a data pipeline, one can consider a pipe that receives input from a source that is carried to give output at the destination. Organizations have built robust data pipelines to mitigate such problems by consolidating data from all distinct data sources into one common destination.

Data Pipeline 52

Data Pipeline Architecture Kafka AWS 52

Monte Carlo

FEBRUARY 21, 2023

A data fabric isn’t a standalone technology—it’s a data management architecture that leverages an integrated data layer atop underlying data in order to empower business leaders with real-time analytics and data-driven insights. Table of Contents What is a data fabric? Multiple benefits characterize the data fabric.

Data Architecture 52

Data Architecture Architecture Metadata Unstructured Data 52

Monte Carlo

FEBRUARY 21, 2023

A data fabric isn’t a standalone technology—it’s a data management architecture that leverages an integrated data layer atop underlying data in order to empower business leaders with real-time analytics and data-driven insights. Table of Contents What is a data fabric? Multiple benefits characterize the data fabric.

Data Architecture 52

Data Architecture Architecture Metadata Unstructured Data 52

Christophe Blefari

MARCH 2, 2024

Extract and load, still unsolved 🤭 I've started writing data pipelines in 2014 and the movement from sources to destinations has always been one of the most discussed topic in my data engineering spaces. In 2024 we are more than ever tools to move data from sources to destinations. But the field has taken a new direction.

Data 162

Data Python Database BI 162

Monte Carlo

AUGUST 8, 2023

The Definitive Guide to Data Validation Testing Data validation testing ensures your data maintains its quality and integrity as it is transformed and moved from its source to its target destination. Referential Integrity Checking : This technique checks if a data value references an existing value in another table.

Data Validation 52

Data Validation Data Pipeline SQL Data 52

Snowflake

JANUARY 29, 2024

Imagine seamlessly integrating your Google Analytics data with Snowflake, allowing you to combine it effortlessly with other key sources like CRM, ERP, social media metrics, email campaign data, and whatever data sources compose the full scope of your data estate. Here’s a quick guide to get started: 1.

Raw Data 99

Raw Data Aggregated Data Data Government 99

Monte Carlo

APRIL 16, 2022

This is not out of line with the general efforts of providers like Snowflake and Databricks to ensure their customers are being efficient (in other words, saving money and resources) in their storage and consumption. Not to mention, loading (the “L” in ELT) had become much easier to integrate in the cloud.

Data Warehouse 52

Data Warehouse Architecture Data Government 52