Monte Carlo

MARCH 12, 2024

As you add new data sources to your data stores, Dataplex leverages the structured and unstructured metadata with built-in data quality checks to maintain integrity. You can export these results to a BigQuery table, Looker dashboard, or other aggregated report to do a full analysis. Dataplex works with your metadata.

Google Cloud 52

Google Cloud Metadata SQL Data Lake 52

Data Engineering Podcast

NOVEMBER 6, 2022

Prefect specializes in glueing together the disparate pieces of a pipeline, and integrating with modern distributed compute libraries to bring power where you need it, when you need it. Especially once they realize 90% of all major data sources like Google Analytics, Salesforce, Adwords, Facebook, Spreadsheets, etc.,

Building 130

Building IT Metadata MongoDB 130

Data Engineering Podcast

OCTOBER 30, 2022

Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools.

Engineering 100

Engineering MongoDB Scala MySQL 100

Christophe Blefari

APRIL 28, 2023

This is mainly a roadmap of the MeticFlow integration within dbt ecosystem. At the moment we have a dbt Semantic Layer that correspond to YAML definitions and MetricFlow—which was Transform open-source project—that is able to understand the semantics to generates SQL. A lot of changes will happen to MetricFlow incl.

SQL 100

SQL Food PostgreSQL Data 100

Christophe Blefari

JULY 8, 2023

Aujourd'hui il s'occupe du produit chez Kestra, un orchestrateur open-source développé en France. The new dbt Semantic Layer Following the acquisition of Transform by dbt Labs a few months ago, dbt Core integrates MetricsFlow. PS : est-ce qu'une version française de mon contenu t'intéresse ?

Kafka 130

Kafka PostgreSQL Transportation Data 130

Data Engineering Podcast

JUNE 12, 2022

Open Source DataHub is running in production at several companies like Peloton, Optum, Udemy, Zynga and others. Ascend automates workloads on Snowflake, Databricks, BigQuery, and open source Spark, and can be deployed in AWS, Azure, or GCP. What are the types of storage and data systems that you integrate with?

Unstructured Data 130

Unstructured Data MongoDB Scala MySQL 130

Monte Carlo

OCTOBER 5, 2023

Enter the world of data clean rooms – the super secure havens where you can mix and mingle data from different sources to get insights without getting your hands dirty with the raw data. How data clean rooms work Data clean rooms combine and analyze different data sources without directly accessing the raw data.

Raw Data 52

Raw Data Medical Manufacturing AWS 52

Data Engineering Podcast

SEPTEMBER 25, 2022

Summary Data integration from source systems to their downstream destinations is the foundational step for any data product. In this episode Mark Van de Wiel explains how they integrated CDC functionality into their existing product, discusses the nuances of different approaches to change data capture from various sources.

Food 100

Food MongoDB Scala MySQL 100

Christophe Blefari

SEPTEMBER 1, 2023

There are unit, integration, functional and end-to-end tests. After Excel, Notebooks get a second youth — Meta explained how they schedule Jupyter Notebooks in production , Google announced the BigQuery studio with embedded Notebooks in the UI and Jupyter released Jupyter AI (you call it with %ai ) to bring Gen AI to the notebook.

Food 130

Food Data SQL Python 130

Monte Carlo

DECEMBER 13, 2023

Schema change monitors are critical because we aren’t always informed by the engineering team about schema changes in the source database,” Daniel says. We use BigQuery, and sometimes it just happens that a user will inadvertently set off a job that they don’t realize is really heavy on the database,” she says. “Or

BI 52

BI Data Ingestion SQL Data Warehouse 52

Data Engineering Podcast

OCTOBER 2, 2022

Ascend automates workloads on Snowflake, Databricks, BigQuery, and open source Spark, and can be deployed in AWS, Azure, or GCP. What are some of the other sources/pieces of information that you integrate into the lineage graph? Go to dataengineeringpodcast.com/ascend and sign up for a free trial.

IT 100

IT Food PostgreSQL MongoDB 100

Data Engineering Podcast

APRIL 19, 2021

Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. It builds your customer data warehouse and your identity graph on your data warehouse, with support for Snowflake, Google BigQuery, Amazon Redshift, and more. What is a property?

Data Pipeline 100

Data Pipeline Machine Learning Data Warehouse Datasets 100

Christophe Blefari

JUNE 16, 2023

This year ~90k people answered, they also integrate a small AI category to measure impact on dev work. Generating income from open source — Vadim shares how he makes money from all the different open-source projects he has. You can put space in BigQuery column names — The editors of blef.fr (me) have no comment.

Programming Language 130

Programming Language SQL PostgreSQL Data 130

Monte Carlo

MAY 31, 2023

So for Prasad Govekar , Director of Data Engineering, Data Science, and Data Analytics, sourcing the right technology to support his team was an absolute must. But the data team was ingesting large amounts of data from an ever-increasing number of sources, and the legacy system wasn’t able to perform at scale. No easy feat.

Google Cloud 52

Google Cloud Data Engineering Data Engineer Data 52

Data Engineering Podcast

JULY 3, 2022

Summary The perennial challenge of data engineers is ensuring that information is integrated reliably. In order to quickly identify if and how two data systems are out of sync Gleb Mezhanskiy and Simon Eskildsen partnered to create the open source data-diff utility. Go to dataengineeringpodcast.com/ascend and sign up for a free trial.

Data Integration 100

Data Integration MongoDB Scala MySQL 100

Data Engineering Podcast

NOVEMBER 13, 2022

Prefect specializes in glueing together the disparate pieces of a pipeline, and integrating with modern distributed compute libraries to bring power where you need it, when you need it. Especially once they realize 90% of all major data sources like Google Analytics, Salesforce, Adwords, Facebook, Spreadsheets, etc.,

Building 130

Building Metadata MongoDB MySQL 130

Data Engineering Podcast

APRIL 3, 2022

Open Source DataHub is running in production at several companies like Peloton, Optum, Udemy, Zynga and others. Signup for the SaaS product at dataengineeringpodcast.com/acryl Hey Data Engineering Podcast listeners, want to learn how the Joybird data team reduced their time spent building new integrations and managing data pipelines by 93%?

Designing 100

Designing Data Warehouse Data Engineering Data Engineer 100

Christophe Blefari

APRIL 14, 2023

In a nutshell they replaced Spark (EMR) in-memory transformations by BigQuery. Jeremy wrote a post that showcases how to do it. Hot takes on the Modern Data Stack — Matt gives 5 hot takes about the MDS.

Datasets 130

Datasets Data Deep Learning SQL 130

Data Engineering Podcast

OCTOBER 23, 2022

In this episode field CTO Manjot Singh shares his experiences as an early user of MySQL and MariaDB and explains how the suite of products being built on top of the open source foundation address the growing needs for advanced storage and analytical capabilities. Go to dataengineeringpodcast.com/ascend and sign up for a free trial.

Database 100

Database MySQL Cloud MongoDB 100

Data Engineering Podcast

FEBRUARY 22, 2021

Summary Data integration is a critical piece of every data pipeline, yet it is still far from being a solved problem. There are a number of managed platforms available, but the list of options for an open source system that supports a large variety of sources and destinations is still embarrasingly short.

Data Integration 100

Data Integration Data Warehouse Data Pipeline BI 100

Christophe Blefari

MAY 6, 2023

Dataform: schedule daily updates using Cloud Functions — Dataform is a solution Google bought a few years ago that is a dbt alternative but for BigQuery. They come up with a lot of news: they will implement Delta Sharing protocol, lower the storage for $118/TB to $25, partner with AWS and propose low-code data integration tool.

Programming Language 100

Programming Language Data Python Data Warehouse 100

Christophe Blefari

DECEMBER 23, 2022

Next year I plan to talk about: Data engineering and analytics engineering career paths State of the data integration—related to another 2023 project 📚 We have too many choices, my framework to take a decision Something you want me to write on? Soon we will have a data lineage tab in the BigQuery UI.

Data 100

Data Hadoop Data Engineering Data Engineer 100

Data Engineering Podcast

SEPTEMBER 11, 2022

Ascend automates workloads on Snowflake, Databricks, BigQuery, and open source Spark, and can be deployed in AWS, Azure, or GCP. There are numerous commercial and open source options for collecting, transforming, and integrating data. Go to dataengineeringpodcast.com/ascend and sign up for a free trial.

Data Pipeline 130

Data Pipeline Building MongoDB Scala 130

Rockset

JULY 29, 2022

You can aggregate and analyze relevant data from multiple sources without worrying about inconsistent and inaccessible data. This consistency promotes data integrity, so you can trust the insights to make informed decisions. BigQuery pricing has two main components: query processing costs and storage costs.

Data Analytics 52

Data Analytics Data Warehouse Datasets Google Cloud 52

Data Engineering Weekly

FEBRUARY 19, 2023

The blog mostly focuses on how it optimizes the batch pipeline using Airflow and BigQuery. Plum Living writes about integrating the dbt semantic layer with Superset and the developer workflow. Link to Register: [link] Promo Code: DataWeekly20 Sourygna Luangsay: BigQuery, Spark, or Dataflow?

Data Engineering 52

Data Engineering Data Engineer Engineering Google Cloud 52

Striim

FEBRUARY 10, 2023

Striim is a data integration platform that connects data from different applications and services to deliver real-time data analytics. With real-time analytics, Ciena’s customers no longer have to wait to see their updated data because it is displayed instantly after any changes are made in the source platforms.

Data Analytics 52

Data Analytics Telecommunication Insurance Google Cloud 52

Data Engineering Podcast

SEPTEMBER 18, 2022

Ascend automates workloads on Snowflake, Databricks, BigQuery, and open source Spark, and can be deployed in AWS, Azure, or GCP. What are the sources of information that are needed to be able to answer these questions? What are some of the sources of inefficiency that they are contending with?

Hospitality 100

Hospitality Food MongoDB Scala 100

Data Engineering Podcast

APRIL 26, 2021

Superset is an open source option that has been gaining popularity due to its flexibility and extensible feature set. He digs into how it integrates with your data stack, how you can extend it to fit your use case, and why open source systems are a good choice for your business intelligence. in a Superset dashboard?

Business Intelligence 100

Business Intelligence Data Warehouse Hadoop Data Pipeline 100

Christophe Blefari

DECEMBER 31, 2023

At the same time they released Mixtral 8x7B, the first open-source model of this calibre under an Apache Licence. And the weight are open-source as well. BigQuery now integrates DuetAI — to help you generate or complete SQL queries. It points to a cheat sheet that summarises what you need to know.

Data 100

Data Cloud Storage Datasets Python 100

Data Engineering Podcast

OCTOBER 9, 2022

In this episode Purvi Shah, the VP of Enterprise Big Data Platforms at American Express, explains how they have invested in the cloud to power this visibility and the complex suite of integrations they have built and maintained across legacy and modern systems to make it possible.

Food 100

Food MongoDB Scala MySQL 100

Data Engineering Podcast

MAY 29, 2022

Open Source DataHub is running in production at several companies like Peloton, Optum, Udemy, Zynga and others. The most important piece of any data project is the data itself, which is why it is critical that your data source is high quality. And don’t forget to thank them for their continued support of this show!

Cloud 100

Cloud Data Warehouse Data Lake Data 100

Data Engineering Weekly

APRIL 30, 2023

Watch On-demand Trivago: Implementing Data Validation with Great Expectations in Hybrid Environments The article by Trivago discusses the integration of data validation with Great Expectations. The author provides a few hacks while running dbt on BigQuery to save 💵 [link] All rights reserved ProtoGrowth Inc, India.

Data Engineering 62

Data Engineering Data Engineer Engineering PostgreSQL 62

Data Engineering Podcast

OCTOBER 16, 2022

Ascend automates workloads on Snowflake, Databricks, BigQuery, and open source Spark, and can be deployed in AWS, Azure, or GCP. What are some of the investments that Dremio has made to the broader open source/open lakehouse ecosystem? integrations, scalability, deployment models, etc.)

Data Lake 100

Data Lake Food MongoDB Scala 100

Data Engineering Podcast

JUNE 5, 2022

The team at Skyflow decided that the second best way is to build a storage system dedicated to securely managing your sensitive information and making it easy to integrate with your applications and data systems. Ascend automates workloads on Snowflake, Databricks, BigQuery, and open source Spark, and can be deployed in AWS, Azure, or GCP.

Data Security 100

Data Security Metadata MongoDB Scala 100

Data Engineering Podcast

MARCH 22, 2021

Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. It builds your customer data warehouse and your identity graph on your data warehouse, with support for Snowflake, Google BigQuery, Amazon Redshift, and more. What are the barriers to entry?

Data Warehouse 100

Data Warehouse Metadata Data Lake Hadoop 100

ProjectPro

FEBRUARY 6, 2023

Pull, Push, and BigQuery subscriptions are the three types of subscriptions that Pub/Sub offers. Developers can swiftly integrate systems hosted within the Google Cloud Platform and externally using Pub/Sub's support for durable messaging. The same subscription cannot be used for several topics.

Google Cloud 52

Google Cloud Cloud Cloud Storage Data Ingestion 52

Knowledge Hut

JUNE 26, 2023

Top Data Engineering Projects with Source Code Data engineers make unprocessed data accessible and functional for other data professionals. When constructing a data engineering project, you should prioritize the following areas: Multiple sources of data (APIs, websites, CSVs, JSON, etc.) Which queries do you have?

Data Engineering 98

Data Engineering Data Engineer Coding Project 98