Data Engineering Podcast

DECEMBER 17, 2023

Summary The "modern data stack" promised a scalable, composable data platform that gave everyone the flexibility to use the best tools for every job. The team at 5X understand the pain involved and the barriers to productivity and set out to solve it by pre-integrating the best tools from each layer of the stack.

Data Lake 130

Data Lake High Quality Data SQL Architecture 130

Data Engineering Podcast

MARCH 10, 2023

Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools. There are many different players (both commercial and open source) in the product analytics space. When is NetSpring the wrong choice? Data Council : !

Data Warehouse 130

Data Warehouse Data Lake Machine Learning Data Science 130

Data Engineering Podcast

JANUARY 7, 2022

Summary Reverse ETL is a product category that evolved from the landscape of customer data platforms with a number of companies offering their own implementation of it. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm.

ETL System 100

ETL System Data Pipeline Data Warehouse Architecture 100

Christophe Blefari

SEPTEMBER 28, 2023

This era was necessary and opened doors to the future, fostering innovation. With a handy warehouse at the center and multiple SaaS tools revolving around to answer useful—sometimes not—use-cases. When you put the things like this it opens the doors and does not limit the modern data stack to 4 vendors.

Cloud Storage 147

Cloud Storage Big Data Hadoop SQL 147

Monte Carlo

FEBRUARY 20, 2024

The growing field of data ingestion tools offers a range of answers, each with implications to ponder. Let’s explore what to consider when thinking about data ingestion tools and explore the leading tools in the field.

Data Ingestion 52

Data Ingestion Google Cloud Kafka AWS 52

Data Engineering Weekly

DECEMBER 18, 2023

Visit rudderstack.com to learn more. link] Instacart: Monte Carlo, Puppetry and Laughter: The Unexpected Joys of Prompt Engineering Instacart writes an article exploring prompt techniques used for internal productivity tooling. RudderStack just launched Trino as a Reverse ETL source. for certain applications.

Data Engineering 67

Data Engineering Data Engineer Engineering Food 67

Monte Carlo

JANUARY 23, 2024

One of the first data engineers at Facebook and Airbnb, he wrote and open sourced the wildly popular orchestrator, Apache Airflow , followed shortly thereafter by Apache Superset , a data exploration tool that’s taking the data viz landscape by storm. Enter, the data engineer. On the flipside, costs are more constrained.

Data Engineering 98

Data Engineering Data Engineer Engineering Data 98

Data Engineering Podcast

DECEMBER 21, 2021

and the various ways that it is being used in the open source community. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. What are some of the data modeling steps that are needed in the source systems?

Data Analytics 130

Data Analytics BI Computer Science SQL 130

Data Engineering Podcast

DECEMBER 26, 2021

In this episode Zhamak re-joins the show to discuss the real world benefits that have been seen, the lessons that she has learned while working with her clients and the community, and her vision for the future of the data mesh. Go to dataengineeringpodcast.com/atlan today and sign up for a free trial.

BI 130

BI Data Warehouse Data Engineering Data Engineer 130

Data Engineering Podcast

MARCH 27, 2022

With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. Open Source DataHub is running in production at several companies like Peloton, Optum, Udemy, Zynga and others. When is Privacera the wrong choice?

Data Governance 130

Data Governance Government Cloud Building 130

Data Engineering Podcast

MAY 22, 2022

Summary Machine learning has become a meaningful target for data applications, bringing with it an increase in the complexity of orchestrating the entire data flow. Flyte is a project that was started at Lyft to address their internal needs for machine learning and integrated closely with Kubernetes as the execution manager.

Machine Learning 130

Machine Learning Data Engineering Data Engineer Cloud 130

Cloudera

JUNE 26, 2023

Over time, using the wrong tool for the job can wreak havoc on environmental health. Take precaution using CDSW as an all-purpose workflow management and scheduling tool. For data engineering and data science teams, CDSW is highly effective as a comprehensive platform that trains, develops, and deploys machine learning models.

ETL Tools 76

ETL Tools Programming Language Datasets Data Pipeline 76

ProjectPro

FEBRUARY 8, 2023

Do ETL and data integration activities seem complex to you? In 2023, more than 5140 businesses worldwide have started using AWS Glue as a big data tool. The popular data integration tool, AWS Glue, enables data analytics users to quickly acquire, analyze, migrate, and integrate data from multiple sources. billion by 2026?

AWS 98

AWS Scala Metadata Data Lake 98

Knowledge Hut

JANUARY 30, 2024

However, this valuable data often lies scattered across various sources, like hidden gems waiting to be discovered and polished. Whether you're a seasoned data scientist or just stepping into the world of data, come with me as we unravel the secrets of data extraction and learn how it empowers us to unleash the full potential of data.

ETL Tools 52

ETL Tools Database-centric Data Mining Data Cleanse 52

Data Engineering Podcast

OCTOBER 22, 2021

Lightdash has fully embraced that shift by building an entire open source business intelligence framework that is powered by dbt models. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm.

Engineering 100

Engineering Business Intelligence BI Data Warehouse 100

Data Engineering Podcast

JUNE 12, 2022

Open Source DataHub is running in production at several companies like Peloton, Optum, Udemy, Zynga and others. Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool. When is Aparavi the wrong choice?

Unstructured Data 130

Unstructured Data MongoDB Scala MySQL 130

Towards Data Science

FEBRUARY 9, 2024

Fivetran still seems like the undeniable leader in the managed ETL category, but it has some stiff competition via up & comers like Airbyte and big cloud providers that have been strengthening their platform offerings. These include open-source options like Weaviate , managed solutions like Pinecone and many more.

Building 81

Building Transportation Data Lake Metadata 81

Data Engineering Podcast

OCTOBER 23, 2022

In this episode field CTO Manjot Singh shares his experiences as an early user of MySQL and MariaDB and explains how the suite of products being built on top of the open source foundation address the growing needs for advanced storage and analytical capabilities. Data teams are increasingly under pressure to deliver.

Database 100

Database MySQL Cloud MongoDB 100

Knowledge Hut

APRIL 19, 2023

In today's data-driven world, businesses need to extract, transform, and load data from multiple data sources because of the large amount of data which businesses generate. An ETL pipeline is one of the most common solutions for the efficient processing of large data. What is ETL? ETL stands for Extract, Transform, and Load.

Data Warehouse 52

Data Warehouse Business Intelligence ETL Tools Data Pipeline 52

Data Engineering Podcast

DECEMBER 11, 2021

In order to make it easier for developers to build customer profiles in a way that respects their privacy Serge Huber helped to create the Apache Unomi framework as an open source customer data platform. Visit dataengineeringpodcast.com/montecarlo to learn more. Start trusting your data with Monte Carlo today!

Data Warehouse 100

Data Warehouse Raw Data Data Lake BI 100

Netflix Tech

MARCH 7, 2024

Berg , Romain Cledat , Kayla Seeley , Shashank Srikanth , Chaoying Wang , Darin Yu Netflix uses data science and machine learning across all facets of the company, powering a wide range of business applications from our internal infrastructure and content demand modeling to media understanding.

Systems 90

Systems Media Machine Learning Data Warehouse 90

Data Engineering Podcast

JULY 3, 2022

In order to quickly identify if and how two data systems are out of sync Gleb Mezhanskiy and Simon Eskildsen partnered to create the open source data-diff utility. Ascend automates workloads on Snowflake, Databricks, BigQuery, and open source Spark, and can be deployed in AWS, Azure, or GCP.

Data Integration 100

Data Integration MongoDB Scala MySQL 100

Data Engineering Podcast

JUNE 19, 2022

Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool. Ascend automates workloads on Snowflake, Databricks, BigQuery, and open source Spark, and can be deployed in AWS, Azure, or GCP.

Metadata 130

Metadata MongoDB Scala MySQL 130

DareData

JULY 5, 2023

Learn how we build data lake infrastructures and help organizations all around the world achieving their data goals. Data Sources: How different are your data sources? The Data Lake acts as the central repository for aggregating data from diverse sources in its raw format. And what is their format?

Data Lake 52

Data Lake Building Raw Data ETL Tools 52

Data Engineering Podcast

OCTOBER 8, 2021

Business intelligence tools have provided this capability for years, but they don’t offer a means of exposing those metrics to other systems. Metriql is an open source project that provides a headless BI system where you can define your metrics and share them with all of your other processes.

BI 100

BI Business Intelligence SQL Data Warehouse 100

Knowledge Hut

APRIL 5, 2024

DevOps practices in collaboration with relevant tools and techniques, motivate organizations to complete tasks as effectively as possible. DevOps online classes provide a great way to learn more about the best DevOps practices and enhance your workplace experience. Following ETL execution, the operations team runs tests daily.

Coding 52

Coding Java Project Programming 52

Data Engineering Podcast

APRIL 26, 2021

Superset is an open source option that has been gaining popularity due to its flexibility and extensible feature set. He digs into how it integrates with your data stack, how you can extend it to fit your use case, and why open source systems are a good choice for your business intelligence.

Business Intelligence 100

Business Intelligence Data Warehouse Hadoop Data Pipeline 100

Data Engineering Podcast

OCTOBER 16, 2022

Dremio is one of the companies leading the development of products and services that support the open lakehouse. In this episode Jason Hughes explains what it means for a lakehouse to be "open" and describes the different components that the Dremio team build and contribute to.

Data Lake 100

Data Lake Food MongoDB Scala 100

Christophe Blefari

JUNE 16, 2023

Yann's vision goes toward AI systems learning and reasoning like animals and humans. Deep multi-task learning and real-time personalisation for closeup recommendations — Pinterest still doing deep learning. The ultimate SQL guide — After the last canva on data interviews, here's a canva to learn SQL.

Programming Language 130

Programming Language SQL PostgreSQL Data 130

Monte Carlo

AUGUST 10, 2023

Data observability tools employ automated monitoring, root cause analysis, data lineage, and data health insights to proactively detect, resolve, and prevent data anomalies. Volume : Volume refers to the completeness of your data tables and offers insights on the health of your data sources. Unfortunately, this problem wasn’t unique.

Data Pipeline 98

Data Pipeline Software Engineer Software Engineering Machine Learning 98

Data Engineering Podcast

JUNE 5, 2022

Open Source DataHub is running in production at several companies like Peloton, Optum, Udemy, Zynga and others. Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool. When is Y42 the wrong choice?

MongoDB 100

MongoDB MySQL Data Lake Data Warehouse 100

Data Engineering Podcast

APRIL 24, 2022

Summary There are very few tools which are equally useful for data engineers, data scientists, and machine learning engineers. WhyLogs is a powerful library for flexibly instrumenting all of your data systems to understand the entire lifecycle of your data from source to productionized model.

Machine Learning 100

Machine Learning Systems Data Lake Metadata 100

Snowflake

SEPTEMBER 7, 2023

Snowpark External Access is leveraged to build a Ingest and Reverse ETL data pipeline for production workload. Ingest data from source systems: Customers looking to ingest data from sources, such as Twitter, Google Sheets, MySQL or other data sources available on the public internet, can use External Access.

Amazon Web Services 111

Amazon Web Services AWS Government Python 111

Data Engineering Podcast

OCTOBER 15, 2021

With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. Are you bored with writing scripts to move data into SaaS tools like Salesforce, Marketo, or Facebook Ads? No more scripts, just SQL.

Metadata 100

Metadata BI Data Warehouse Government 100

Data Engineering Podcast

NOVEMBER 10, 2021

Summary A significant source of friction and wasted effort in building and integrating data management systems is the fragmentation of metadata across various tools. Are you bored with writing scripts to move data into SaaS tools like Salesforce, Marketo, or Facebook Ads? Start trusting your data with Monte Carlo today!

Metadata 100

Metadata Data Warehouse Data Lake BI 100

ProjectPro

FEBRUARY 8, 2023

ETL developers play a vital role in designing, implementing, and maintaining the processes that help organizations extract valuable business insights from data. What is an ETL Developer? The purpose of ETL is to provide a centralized, consistent view of the data used for reporting and analysis.

ETL Tools 52

ETL Tools Data Cleanse Data Warehouse Big Data 52

RandomTrees

APRIL 18, 2024

Among the various tools available for data integration, Informatica and Talend stand out as popular choices, each with its strengths and capabilities. On the other hand, Talend provides a comprehensive suite of open-source tools for data integration, offering similar functionalities with a focus on ease of use and flexibility.

Coding 52

Coding Retail Metadata Python 52