Data Lake and Python - Data Engineering Digest

Charting A Path For Streaming Data To Fill Your Data Lake With Hudi

Data Engineering Podcast

AUGUST 3, 2021

Summary Data lake architectures have largely been biased toward batch processing workflows due to the volume of data that they are designed for. With more real-time requirements and the increasing use of streaming data there has been a struggle to merge fast, incremental updates with large, historical analysis.

Data Lake

Data Lake Data Warehouse Hadoop Architecture

Insights And Advice On Building A Data Lake Platform From Someone Who Learned The Hard Way

Data Engineering Podcast

MAY 15, 2022

Summary Designing a data platform is a complex and iterative undertaking which requires accounting for many conflicting needs. Designing a platform that relies on a data lake as its central architectural tenet adds additional layers of difficulty. When is a data lake architecture the wrong choice?

Data Lake

Data Lake Building BI Architecture

Tips to Build a Robust Data Lake Infrastructure

DareData

JULY 5, 2023

Learn how we build data lake infrastructures and help organizations all around the world achieving their data goals. In today's data-driven world, organizations are faced with the challenge of managing and processing large volumes of data efficiently.

Data Lake

Data Lake Building Raw Data ETL Tools

Webinars

How to Optimize the Developer Experience for Monumental Impact

Generative AI Deep Dive: Advancing from Proof of Concept to Production

Understanding User Needs and Satisfying Them

Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You Need to Know

Leading the Development of Profitable and Sustainable Products

MORE WEBINARS

Top Data Lake Vendors (Quick Reference Guide)

Monte Carlo

APRIL 24, 2023

Data lakes are useful, flexible data storage repositories that enable many types of data to be stored in its rawest state. Traditionally, after being stored in a data lake, raw data was then often moved to various destinations like a data warehouse for further processing, analysis, and consumption.

Data Lake

Data Lake Google Cloud Data Warehouse AWS

Add Version Control To Your Data Lake With LakeFS

Data Engineering Podcast

NOVEMBER 2, 2020

Summary Data lakes are gaining popularity due to their flexibility and reduced cost of storage. Along with the benefits there are some additional complexities to consider, including how to safely integrate new data sources or test out changes to existing pipelines. Closing Announcements Thank you for listening!

Data Lake

Data Lake PostgreSQL Datasets Machine Learning

Data News — Week 24.08

Christophe Blefari

FEBRUARY 23, 2024

SQL data engineer — There's a big discussion in the community about what real data engineering is. Is it Java/Scala or Python? Is it lake or warehouse? It's a sterile debate: both are useful and can serve different organisations with different service level for data users and stakeholders.

Data Lake

Data Lake PostgreSQL MongoDB Scala

Python for Data Engineering

Ascend.io

SEPTEMBER 14, 2023

As the demand to efficiently collect, process, and store data increases, data engineers have started to rely on Python to meet this escalating demand. In this article, our primary focus will be to unpack the reasons behind Python’s prominence in the data engineering domain. Why Python for Data Engineering?

Data Engineering

Data Engineering Data Engineer Python Engineering

KDnuggets News, January 18: 7 Best Platforms to Practice SQL • Explainable AI: 10 Python Libraries for Demystifying Your Model’s Decisions

KDnuggets

JANUARY 18, 2023

7 Best Platforms to Practice SQL • Explainable AI: 10 Python Libraries for Demystifying Your Model's Decisions • ChatGPT: Everything You Need to Know • Data Lakes and SQL: A Match Made in Data Heaven • Google Data Analytics Certification Review for 2023

SQL

SQL Python Data Lake Certification

Cloud Native Data Orchestration For Machine Learning And Data Engineering With Flyte

Data Engineering Podcast

MAY 22, 2022

Acryl Data provides DataHub as an easy to consume SaaS product which has been adopted by several companies. Signup for the SaaS product at dataengineeringpodcast.com/acryl RudderStack helps you build a customer data platform on your warehouse or data lake. Stop struggling to speed up your data lake.

Machine Learning

Machine Learning Data Engineering Data Engineer Cloud

The View Below The Waterline Of Apache Iceberg And How It Fits In Your Data Lakehouse

Data Engineering Podcast

FEBRUARY 19, 2023

Because of their complete ownership of your data they constrain the possibilities of what data you can store and how it can be used. Can you describe what Iceberg is and its position in the data lake/lakehouse ecosystem? __init__ covers the Python language, its community, and the innovative ways it is being used.

IT

IT Data Lake Metadata Data Warehouse

How to Keep Track of Data Versions Using Versatile Data Kit

Towards Data Science

MAY 3, 2023

One such tool is the Versatile Data Kit (VDK), which offers a comprehensive solution for controlling your data versioning needs. VDK helps you easily perform complex operations, such as data ingestion and processing from different sources, using SQL or Python. Use VDK to build a data lake and merge multiple sources.

Data Lake

Data Lake Data SQL Data Warehouse

Now in Public Preview: Processing Files and Unstructured Data with Snowpark for Python

Snowflake

JULY 10, 2023

Announced at Summit, we’ve recently added to Snowpark the ability to process files programmatically, with Python in public preview and Java generally available. Previously, working with these large and complex files would require a unique set of tools, creating data silos. Mike Tuck, Air Pollution Specialist Why unstructured data?

Unstructured Data

Unstructured Data Python Process Scala

An Exploration Of The Open Data Lakehouse And Dremio's Contribution To The Ecosystem

Data Engineering Podcast

OCTOBER 16, 2022

Summary The "data lakehouse" architecture balances the scalability and flexibility of data lakes with the ease of use and transaction support of data warehouses. Mention the podcast to get a free "In Data We Trust World Tour" t-shirt. Closing Announcements Thank you for listening!

Data Lake

Data Lake Food MongoDB Scala

How to learn data engineering

Christophe Blefari

JANUARY 20, 2024

The Rise of the Data Engineer The Downfall of the Data Engineer Functional Data Engineering — a modern paradigm for batch data processing There is a global consensus stating that you need to master a programming language (Python or Java based) and SQL in order to be self-sufficient.

Data Engineering

Data Engineering Data Engineer Engineering Hadoop

Adopting Real-Time Data At Organizations Of Every Size

Data Engineering Podcast

DECEMBER 4, 2022

Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. RudderStack helps you build a customer data platform on your warehouse or data lake. __init__ covers the Python language, its community, and the innovative ways it is being used.

Data Lake

Data Lake MongoDB MySQL Data Warehouse

How to Use ChatGPT ETL Prompts For Your ETL Game

Monte Carlo

DECEMBER 4, 2023

Can you help me craft a Python script that uses the tweepy library to get the most recent 100 tweets from a given user handle? Can you provide a Python snippet using the pandas library to help me filter this data? Normalization I have a DataFrame in Python with a column salary that contains employee salaries.

PostgreSQL

PostgreSQL Data Lake ETL Tools MySQL

Run Your Applications Worldwide Without Worrying About The Database With Planetscale

Data Engineering Podcast

DECEMBER 11, 2022

Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. RudderStack helps you build a customer data platform on your warehouse or data lake. __init__ covers the Python language, its community, and the innovative ways it is being used.

Database

Database MySQL Data Lake MongoDB

Data News — Week 23.09

Christophe Blefari

MARCH 4, 2023

AWS lambdas are still on Python 3.9 — Corey rant about AWS lambdas that are still using Python 3.9 while all the competition upgraded to at least Python 3.10. This is funny to see their offering because they offer a "managed data warehouse storage", which means without the compute. At least for me.

Machine Learning

Machine Learning AWS Data Data Lake

Data News — Week 23.08

Christophe Blefari

FEBRUARY 24, 2023

Data Engineers and Kubernetes — A 101 guide about Kubernetes concepts and why you should as a data engineers understand kube basic entities. Data Economy 💰 Qbeast raises €2.5m This is interesting to see that data lakes platforms can still raise money in 2023.

Kafka

Kafka Data Lake Data Storage Data

Build A Data Lake For Your Security Logs With Scanner

Data Engineering Podcast

JANUARY 28, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data.

Data Lake

Data Lake Building High Quality Data AWS

Clean Up Your Data Using Scalable Entity Resolution And Data Mastering With Zingg

Data Engineering Podcast

NOVEMBER 6, 2022

RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control.

MongoDB

MongoDB Scala MySQL Data Lake

Top 10 Azure Data Engineer Job Opportunities in 2024 [Career Options]

Knowledge Hut

MARCH 28, 2024

Role Level Intermediate Responsibilities Design and develop data pipelines to ingest, process, and transform data. Implemented and managed data storage solutions using Azure services like Azure SQL Database , Azure Data Lake Storage, and Azure Cosmos DB. Machine learning frameworks (e.g.,

Data Engineering

Data Engineering Data Engineer Engineering Data Warehouse

Discover And De-Clutter Your Unstructured Data With Aparavi

Data Engineering Podcast

JUNE 12, 2022

Acryl Data provides DataHub as an easy to consume SaaS product which has been adopted by several companies. Signup for the SaaS product at dataengineeringpodcast.com/acryl RudderStack helps you build a customer data platform on your warehouse or data lake. Closing Announcements Thank you for listening!

Unstructured Data

Unstructured Data MongoDB Scala MySQL

Let Your Analysts Build A Data Lakehouse With Cuelake

Data Engineering Podcast

AUGUST 20, 2021

Summary Data lakes have been gaining popularity alongside an increase in their sophistication and usability. Despite improvements in performance and data architecture they still require significant knowledge and experience to deploy and manage. Can you describe what Cuelake is and the story behind it?

Building

Building Data Lake Data Warehouse SQL

How to Become an Azure Data Engineer? 2023 Roadmap

Knowledge Hut

NOVEMBER 17, 2023

To provide end users with a variety of ready-made models, Azure Data engineers collaborate with Azure AI services built on top of Azure Cognitive Services APIs. The data engineers are responsible for creating conversational chatbots with the Azure Bot Service and automating metric calculations using the Azure Metrics Advisor.

Data Engineering

Data Engineering Data Engineer Engineering Scala

How Software Bill of Materials change the dependency game

Zalando Engineering

APRIL 12, 2023

We publish a curated data set containing dependency data from the SBOM for every application we deploy, based on its Container image. The data set is available in our data lake and thus can be easily queried and visualized by any engineer. Looking at the data, the number of dependencies grows exponentially.

Java

Java Scala Python Metadata

Joe Reis Flips The Script And Interviews Tobias Macey About The Data Engineering Podcast

Data Engineering Podcast

JULY 17, 2022

RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. It’s a lot of work.

Data Engineering

Data Engineering Data Engineer Engineering MongoDB

Simplify Your Data Architecture With The Presto Distributed SQL Engine

Data Engineering Podcast

SEPTEMBER 7, 2020

In this episode he explains how it is designed to allow for querying and combining data where it resides, the use cases that such an architecture unlocks, and the innovative ways that it is being employed at companies across the world. What are the tradeoffs of using Presto on top of a data lake vs a vertically integrated warehouse solution?

Architecture

Architecture Data Architecture SQL Engineering

Moving Past ETL and ELT: Understanding the EtLT Approach

Ascend.io

AUGUST 31, 2023

Secondly , the rise of data lakes that catalyzed the transition from ELT to ELT and paved the way for niche paradigms such as Reverse ETL and Zero-ETL. Still, these methods have been overshadowed by EtLT — the predominant approach reshaping today’s data landscape.

Data Lake

Data Lake ETL Tools Data Warehouse Data Pipeline

Scaling Analysis of Connected Data And Modeling Complex Relationships With The TigerGraph Graph Database

Data Engineering Podcast

MAY 8, 2022

Acryl Data provides DataHub as an easy to consume SaaS product which has been adopted by several companies. Signup for the SaaS product at dataengineeringpodcast.com/acryl RudderStack helps you build a customer data platform on your warehouse or data lake. Closing Announcements Thank you for listening!

Database

Database Data Lake BI Business Intelligence

Make Data Lineage A Ubiquitous Part Of Your Work By Simplifying Its Implementation With Alvin

Data Engineering Podcast

OCTOBER 2, 2022

Mention the podcast to get a free "In Data We Trust World Tour" t-shirt. RudderStack helps you build a customer data platform on your warehouse or data lake. The Ascend Data Automation Cloud provides a unified platform for data ingestion, transformation, orchestration, and observability.

IT

IT Food PostgreSQL MongoDB

Taking A Look Under The Hood At CreditKarma's Data Platform

Data Engineering Podcast

NOVEMBER 13, 2022

Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. RudderStack helps you build a customer data platform on your warehouse or data lake. __init__ covers the Python language, its community, and the innovative ways it is being used.

MongoDB

MongoDB Scala Google Cloud MySQL

Analytics Engineering Without The Friction Of Complex Pipeline Development With Optimus and dbt

Data Engineering Podcast

OCTOBER 30, 2022

Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. RudderStack helps you build a customer data platform on your warehouse or data lake. __init__ covers the Python language, its community, and the innovative ways it is being used.

Engineering

Engineering MongoDB Scala MySQL

Data Science Foundations & Learning Path

Knowledge Hut

APRIL 26, 2024

Analysts also do data mining and modelling tasks occasionally, but typically prefer to use visual tools for data science activities, such as IBM SPSS Modeler, Rapid Miner, SAS, and KNIME. Data scientists are also more accountable for teaching linear, non-linear algorithms in mathematical models.

Data Science

Data Science Machine Learning Hadoop Programming Language

Going From Transactional To Analytical And Self-managed To Cloud On One Database With MariaDB

Data Engineering Podcast

OCTOBER 23, 2022

Mention the podcast to get a free "In Data We Trust World Tour" t-shirt. RudderStack helps you build a customer data platform on your warehouse or data lake. The Ascend Data Automation Cloud provides a unified platform for data ingestion, transformation, orchestration, and observability.

Database

Database MySQL Cloud MongoDB

Power Your Real-Time Analytics Without The Headache Using Fivetran's Change Data Capture Integrations

Data Engineering Podcast

SEPTEMBER 25, 2022

Mention the podcast to get a free "In Data We Trust World Tour" t-shirt. RudderStack helps you build a customer data platform on your warehouse or data lake. The Ascend Data Automation Cloud provides a unified platform for data ingestion, transformation, orchestration, and observability.

Food

Food MongoDB Scala MySQL

Operational Analytics To Increase Efficiency For Multi-Location Businesses With OpsAnalitica

Data Engineering Podcast

SEPTEMBER 18, 2022

RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control.

Hospitality

Hospitality Food MongoDB Scala

Building Data Pipelines That Run From Source To Analysis And Activation With Hevo Data

Data Engineering Podcast

SEPTEMBER 11, 2022

RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control.

Data Pipeline

Data Pipeline Building MongoDB Scala

Slowly Changing Dimensions (SCD Type 1) with Delta and Databricks

Advancing Analytics: Data Engineering

SEPTEMBER 7, 2021

1 SCD Type 1 in SQL and Python Introduction With the move to cloud based Data Lake platforms there has often been criticism from the more traditional Data Warehousing community. By its definition, a data lake accepts any data, without oversight or governance. From Warehouse to Lakehouse Pt.1

Data Lake

Data Lake Datasets Data Warehouse SQL

Building A Data Governance Bridge Between Cloud And Datacenters For The Enterprise At Privacera

Data Engineering Podcast

MARCH 27, 2022

Acryl Data provides DataHub as an easy to consume SaaS product which has been adopted by several companies. Signup for the SaaS product at dataengineeringpodcast.com/acryl RudderStack helps you build a customer data platform on your warehouse or data lake. Closing Announcements Thank you for listening!

Data Governance

Data Governance Government Cloud Building

Re-Bundling The Data Stack With Data Orchestration And Software Defined Assets Using Dagster

Data Engineering Podcast

JULY 24, 2022

RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control.

MongoDB

MongoDB Scala MySQL Data Lake

Gain Visibility Into Your Entire Machine Learning System Using Data Logging With WhyLogs

Data Engineering Podcast

APRIL 24, 2022

Acryl Data provides DataHub as an easy to consume SaaS product which has been adopted by several companies. Signup for the SaaS product at dataengineeringpodcast.com/acryl RudderStack helps you build a customer data platform on your warehouse or data lake. Can you describe how Whylogs is implemented?

Machine Learning

Machine Learning Systems Data Lake Metadata

Maintain Your Data Engineers' Sanity By Embracing Automation

Data Engineering Podcast

JULY 10, 2022

RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control.

Data Engineering

Data Engineering Data Engineer Engineering MongoDB

Be Confident In Your Data Integration By Quickly Validating Matching Records With data-

Data Engineering Podcast

JULY 3, 2022

The Ascend Data Automation Cloud provides a unified platform for data ingestion, transformation, orchestration, and observability. Ascend users love its declarative pipelines, powerful SDK, elegant UI, and extensible plug-in architecture, as well as its support for Python, SQL, Scala, and Java.

Data Integration

Data Integration MongoDB Scala MySQL

Charting A Path For Streaming Data To Fill Your Data Lake With Hudi

Insights And Advice On Building A Data Lake Platform From Someone Who Learned The Hard Way

Webinars

Trending Sources

Tips to Build a Robust Data Lake Infrastructure

Webinars

Top Data Lake Vendors (Quick Reference Guide)

Add Version Control To Your Data Lake With LakeFS

Data News — Week 24.08

Python for Data Engineering

KDnuggets News, January 18: 7 Best Platforms to Practice SQL • Explainable AI: 10 Python Libraries for Demystifying Your Model’s Decisions

Cloud Native Data Orchestration For Machine Learning And Data Engineering With Flyte

The View Below The Waterline Of Apache Iceberg And How It Fits In Your Data Lakehouse

How to Keep Track of Data Versions Using Versatile Data Kit

Now in Public Preview: Processing Files and Unstructured Data with Snowpark for Python

An Exploration Of The Open Data Lakehouse And Dremio's Contribution To The Ecosystem

How to learn data engineering

Adopting Real-Time Data At Organizations Of Every Size

How to Use ChatGPT ETL Prompts For Your ETL Game

Run Your Applications Worldwide Without Worrying About The Database With Planetscale

Data News — Week 23.09

Data News — Week 23.08

Build A Data Lake For Your Security Logs With Scanner

Clean Up Your Data Using Scalable Entity Resolution And Data Mastering With Zingg

Top 10 Azure Data Engineer Job Opportunities in 2024 [Career Options]

Discover And De-Clutter Your Unstructured Data With Aparavi

Let Your Analysts Build A Data Lakehouse With Cuelake

How to Become an Azure Data Engineer? 2023 Roadmap

How Software Bill of Materials change the dependency game

Joe Reis Flips The Script And Interviews Tobias Macey About The Data Engineering Podcast

Simplify Your Data Architecture With The Presto Distributed SQL Engine

Moving Past ETL and ELT: Understanding the EtLT Approach

Scaling Analysis of Connected Data And Modeling Complex Relationships With The TigerGraph Graph Database

Make Data Lineage A Ubiquitous Part Of Your Work By Simplifying Its Implementation With Alvin

Taking A Look Under The Hood At CreditKarma's Data Platform

Analytics Engineering Without The Friction Of Complex Pipeline Development With Optimus and dbt

Data Science Foundations & Learning Path

Going From Transactional To Analytical And Self-managed To Cloud On One Database With MariaDB

Power Your Real-Time Analytics Without The Headache Using Fivetran's Change Data Capture Integrations

Operational Analytics To Increase Efficiency For Multi-Location Businesses With OpsAnalitica

Building Data Pipelines That Run From Source To Analysis And Activation With Hevo Data

Slowly Changing Dimensions (SCD Type 1) with Delta and Databricks

Building A Data Governance Bridge Between Cloud And Datacenters For The Enterprise At Privacera

Re-Bundling The Data Stack With Data Orchestration And Software Defined Assets Using Dagster

Gain Visibility Into Your Entire Machine Learning System Using Data Logging With WhyLogs

Maintain Your Data Engineers' Sanity By Embracing Automation

Be Confident In Your Data Integration By Quickly Validating Matching Records With data-

Stay Connected