Data Engineering Podcast

FEBRUARY 19, 2023

Projects like Apache Iceberg provide a viable alternative in the form of data lakehouses that provide the scalability and flexibility of data lakes, combined with the ease of use and performance of data warehouses. What are the most interesting, unexpected, or challenging lessons that you have learned while working on Iceberg/Tabular?

IT 147

IT Data Lake Metadata Data Warehouse 147

Snowflake

DECEMBER 4, 2023

Apache Iceberg’s ecosystem of diverse adopters, contributors and commercial support continues to grow, establishing itself as the industry standard table format for an open data lakehouse architecture. Your other engines that may be writing to the table, such as Apache Spark or Apache Flink, can continue to write, and Snowflake can read.

Building 95

Building Metadata Cloud Storage AWS 95

Lyft Engineering

NOVEMBER 29, 2023

Introduction At Lyft, we have used systems like Apache ClickHouse and Apache Druid for near real-time and sub-second analytics. Druid at Lyft Apache Druid is an in-memory, columnar, distributed, open-source data store designed for sub-second queries on real-time and historical data. This was our main form of ingestion.

Kafka 104

Kafka Data Ingestion Datasets Architecture 104

Confluent

AUGUST 29, 2023

Apache Flink can be used for multiple stream processing use cases. Learn how developers can use Flink to build real-time applications, run analytical workloads or build real-time pipelines.

Process 70

Process Kafka Building 70

Christophe Blefari

APRIL 21, 2023

We announced this week the May Paris Apache Airflow meetup. Fast News ⚡️ Building a Flink self-serve platform on Kubernetes at scale — Instacart engineering team migrated from Flink on EMR to Flink on Kubernetes. When you look at the first edition we came a long way—lmao. CoreDB raises $6.5m

Raw Data 130

Raw Data Data Datasets SQL 130

Data Engineering Weekly

MARCH 24, 2024

As we predicted in the key trends of 2023 about Apache Flink as a clear winner in the stream processing frameworks, we see Confluent offering Flink as a service. The author writes excellent tips on optimizing Apache Hudi LakeHouse infrastructure by adopting indexes, partition pruning, handling incomplete versions, etc.

Data Engineering 62

Data Engineering Data Engineer Engineering Metadata 62

Jesse Anderson

DECEMBER 12, 2022

Doug Cutting took those papers and created Apache Hadoop in 2005. Hadoop was hard to program, and Apache Hive came along in 2010 to add SQL. Apache Pig in 2008 came too, but it didn’t ever see as much adoption. Apache HBase came in 2007, and Apache Cassandra came in 2008. We lacked a scalable pub/sub system.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Marc Lamberti

SEPTEMBER 22, 2023

What are XCOMs in Apache Airflow? In this tutorial, you will learn about XComs in Airflow. If you checked my course “Apache Airflow: The Hands-On Guide”, Aiflow XCom should not sound unfamiliar. If you want to learn more about the differences between JSON/Pickle, click here. It is not Spark or Flink.

MySQL 246

MySQL Data Pipeline Database Python 246

Lyft Engineering

JUNE 28, 2023

In early 2022, Lyft already had a comprehensive Machine Learning Platform called LyftLearn composed of model serving , training , CI/CD, feature serving , and model monitoring systems. Lyft is a real-time marketplace and many teams benefit from enhancing their machine learning models with real-time signals.

Machine Learning 115

Machine Learning Building Metadata Kafka 115

Confluent

SEPTEMBER 26, 2023

Learn how Confluent re-architected Apache Flink into a serverless, cloud-native service on Confluent Cloud for high-performance, efficient stream processing at any scale.

Cloud 74

Cloud Process 74

Confluent

SEPTEMBER 14, 2023

Learn how to use Apache Flink to build a Java pipeline that consumes clickstream data from Apache Kafka.

Kafka 70

Kafka Programming Java Building 70

Cloudera

OCTOBER 4, 2023

What we need is: An openness to support a wide range in streaming ingest sources, including NiFi, Spark Streaming, Flink, as well as APIs for languages like C++, Java, and Python. However, there are some key capabilities that will make real-time analytics a practical and applied reality. Kudu has this covered.

Telecommunication 84

Telecommunication Data Warehouse Java Manufacturing 84

Cloudera

JULY 18, 2022

In part 1 of this blog we discussed how Cloudera DataFlow for the Public Cloud (CDF-PC), the universal data distribution service powered by Apache NiFi, can make it easy to acquire data from wherever it originates and move it efficiently to make it available to other applications in a streaming fashion. Apache Flink.

Process 86

Process Kafka Scala SQL 86

Netflix Tech

JANUARY 13, 2022

These steps launch jobs on clusters running Apache Spark and Presto via Genie. We have now moved to a more systematic approach where unknown errors are fed into a Machine Learning process that performs clustering to propose new regular expressions for commonly occurring errors.

Kafka 95

Kafka Big Data Data Machine Learning 95

Cloudera

SEPTEMBER 26, 2023

Learn more about how you can benefit from a well-supported data management platform and ecosystem of products, services and support by visiting the IBM and Cloudera partnership page.

Kafka 90

Kafka Technology IT Government 90

Confluent

OCTOBER 12, 2023

Learn about the key capabilities of a data streaming platform and what factors to consider when choosing a stream processing engine like Apache Flink® to fuel use cases with real-time data.

Process 62

Process Engineering Data 62

Cloudera

MARCH 5, 2024

Cloudera is now the only provider to offer an open data lakehouse with Apache Iceberg for cloud and on-premises. This approach brings new compute engines into the fold, adding Spark, Flink, Impala, and NiFi, enabling concurrent access and processing of datasets within Iceberg. Available for cloud and now also for the data center.

Data Lake 86

Data Lake Data Storage Government Kafka 86

Cloudera

JUNE 7, 2021

We think that this is a good validation of our data-in-motion philosophy that a streaming architecture is made up of needs across data ingestion , messaging and analytics and in our case, this is powered by Apache NiFi, Apache Kafka and Apache Flink. Learn more about CDF at cloudera.com/cdf.

Kafka 101

Kafka Data Ingestion Architecture Cloud 101

Data Engineering Weekly

APRIL 14, 2024

However, TIL about Apache Paimon, a lake format that enables building a real-time lakehouse architecture with Flink and Spark for both streaming and batch operations. The blog post brings in-depth insights into how Apache Hudi works on the right side, read side & underlying storage formats.

Data Engineering 62

Data Engineering Data Engineer Engineering Business Intelligence 62

Pinterest Engineering

SEPTEMBER 29, 2023

Sanchay Javeria | Software Engineer, Ads Data Infrastructure To support metrics reporting for ads from external advertisers and real-time ad budget calculations at Pinterest, we run streaming pipelines using Apache Flink. Debugging application code Our Flink application consists of several thousand lines of code.

Utilities 93

Utilities Coding Kafka Engineering 93

Cloudera

JUNE 7, 2021

It enabled users to easily write, run and manage real-time SQL queries on streams from Apache Kafka with an exceptionally smooth user experience. . Flink SQL DDL and Catalog support. Flink SQL DDL Support. Take the next step and learn more about Cloudera Streaming Analytics. Feature Highlights. Improved Table Management.

Kafka 101

Kafka SQL Accessible Accessibility 101

Cloudera

OCTOBER 13, 2021

Imagine if you had to explain what machine learning is and how to use it. Cloudera produced a series of ebooks — Production Machine Learning For Dummies , Apache NiFi For Dummies , and Apache Flink For Dummies (coming soon) — to help simplify even the most complex tech topics. There’s no need to panic.

Machine Learning 74

Machine Learning Data Ingestion Algorithm Technology 74

Data Engineering Podcast

JUNE 15, 2020

Summary Machine learning is a process driven by iteration and experimentation which requires fast and easy access to relevant features of the data being processed. Can you describe what a machine learning feature is? __init__ to learn about the Python language, its community, and the innovative ways it is being used.

Machine Learning 100

Machine Learning Google Cloud Kafka Data Engineering 100

Data Engineering Weekly

DECEMBER 3, 2023

Visit rudderstack.com to learn more. link] Netflix: Incremental Processing using Netflix Maestro and Apache Iceberg Netflix writes about its incremental processing design with its orchestration engine Maestro on top of Iceberg. Github writes an excellent blog to capture the current state of the LLM integration architecture.

Data Engineering 70

Data Engineering Data Engineer Engineering Bytes 70

Christophe Blefari

OCTOBER 20, 2023

Read my dbt multi-project guide 📺 On the content side I'll also present next week the Fancy Data Stack project at the Data Engineering And Machine Learning Summit 2023 organised by Seattle Data Guy. dbt related stuff To dbt or not to dbt — A few lessons learned while implementing dbt at Intercom.

Generalist 130

Generalist Entertainment NoSQL Datasets 130

Data Engineering Podcast

APRIL 19, 2020

And for your machine learning workloads, they’ve got dedicated CPU and GPU instances. What is the tipping point for when SQL is insufficient for a given task and a user might want to leverage Flink? __init__ to learn about the Python language, its community, and the innovative ways it is being used.

Building 100

Building PostgreSQL MongoDB SQL 100

Christophe Blefari

DECEMBER 31, 2023

At the same time they released Mixtral 8x7B, the first open-source model of this calibre under an Apache Licence. People underestimate how impactful Scikit-learn continues to be — The year is coming to an end and LinkedIn is playing at the 2024 predictions game. And the weight are open-source as well. It gives so many ideas.

Data 100

Data Cloud Storage Datasets Python 100

Data Engineering Weekly

AUGUST 6, 2023

Learn more and sign up to join the Great Data Debate on August 16 → DuckDB: DuckDB ADBC - Zero-Copy data transfer via Arrow Database Connectivity DuchDB writes about the support for Arrow Database Connectivity (ADBC) protocol support highlighting the performance benefit from ODBC. Look no further than the Gartner latest report.

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Data Engineering Weekly

APRIL 28, 2023

Here are the top 5 key learnings from the report. Spark attempt to solve this by creating a unified RDD model for streaming and batch; Flink introduces the table format to bridge the gap in batch processing.

Consulting 52

Consulting Kafka Lambda Architecture Engineering 52

Data Engineering Weekly

NOVEMBER 27, 2022

Machine Learning powers every application feature listed above. Learn how JetBlue’s data engineering and data science teams leverage Monte Carlo and Snowflake together to accelerate data analysis and drive business value. The blog captures the history of Flink SQL, its current state, and the challenges ahead of it.

Data Engineering 52

Data Engineering Data Engineer Engineering SQL 52

Data Engineering Podcast

AUGUST 3, 2021

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management You listen to this show to learn about all of the latest tools, patterns, and practices that power data engineering projects across every domain. If you’ve learned something or tried out a project from the show then tell us about it!

Data Lake 130

Data Lake Data Warehouse Hadoop Architecture 130

Cloudera

JUNE 28, 2022

We discussed how Cloudera Stream Processing (CSP) with Apache Kafka and Apache Flink could be used to process this data in real time and at scale. For each transaction, NiFi makes a call to a production model in Cloudera Machine Learning (CML) to score the fraud potential of the transaction.

Process 81

Process Kafka SQL Machine Learning 81

Big Data Tools

NOVEMBER 9, 2022

News A lot of engineering is about learning new things and keeping a finger on the pulse of new technologies. Apache Doris 1.1.3 – Here’s another interesting database for you. We aren’t aware of many MPP databases, and none of them are under the motley umbrella of the Apache Software Foundation. For example, the current 1.1.3

Data Engineering 52

Data Engineering Data Engineer Engineering Big Data Tools 52

Big Data Tools

NOVEMBER 9, 2022

News A lot of engineering is about learning new things and keeping a finger on the pulse of new technologies. Apache Doris 1.1.3 – Here’s another interesting database for you. We aren’t aware of many MPP databases, and none of them are under the motley umbrella of the Apache Software Foundation. For example, the current 1.1.3

Data Engineering 52

Data Engineering Data Engineer Engineering Big Data Tools 52

Big Data Tools

NOVEMBER 8, 2021

News A lot of engineering is about learning new things and keeping a finger on the pulse of new technologies. Apache Spark® has been released and there are a load of changes, including ANSI SQL support, Pandas API layer over PySpark, and lots and lots of other things. Apache Ranger 2.2.0 Flink isn’t aware of this – yet.

Data Engineering 52

Data Engineering Data Engineer Engineering Big Data Tools 52

Big Data Tools

NOVEMBER 8, 2021

News A lot of engineering is about learning new things and keeping a finger on the pulse of new technologies. Apache Spark® has been released and there are a load of changes, including ANSI SQL support, Pandas API layer over PySpark, and lots and lots of other things. Apache Ranger 2.2.0 Flink isn’t aware of this – yet.

Data Engineering 52

Data Engineering Data Engineer Engineering Big Data Tools 52

Big Data Tools

MAY 19, 2022

News A lot of engineering is about learning new things and keeping a finger on the pulse of new technologies. Apache Hudi 1.11.0 – This release of the well-known data lake has added many interesting changes. Kyuubi 1.5.1 – Kyuubi is a JDBC server built over Apache Spark, but as of version 1.5.0,

Data Engineering 52

Data Engineering Data Engineer Engineering Big Data Tools 52