Data Engineering Podcast

AUGUST 3, 2021

Vinoth Chandar helped to create the Hudi project while at Uber to address this challenge. By adding support for small, incremental inserts into large table structures, and building support for arbitrary update and delete operations the Hudi project brings the best of both worlds together. Can you describe how Hudi is architected?

Data Lake 130

Data Lake Data Warehouse Hadoop Architecture 130

Data Engineering Podcast

AUGUST 19, 2019

Summary Managing big data projects at scale is a perennial problem, with a wide variety of solutions that have evolved over the past 20 years. One of the early entrants that predates Hadoop and has since been open sourced is the HPCC (High Performance Computing Cluster) system.

Big Data 100

Big Data Hadoop Data Lake Media 100

Data Engineering Podcast

MARCH 27, 2022

Privacera is an enterprise grade solution for cloud and hybrid data governance built on top of the robust and battle tested Apache Ranger project. Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool.

Data Governance 130

Data Governance Government Cloud Building 130

Data Engineering Podcast

DECEMBER 11, 2022

Summary One of the most critical aspects of software projects is managing its data. Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools. Sign up free at dataengineeringpodcast.com/rudder Build Data Pipelines.

Database 100

Database MySQL Data Lake MongoDB 100

ProjectPro

MAY 26, 2015

In view of the above we have launched Industry Interview Series – where every month we interview someone from the industry to speak on Big Data Hadoop use cases. We had the pleasure to invite Garima Batra, a core platform engineer at MobStac involved in the development of Beaconstac iOS SDK to speak on “How IoT leverages Hadoop?”

Hadoop 52

Hadoop Unstructured Data Big Data Retail 52

Data Engineering Podcast

MAY 11, 2020

In this episode Sijie Guo discusses his motivations for spending so much of his time and energy on contributing to the project and growing the community. This was a great conversation about the strengths of the Pulsar project, how it has evolved in recent years, and some of the innovative ways that it is being used.

Lambda Architecture 100

Lambda Architecture Cloud Kafka Hadoop 100

Data Engineering Podcast

AUGUST 31, 2020

Summary Data warehouse technology has been around for decades and has gone through several generational shifts in that time. I’m working with O’Reilly on a project to collect the 97 things that every data engineer should know, and I need your help. What technologies might someone replace with Firebolt?

Data Warehouse 100

Data Warehouse Cloud Building Data Lake 100

Data Engineering Podcast

JUNE 8, 2021

In this episode Matteo Merli shares the story behind the creation of BookKeeper, the various ways that it is being used today, and the architectural aspects that make it such a strong building block for projects such as Pulsar. If you’ve learned something or tried out a project from the show then tell us about it!

Data Warehouse 100

Data Warehouse Hadoop Metadata Architecture 100

Data Engineering Podcast

OCTOBER 9, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. When is a C360 project the wrong choice?

Food 100

Food MongoDB Scala MySQL 100

Data Engineering Podcast

FEBRUARY 9, 2020

If you’re struggling with unwieldy dimensional models, slow moving projects, or challenges integrating new data sources then listen in on this conversation and then give data vault a try for yourself. Upcoming events include the Software Architecture Conference in NYC, Strata Data in San Jose, and PyCon US in Pittsburgh.

Data Lake 100

Data Lake Data Warehouse Hadoop NoSQL 100

Data Engineering Podcast

DECEMBER 30, 2019

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. Closing Announcements Thank you for listening!

Process 100

Process Building Hadoop Java 100

ProjectPro

JULY 23, 2021

This blog presents some of the most unique and innovative AWS projects from beginner to advanced levels. These AWS project ideas will give you a better idea of various AWS tools and their business applications. Applications of AWS Projects Why should you work on AWS Projects? What are AWS projects? What is AWS?

AWS 52

AWS Project Amazon Web Services Cloud Computing 52

Data Engineering Podcast

JUNE 14, 2021

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. Closing Announcements Thank you for listening!

Machine Learning 100

Machine Learning Database Data Warehouse Hadoop 100

Data Engineering Podcast

OCTOBER 21, 2019

In this episode Dipti Borkar explains how the emerging category of data orchestration tools fills this need, some of the existing projects that fit in this space, and some of the ways that they can work together to simplify projects such as cloud migration and hybrid cloud environments.

Cloud 100

Cloud Data Lake Hadoop Programming Language 100

Data Engineering Podcast

SEPTEMBER 7, 2020

I’m working with O’Reilly on a project to collect the 97 things that every data engineer should know, and I need your help. When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Architecture 100

Architecture Data Architecture SQL Engineering 100

Knowledge Hut

APRIL 26, 2024

Now that the issue of storage of big data has been solved successfully by Hadoop and various other frameworks, the concern has shifted to processing these data. This tool helps businesses measure the probability of such events, such as whether or not purchases made by a customer will be on time.

Data Science 52

Data Science Machine Learning Hadoop Programming Language 52

Data Engineering Podcast

JULY 10, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. Closing Announcements Thank you for listening!

Data Engineering 100

Data Engineering Data Engineer Engineering MongoDB 100

Data Engineering Podcast

NOVEMBER 18, 2018

Summary Modern applications and data platforms aspire to process events and data in real time at scale and with low latency. Can you start by describing what Flink is and how the project got started? How do you handle out of order events in Flink, especially as the delay for a given event increases?

Process 100

Process Scala Google Cloud Kafka 100

Data Engineering Podcast

SEPTEMBER 18, 2019

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. is and how the project and business got started?

Data Lake 100

Data Lake Hadoop BI Kafka 100

ProjectPro

FEBRUARY 21, 2023

Companies seek to hire Spark developers for various tasks, including enhancing programming efficiency, event stream processing, quick, real-time data querying, batch processing of large data sets, etc. Apache Spark developers should have a good understanding of distributed systems and big data technologies.

Scala 52

Scala Programming Language Java Hadoop 52

Data Engineering Podcast

NOVEMBER 18, 2019

Summary With the constant evolution of technology for data management it can seem impossible to make an informed decision about whether to build a data warehouse, or a data lake, or just leave your data wherever it currently rests. If you’ve learned something or tried out a project from the show then tell us about it!

Data Lake 100

Data Lake Scala Data Warehouse Hadoop 100

Data Engineering Podcast

APRIL 26, 2021

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. Closing Announcements Thank you for listening!

Business Intelligence 100

Business Intelligence Data Warehouse Hadoop Data Pipeline 100

Christophe Blefari

DECEMBER 23, 2022

Next year I plan to talk about: Data engineering and analytics engineering career paths State of the data integration—related to another 2023 project 📚 We have too many choices, my framework to take a decision Something you want me to write on? Let's go back to dbt. From the first contact, the local installation, the (web?)

Data 100

Data Hadoop Data Engineering Data Engineer 100

Data Engineering Podcast

MARCH 22, 2021

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. When is CDC the wrong choice?

Data Warehouse 100

Data Warehouse Metadata Data Lake Hadoop 100

Knowledge Hut

MAY 2, 2024

Since then, Apache Spark has seen a very high adoption rate from top-notch technology companies like Google, Facebook, Apple, Netflix etc. Features of Spark Speed : According to Apache, Spark can run applications on Hadoop cluster up to 100 times faster in memory and up to 10 times faster on disk.

Scala 52

Scala Hospitality Healthcare Retail 52

Data Engineering Weekly

MARCH 19, 2023

Contribute to the Rudderstack Transformations Library, Win $1000 RudderStack Transformations lets you customize event data in real time with your own JavaScript or Python code. Hadoop put forward the schema-on-read strategy that leads to the disruption of data modeling techniques as we know until then. Is chatGPT a data product?

Data Engineering 57

Data Engineering Data Engineer Engineering Media 57

Data Engineering Podcast

JANUARY 13, 2019

release, how the use cases for timeseries data have proliferated, and how they are continuing to simplify the task of processing your time oriented events. What has changed in the focus and features of the TimescaleDB project and company? What has changed in the focus and features of the TimescaleDB project and company?

Database 100

Database PostgreSQL SQL MongoDB 100

Data Engineering Podcast

JULY 22, 2019

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Lake 100

Data Lake Hadoop Data Architecture 100

Knowledge Hut

JUNE 16, 2023

Artificial Intelligence is the core of the technology industry and is growing more competitive. AI, the driving force behind computer vision, voice analysis, robotics, and Natural Language Processing (NLP), has tremendously impacted the industry and is projected to do so in the decades to come. AI Engineer Career Opportunities?

Engineering 52

Engineering Deep Learning Programming Language Software Engineer 52

ProjectPro

MARCH 10, 2016

Table of Contents LinkedIn Hadoop and Big Data Analytics The Big Data Ecosystem at LinkedIn LinkedIn Big Data Products 1) People You May Know 2) Skill Endorsements 3) Jobs You May Be Interested In 4) News Feed Updates Wondering how LinkedIn keeps up with your job preferences, your connection suggestions and stories you prefer to read? .”-

Hadoop 40

Hadoop Big Data Data Analytics Big Data Ecosystem 40

Cloudera

SEPTEMBER 24, 2021

Advanced predictive analytics technologies were scaling up, and streaming analytics was allowing on-the-fly or data-in-motion analysis that created more options for the data architect. Suddenly, it was possible to build a data model of the network and create both a historical and predictive view of its behaviour.

Data Architect 91

Data Architect Government NoSQL Big Data 91

Data Engineering Podcast

DECEMBER 16, 2019

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. Closing Announcements Thank you for listening!

Metadata 100

Metadata PostgreSQL Datasets Data Warehouse 100

Big Data Tools

JULY 13, 2022

News A lot of engineering is about learning new things and keeping a finger on the pulse of new technologies. It made me think that the era of on-premises free Hadoop installations had come to an end. InLong 1.2.0 – This is one of the more interesting projects I hadn’t already heard of before preparing this installment.

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Big Data Tools

JULY 13, 2022

News A lot of engineering is about learning new things and keeping a finger on the pulse of new technologies. It made me think that the era of on-premises free Hadoop installations had come to an end. InLong 1.2.0 – This is one of the more interesting projects I hadn’t already heard of before preparing this installment.

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Data Engineering Podcast

SEPTEMBER 16, 2018

In this episode he explains how the project and company got started, how the platform is architected, and how you can start using it today to get a clearer view of how your customers are interacting with your web and mobile applications. What is unique about customer event data from an ingestion and processing perspective?

Google Cloud 100

Google Cloud Hadoop Business Intelligence Consulting 100

ProjectPro

APRIL 2, 2015

Retail industry is rapidly adopting the data centric technology to boost sales. 2) Trends in Retail are changing at a very rapid pace due to enhanced methods of communications, changing technologies and varying consumer tastes making it difficult for retailers to analyze the changing trends without Retail Analytics.

Retail 40

Retail Hadoop Big Data Data Mining 40

Data Engineering Weekly

OCTOBER 1, 2023

Partitioning : how should we partition our table (in Hadoop)? Register to learn how to build more trustworthy and reliable data & AI products with the latest technologies, processes, and strategies shaping our industry (yes, LLMs will be on the table). What is the data behavior? RudderStack just released its Activation API.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Pipeline 52