Data Engineering Podcast

NOVEMBER 13, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

MongoDB 100

MongoDB Scala MySQL Google Cloud 100

Data Engineering Podcast

AUGUST 13, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Metadata 100

Metadata MongoDB Scala MySQL 100

Data Engineering Podcast

OCTOBER 9, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Food 100

Food MongoDB Scala MySQL 100

Data Engineering Podcast

SEPTEMBER 25, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Food 100

Food MongoDB Scala MySQL 100

Data Engineering Podcast

JULY 24, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

MongoDB 100

MongoDB Scala MySQL Data Lake 100

Data Engineering Podcast

SEPTEMBER 18, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Hospitality 100

Hospitality Food MongoDB Scala 100

Data Engineering Podcast

JULY 3, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Integration 100

Data Integration MongoDB Scala MySQL 100

Data Engineering Podcast

AUGUST 28, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Building 100

Building MongoDB Scala MySQL 100

Data Engineering Podcast

JULY 10, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Engineering 100

Data Engineering Data Engineer Engineering MongoDB 100

Data Engineering Podcast

SEPTEMBER 11, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Pipeline 130

Data Pipeline Building MongoDB Scala 130

Data Engineering Podcast

JUNE 5, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Security 100

Data Security Metadata MongoDB Scala 100

Data Engineering Podcast

AUGUST 13, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Machine Learning 100

Machine Learning Pipeline-centric Database-centric MongoDB 100

Data Engineering Podcast

AUGUST 6, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. What are the goals of the project?

Machine Learning 100

Machine Learning Database MySQL PostgreSQL 100

Data Engineering Podcast

OCTOBER 2, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

IT 100

IT Food PostgreSQL MongoDB 100

Data Engineering Podcast

JUNE 26, 2022

In this episode Malcolm Hawker shares his years of experience working in this domain to explore the combination of technical and social skills that are necessary to make an MDM project successful both at the outset and over the long term. If you’ve learned something or tried out a project from the show then tell us about it!

Data Management 100

Data Management Management MongoDB Scala 100

Data Engineering Podcast

SEPTEMBER 4, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

MongoDB 100

MongoDB Scala MySQL Data Ingestion 100

Knowledge Hut

MAY 2, 2024

As per Apache, “ Apache Spark is a unified analytics engine for large-scale data processing ” Spark is a cluster computing framework, somewhat similar to MapReduce but has a lot more capabilities, features, speed and provides APIs for developers in many languages like Scala, Python, Java and R.

Scala 52

Scala Hospitality Healthcare Retail 52

Knowledge Hut

JULY 27, 2023

They are skilled in working with tools like MapReduce, Hive, and HBase to manage and process huge datasets, and they are proficient in programming languages like Java and Python. Using the Hadoop framework, Hadoop developers create scalable, fault-tolerant Big Data applications. What do they do? How to Improve Hadoop Developer Salary?

Hadoop 52

Hadoop Programming Language Banking Scala 52

Snowflake

AUGUST 16, 2023

Data comes in a continuous manner, and often a separate architecture is required to handle streaming data. What remains challenging is how streaming data is brought together with batch data. That’s why we built Snowpipe Streaming, now generally available to handle row-set data ingestion. Learn more here.

Scala 52

Scala Transportation Kafka Data Lake 52

Knowledge Hut

OCTOBER 3, 2023

Recommended Skills When it comes to hard skills, Data warehousing solutions, ETL tools, NoSQL and SQL databases, Machine Learning, Python, Scala, and Java programming, Data APIs, understanding the fundamentals of distributed systems, and a solid understanding of data structures and algorithms are all skills that Azure data engineers should possess.

Data Engineering 52

Data Engineering Data Engineer Engineering Cloud Computing 52

Knowledge Hut

APRIL 25, 2023

Here are some essential skills for data engineers when working with data engineering tools. Strong programming skills: Data engineers should have a good grasp of programming languages like Python, Java, or Scala, which are commonly used in data engineering.

Data Engineering 52

Data Engineering Data Engineer Engineering Google Cloud 52

ProjectPro

FEBRUARY 21, 2023

For instance, with a projected average annual salary of $171,749, the GCP Professional Data Engineer certification was the top-paying one on this list in 2021. Boost Your Skills and Knowledge You can keep up with the newest technology and best practices in the industry by earning data engineering certifications.

Certification 52

Certification Data Engineering Data Engineer Engineering 52

Cloudera

JUNE 28, 2022

In part two we will explore how we can run real-time streaming analytics using Apache Flink, and we will use Cloudera SQL Stream Builder GUI to easily create streaming jobs using only SQL language (no Java/Scala coding required). The use case. Fraud detection is a great example of a time-critical use case for us to explore.

Process 85

Process Kafka SQL Machine Learning 85

Knowledge Hut

APRIL 25, 2023

An end-to-end Data Science pipeline starts from business discussion to delivering the product to the customers. One of the key components of this pipeline is Data ingestion. It helps in integrating data from multiple sources such as IoT, SaaS, on-premises, etc., What is Data Ingestion?

Data Ingestion 52

Data Ingestion Lambda Architecture Raw Data Kafka 52

Data Engineering Podcast

NOVEMBER 20, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Lake 130

Data Lake Data Ingestion MongoDB Scala 130

Data Engineering Podcast

JUNE 19, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Metadata 130

Metadata MongoDB Scala MySQL 130

Data Engineering Podcast

JULY 31, 2022

The Arkouda project is a Python interface built on top of the Chapel compiler to bring back those interactive speeds for exploratory analysis on horizontally scalable compute that parallelizes operations on large volumes of data. What are the main goals of the project? Can you describe what Arkouda is and the story behind it?

Data Analysis 100

Data Analysis MongoDB Scala Algorithm 100

Data Engineering Podcast

MAY 29, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Architecture 100

Architecture Database Data Architecture PostgreSQL 100

ProjectPro

JANUARY 25, 2022

Features of PySpark The PySpark Architecture Popular PySpark Libraries PySpark Projects to Practice in 2022 Wrapping Up FAQs Is PySpark easy to learn? Finally, you'll find a list of PySpark projects to help you gain hands-on experience and land an ideal job in Data Science or Big Data. Why use PySpark? What is PySpark?

Big Data 52

Big Data Data Process Process Kafka 52

ProjectPro

DECEMBER 28, 2021

Whether you're working with semi-structured, structured, streaming, or machine learning data, Apache Spark is a fast, easy-to-use framework that allows you to solve various complex data issues. Many traditional stream processing systems use a continuous operator model to process data. Table of Contents What is Spark streaming?

Architecture 52

Architecture Kafka Java Scala 52

Rock the JVM

JANUARY 21, 2023

The code samples are written in Kotlin, but the implementation should be easy to port in Java or Scala. Our event data is stored in CSV files that we want to ingest into Kafka, and since it is not real-time data ingestion, we don’t really care about latency here, but having a good throughput so that we can ingest them fast.

Kafka 65

Kafka Java Scala Coding 65

AltexSoft

JULY 29, 2022

Apache Hadoop is an open-source Java-based framework that relies on parallel processing and distributed storage for analyzing massive datasets. Developed in 2006 by Doug Cutting and Mike Cafarella to run the web crawler Apache Nutch, it has become a standard for Big Data analytics. What is Hadoop? Scalability.

Hadoop 59

Hadoop Big Data Google Cloud NoSQL 59

ProjectPro

NOVEMBER 22, 2021

PySpark runs a completely compatible Python instance on the Spark driver (where the task was launched) while maintaining access to the Scala-based Spark cluster access. Although Spark was originally created in Scala, the Spark Community has published a new tool called PySpark, which allows Python to be used with Spark.

Hadoop 52

Hadoop Python Datasets Metadata 52

Snowflake

MARCH 30, 2023

3EJHjvm Once a business need is defined and a minimal viable product ( MVP ) is scoped, the data management phase begins with: Data ingestion: Data is acquired, cleansed, and curated before it is transformed. Feature engineering: Data is transformed to support ML model training. ML workflow, ubr.to/3EJHjvm

Engineering 87

Engineering Raw Data Data Science Scala 87

ProjectPro

JANUARY 31, 2023

There are three steps involved in the deployment of a big data model: Data Ingestion: This is the first step in deploying a big data model - Data ingestion, i.e., extracting data from multiple data sources. The end of a data block points to the location of the next chunk of data blocks.

Big Data 52

Big Data Hadoop AWS Relational Database 52

AltexSoft

MARCH 30, 2023

The authors aimed to speed up innovations by eliminating data silos, enabling companies to run machine learning on all types of data, and simplifying collaboration across all parties involved in AI projects. Besides that, it’s fully compatible with various data ingestion and ETL tools.

Scala 64

Scala Data Lake BI Google Cloud 64