Data Engineering Podcast

AUGUST 21, 2022

In this episode Shruti Bhat gives her view on the state of the ecosystem for real-time data and the work that she and her team at Rockset is doing to make it easier for engineers to build those experiences. All thanks to 50+ quality checks, extensive column-level lineage, and 20+ connectors across the Data Stack.

Lambda Architecture 100

Lambda Architecture MongoDB Scala MySQL 100

Data Engineering Podcast

SEPTEMBER 11, 2022

In this episode he shares his journey from building a consumer product to launching a data pipeline service and how his frustrations as a product owner have informed his work at Hevo Data. All thanks to 50+ quality checks, extensive column-level lineage, and 20+ connectors across the Data Stack. In fact, while only 3.5%

Data Pipeline 130

Data Pipeline Building MongoDB Scala 130

Data Engineering Podcast

OCTOBER 23, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Database 100

Database MySQL Cloud MongoDB 100

Data Engineering Podcast

NOVEMBER 6, 2022

Summary Despite the best efforts of data engineers, data is as messy as the real world. Entity resolution and fuzzy matching are powerful utilities for cleaning up data from disconnected sources, but it has typically required custom development and training machine learning models. In fact, while only 3.5%

MongoDB 130

MongoDB Scala MySQL Data Lake 130

Data Engineering Podcast

JUNE 12, 2022

Summary Unstructured data takes many forms in an organization. From a data engineering perspective that often means things like JSON files, audio or video recordings, images, etc. Sign up free… or just get the free t-shirt for being a listener of the Data Engineering Podcast at dataengineeringpodcast.com/rudder.

Unstructured Data 130

Unstructured Data MongoDB Scala MySQL 130

Rockset

SEPTEMBER 3, 2020

MongoDB’s Advantages & Disadvantages MongoDB has comprehensive aggregation capabilities. You can run many analytic queries on MongoDB without exporting your data to a third-party tool. In this situation, the MongoDB cluster doesn’t have to keep up with the read requests. This is never a good thing.

MongoDB 40

MongoDB PostgreSQL SQL Database 40

Data Engineering Podcast

JUNE 19, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Metadata 130

Metadata MongoDB Scala MySQL 130

Knowledge Hut

APRIL 25, 2023

The tremendous growth in data generation, then the rise in data engineer jobs - there’s no arguing the fact that the big data industry is at its best pace and you, as an aspiring data engineer, have a lot to learn and make out of it - including some tools! What are Data Engineering Tools?

Data Engineering 52

Data Engineering Data Engineer Engineering Google Cloud 52

Data Engineering Podcast

NOVEMBER 13, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

MongoDB 100

MongoDB Scala Google Cloud MySQL 100

Data Engineering Podcast

AUGUST 13, 2022

In this episode Shinji Kim discusses the challenges of data discovery and how to collect and preserve additional context about each piece of information so that you can find what you need when you don’t even know what you’re looking for yet. Sifflet also offers a 2-week free trial. In fact, while only 3.5%

Metadata 100

Metadata MongoDB Scala MySQL 100

Data Engineering Podcast

OCTOBER 16, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Data Lake 100

Data Lake Food MongoDB Scala 100

Data Engineering Podcast

JULY 31, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Data Analysis 100

Data Analysis MongoDB Scala Algorithm 100

Data Engineering Podcast

AUGUST 28, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Building 100

Building MongoDB Scala MySQL 100

Data Engineering Podcast

SEPTEMBER 25, 2022

In this episode Mark Van de Wiel explains how they integrated CDC functionality into their existing product, discusses the nuances of different approaches to change data capture from various sources. Sign up free… or just get the free t-shirt for being a listener of the Data Engineering Podcast at dataengineeringpodcast.com/rudder.

Food 100

Food MongoDB Scala MySQL 100

Data Engineering Podcast

SEPTEMBER 18, 2022

He discusses the challenges of making data collection purposeful and efficient without distracting employees from their primary duties and how business owners can use the provided analytics to support their staff in their duties. Data teams are increasingly under pressure to deliver. In fact, while only 3.5%

Hospitality 100

Hospitality Food MongoDB Scala 100

Data Engineering Podcast

OCTOBER 2, 2022

Alvin is a platform that aims to provide a low effort solution for data lineage capabilities focused on simplifying the work of data engineers. Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool.

IT 100

IT Food PostgreSQL MongoDB 100

Data Engineering Podcast

JULY 3, 2022

Summary The perennial challenge of data engineers is ensuring that information is integrated reliably. In order to quickly identify if and how two data systems are out of sync Gleb Mezhanskiy and Simon Eskildsen partnered to create the open source data-diff utility. In fact, while only 3.5%

Data Integration 100

Data Integration MongoDB Scala MySQL 100

Data Engineering Podcast

JULY 24, 2022

In this episode Nick Schrock discusses the importance of orchestration and a central location for managing data systems, the road to Dagster’s 1.0 Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool.

MongoDB 100

MongoDB Scala MySQL Data Lake 100

Data Engineering Podcast

JUNE 5, 2022

In this episode Sean Falconer explains the idea of a data privacy vault and how this new architectural element can drastically reduce the potential for making a mistake with how you manage regulated or personally identifiable information. If you’re a data engineering podcast listener, you get credits worth $5,000 when you become a customer.

Data Security 100

Data Security Metadata MongoDB Scala 100

Data Engineering Podcast

AUGUST 13, 2022

Summary Data engineers have typically left the process of data labeling to data scientists or other roles because of its nature as a manual and process heavy undertaking, focusing instead on building automation and repeatable systems. Sifflet also offers a 2-week free trial. In fact, while only 3.5%

Machine Learning 100

Machine Learning Pipeline-centric Database-centric MongoDB 100

Data Engineering Podcast

AUGUST 6, 2022

In this episode Frank Liu explains how the open source Milvus vector database is implemented to speed up machine learning development cycles, how to think about proper storage and scaling of these vectors, and how data engineering and machine learning teams can collaborate on the creation and maintenance of these data sets.

Machine Learning 100

Machine Learning Database MySQL PostgreSQL 100

Data Engineering Podcast

OCTOBER 9, 2022

In this episode Purvi Shah, the VP of Enterprise Big Data Platforms at American Express, explains how they have invested in the cloud to power this visibility and the complex suite of integrations they have built and maintained across legacy and modern systems to make it possible. Data teams are increasingly under pressure to deliver.

Food 100

Food MongoDB Scala MySQL 100

Data Engineering Podcast

JUNE 26, 2022

Summary The most complicated part of data engineering is the effort involved in making the raw data fit into the narrative of the business. Your newly mimicked datasets are safe to share with developers, QA, data scientists—heck, even distributed teams around the world. In fact, while only 3.5%

Data Management 100

Data Management Management MongoDB Scala 100

Data Engineering Podcast

SEPTEMBER 4, 2022

In this episode Gopal Erinjippurath discusses the data engineering challenges of building and serving those data sets, and how they are distilling complex climate information into consumable facts so you don’t have to be an expert to understand it. Sifflet also offers a 2-week free trial. In fact, while only 3.5%

MongoDB 100

MongoDB Scala MySQL Data Ingestion 100

ProjectPro

AUGUST 24, 2021

Nevertheless, that is not the only job in the data world. Data professionals who work with raw data like data engineers, data analysts, machine learning scientists , and machine learning engineers also play a crucial role in any data science project.

Data Engineering 52

Data Engineering Data Engineer Coding Project 52

Rockset

JANUARY 9, 2023

Technical Challenges Our original data infrastructure was built around an on-premises MongoDB database that ingested and stored all user transaction data. To scale, we thought about creating additional MongoDB slaves, but decided it would be throwing money at a problem without solving it. It took just one week.

MySQL 52

MySQL MongoDB Recruitment Data Ingestion 52

ProjectPro

JANUARY 19, 2022

Data Engineering is gradually becoming a popular career option for young enthusiasts. Explore this page further and learn everything about data engineers to find the answer. We will cover it all, from its definition, skills, responsibilities to the significance of data engineer in an institution.

Data Engineering 52

Data Engineering Data Engineer Engineering Amazon Web Services 52

Rockset

NOVEMBER 23, 2021

With clickstream or sensor data, the volume can be incredibly high — many terabytes of data per day — as well as incredibly unpredictable, scaling up and down rapidly. Change data capture streams. It is now possible to continuously capture changes as they happen in your operational database like MongoDB or Amazon DynamoDB.

MongoDB 59

MongoDB Data Ingestion SQL PostgreSQL 59

Rockset

MARCH 15, 2021

You want this pipeline to take as little time as possible, because stale data doesn’t provide any value in a real-time analytics system. While there’s typically some amount of data engineering required here, there are ways to minimize it. On the other hand, creating indexes during data ingestion can be done inefficiently.

MongoDB 40

MongoDB Data Ingestion Analytics Application Kafka 40

AltexSoft

JUNE 26, 2023

Read our article on Hotel Data Management to have a full picture of what information can be collected to boost revenue and customer satisfaction in hospitality. While all three are about data acquisition, they have distinct differences. For this task, you need a dedicated specialist — a data engineer or ETL developer.

Data Collection 52

Data Collection Machine Learning Unstructured Data Non-relational Database 52

AltexSoft

MAY 12, 2023

A loose schema allows for some data structure flexibility while maintaining a general organization. Semi-structured data is typically stored in NoSQL databases, such as MongoDB, Cassandra, and Couchbase, following hierarchical or graph data models. MongoDB, Cassandra), and big data processing frameworks (e.g.,

Unstructured Data 59

Unstructured Data NoSQL Hadoop Data Lake 59

Rockset

MAY 17, 2022

Their query languages, whether SQL-like variants such as CQL (Cassandra) and Druid SQL or wholly custom languages such as MQL (MongoDB), poorly support joins and other complex query commands that are standard to SQL , if they support them at all. Instant scaleup of data writes or queries to handle bursts of data.

SQL 52

SQL NoSQL Hadoop MongoDB 52

Rockset

MAY 12, 2022

Real-time analytics now drive their operations and bottom line, whether it is through a customer recommendation engine, an automated personalization system or an internal business observability platform. There’s no time to buffer data for leisurely ingestion. One layer processes batches of historic data.

Analytics Application 52

Analytics Application Lambda Architecture Hadoop Electronics 52

Rockset

AUGUST 30, 2021

A Quick Primer on Indexing in Rockset Rockset allows users to connect real-time data sources — data streams (Kafka, Kinesis), OLTP databases (DynamoDB, MongoDB, MySQL, PostgreSQL) and also data lakes (S3, GCS) — using built-in connectors. In example above, these base aggregate metrics are count(*) and sum(error_flag).

SQL 52

SQL Kafka MongoDB MySQL 52

AltexSoft

JULY 29, 2022

What happens, when a data scientist, BI developer , or data engineer feeds a huge file to Hadoop? Under the hood, the framework divides a chunk of Big Data into smaller, digestible parts and allocates them across multiple commodity machines to be processed in parallel. How data engineering works under the hood.

Hadoop 59

Hadoop Big Data Google Cloud NoSQL 59

AltexSoft

MAY 14, 2021

On top of that, new technologies are constantly being developed to store and process Big Data allowing data engineers to discover more efficient ways to integrate and use that data. You may also want to watch our video about data engineering: A short video explaining how data engineering works.

Big Data 78

Big Data Data Analytics IT NoSQL 78

Rockset

DECEMBER 9, 2021

Lifting-and-shifting their big data environment into the cloud only made things more complex. The modern data stack introduced a set of cloud-native data solutions such as Fivetran for data ingestion, Snowflake, Redshift or BigQuery for data warehousing , and Looker or Mode for data visualization.

Transportation 52

Transportation BI SQL Data Warehouse 52