Data Engineering Podcast

JULY 17, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Data Engineering 100

Data Engineering Data Engineer Engineering MongoDB 100

Data Engineering Podcast

SEPTEMBER 25, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Food 100

Food MongoDB Scala MySQL 100

Data Engineering Podcast

OCTOBER 16, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Data Lake 100

Data Lake Food MongoDB Scala 100

Data Engineering Podcast

OCTOBER 9, 2022

In this episode Purvi Shah, the VP of Enterprise Big Data Platforms at American Express, explains how they have invested in the cloud to power this visibility and the complex suite of integrations they have built and maintained across legacy and modern systems to make it possible. In fact, while only 3.5%

Food 100

Food MongoDB Scala MySQL 100

Data Engineering Podcast

SEPTEMBER 18, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Hospitality 100

Hospitality Food MongoDB Scala 100

Data Engineering Podcast

JULY 10, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Data Engineering 100

Data Engineering Data Engineer Engineering MongoDB 100

Data Engineering Podcast

JULY 3, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Data Integration 100

Data Integration MongoDB Scala MySQL 100

Data Engineering Podcast

AUGUST 6, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Machine Learning 100

Machine Learning Database MySQL PostgreSQL 100

Data Engineering Podcast

AUGUST 13, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Metadata 100

Metadata MongoDB Scala MySQL 100

Data Engineering Podcast

OCTOBER 30, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Engineering 100

Engineering MongoDB Scala MySQL 100

Data Engineering Podcast

OCTOBER 2, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

IT 100

IT Food PostgreSQL MongoDB 100

Data Engineering Podcast

AUGUST 28, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Building 100

Building MongoDB Scala MySQL 100

Data Engineering Podcast

JUNE 26, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Data Management 100

Data Management Management MongoDB Scala 100

Data Engineering Podcast

JUNE 5, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Data Security 100

Data Security Metadata MongoDB Scala 100

Data Engineering Podcast

AUGUST 13, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Machine Learning 100

Machine Learning Pipeline-centric Database-centric MongoDB 100

Data Engineering Podcast

SEPTEMBER 4, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

MongoDB 100

MongoDB Scala MySQL Data Ingestion 100

Knowledge Hut

APRIL 25, 2023

Here, we'll take a look at the top data engineer tools in 2023 that are essential for data professionals to succeed in their roles. These tools include both open-source and commercial options, as well as offerings from major cloud providers like AWS, Azure, and Google Cloud. What are Data Engineering Tools?

Data Engineering 52

Data Engineering Data Engineer Engineering Google Cloud 52

Striim

NOVEMBER 13, 2023

Introduction In the fast-evolving world of data integration, Striim’s collaboration with Snowflake stands as a beacon of innovation and efficiency. Snowpipe Streaming: Unleashing Real-Time Data Integration and AI Snowpipe Streaming, when teamed up with Striim, is kind of like a superhero for real-time data needs.

Data Ingestion 59

Data Ingestion Utilities Data Integration Data 59

Rockset

AUGUST 30, 2021

It becomes prohibitively complex and expensive to use a data warehouse to serve real-time analytics. Rockset: Real-time Analytics Built for the Cloud Rockset is doing for real-time analytics what Snowflake did for batch. In example above, these base aggregate metrics are count(*) and sum(error_flag).

SQL 52

SQL Kafka MongoDB MySQL 52

Rockset

MAY 17, 2022

Their query languages, whether SQL-like variants such as CQL (Cassandra) and Druid SQL or wholly custom languages such as MQL (MongoDB), poorly support joins and other complex query commands that are standard to SQL , if they support them at all. This is intentionally not their forte. Learn more at rockset.com.

SQL 52

SQL NoSQL Hadoop MongoDB 52

Rockset

MAY 12, 2022

Lambda systems try to accommodate the needs of both big data-focused data scientists as well as streaming-focused developers by separating data ingestion into two layers. One layer processes batches of historic data. It provides instant views of the real-time data. No need to overprovision in advance.

Analytics Application 52

Analytics Application Lambda Architecture Hadoop Electronics 52

ProjectPro

JANUARY 19, 2022

Independently create data-driven solutions that are accurate and informative. Interact with the data scientists team and assist them in providing suitable datasets for analysis. Leverage various big data engineering tools and cloud service providing platforms to create data extractions and storage pipelines.

Data Engineering 52

Data Engineering Data Engineer Engineering Amazon Web Services 52

ProjectPro

DECEMBER 7, 2021

Consequently, data engineers implement checkpoints so that no event is missed or processed twice. It not only consumes more memory but also slackens data transfer. Modern cloud-based data pipelines are agile and elastic to automatically scale compute and storage resources. ADF does not store any data on its own.

Data Pipeline 52

Data Pipeline Architecture Kafka AWS 52

Rockset

OCTOBER 11, 2022

Introduction Managing streaming data from a source system, like PostgreSQL, MongoDB or DynamoDB, into a downstream system for real-time analytics is a challenge for many teams. Elasticsearch was designed for log analytics where data is not frequently changing, posing additional challenges when dealing with transactional data.

Data Ingestion 52

Data Ingestion Kafka Relational Database PostgreSQL 52

Data Engineering Podcast

NOVEMBER 20, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Data Lake 130

Data Lake Data Ingestion MongoDB Scala 130

Rockset

NOVEMBER 12, 2020

Offloading analytics from MongoDB establishes clear isolation between write-intensive and read-intensive operations. In most scenarios, MongoDB can be used as the primary data storage for write-only operations and as support for quick data ingestion. Monstache is also available as a sync daemon and a container.

MongoDB 40

MongoDB NoSQL Data Pipeline Data Storage 40

Data Engineering Podcast

JUNE 19, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Metadata 130

Metadata MongoDB Scala MySQL 130

Rockset

JANUARY 9, 2023

Modern Snack-Sized Sales Training At ConveYour , we provide automated sales training via the cloud. Technical Challenges Our original data infrastructure was built around an on-premises MongoDB database that ingested and stored all user transaction data. First is its speed at data ingestion.

MySQL 52

MySQL MongoDB Recruitment Data Ingestion 52

Data Engineering Podcast

JULY 31, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. In fact, while only 3.5%

Data Analysis 100

Data Analysis MongoDB Scala Algorithm 100

DareData

JANUARY 30, 2023

Our goal is to help data scientists better manage their models deployments or work more effectively with their data engineering counterparts, ensuring their models are deployed and maintained in a robust and reliable way. AWS Glue: A fully managed data orchestrator service offered by Amazon Web Services (AWS).

Data Engineering 52

Data Engineering Data Engineer NoSQL Engineering 52

Rockset

MARCH 15, 2021

While there’s typically some amount of data engineering required here, there are ways to minimize it. For example, instead of denormalizing the data, you could use a query engine that supports joins. This will avoid unnecessary processing during data ingestion and reduce the storage bloat due to redundant data.

MongoDB 40

MongoDB Data Ingestion Analytics Application Kafka 40

ProjectPro

AUGUST 24, 2021

Data Engineering Project for Beginners If you are a newbie in data engineering and are interested in exploring real-world data engineering projects, check out the list of data engineering project examples below. This big data project discusses IoT architecture with a sample use case.

Data Engineering 52

Data Engineering Data Engineer Coding Project 52

AltexSoft

JUNE 26, 2023

Read our article on Hotel Data Management to have a full picture of what information can be collected to boost revenue and customer satisfaction in hospitality. While all three are about data acquisition, they have distinct differences. They can be accumulated in NoSQL databases like MongoDB or Cassandra.

Data Collection 52

Data Collection Machine Learning Unstructured Data Non-relational Database 52

Rockset

NOVEMBER 28, 2022

It works with existing streaming systems like Apache Kafka, Amazon Kinesis, and Azure Events Hubs, making it easier than ever to build a real-time data pipeline. using Amazon’s Database Migration Service (DMS), we now continuously replicate data from Aurora into Rockset, which does all of the data processing, aggregations and calculations.”

Data Warehouse 52

Data Warehouse PostgreSQL MongoDB Data Pipeline 52

AltexSoft

JULY 29, 2022

To facilitate data ingestion, there are Apache Flume aggregating log data from multiple servers and Apache Sqoop designed to transport information between Hadoop and relational (SQL) databases. Today, companies have the opportunity to run Big Data analytics on Hadoop without investing in hardware.

Hadoop 59

Hadoop Big Data Google Cloud NoSQL 59

Rockset

OCTOBER 4, 2022

Separation of Compute and Storage Design for the cloud is another area where Rockset and ClickHouse diverge. ClickHouse is offered as software, which can be self-managed on-premises or on cloud infrastructure. Several vendors also offer cloud versions of ClickHouse.

MySQL 52

MySQL Kafka Aggregated Data Architecture 52

AltexSoft

MAY 14, 2021

Big Data analytics encompasses the processes of collecting, processing, filtering/cleansing, and analyzing extensive datasets so that organizations can use them to develop, grow, and produce better products. Big Data analytics processes and tools. Data ingestion. Data storage and processing.

Big Data 78

Big Data Data Analytics IT NoSQL 78