Data Engineering Podcast

OCTOBER 30, 2022

In this episode he shares his experiences working with organizations to adopt analytics engineering patterns and the ways that Optimus and dbt were combined to let data analysts deliver insights without the roadblocks of complex pipeline management. Interview Introduction How did you get involved in the area of data management?

Engineering 100

Engineering MongoDB Scala MySQL 100

Data Engineering Podcast

OCTOBER 23, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Database 100

Database MySQL Cloud MongoDB 100

Data Engineering Podcast

OCTOBER 16, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Lake 100

Data Lake Food MongoDB Scala 100

Data Engineering Podcast

JULY 24, 2022

Summary The current stage of evolution in the data management ecosystem has resulted in domain and use case specific orchestration capabilities being incorporated into various tools. In this episode Nick Schrock discusses the importance of orchestration and a central location for managing data systems, the road to Dagster’s 1.0

MongoDB 100

MongoDB Scala MySQL Data Lake 100

Data Engineering Podcast

SEPTEMBER 25, 2022

In this episode Mark Van de Wiel explains how they integrated CDC functionality into their existing product, discusses the nuances of different approaches to change data capture from various sources. The Ascend Data Automation Cloud provides a unified platform for data ingestion, transformation, orchestration, and observability.

Food 100

Food MongoDB Scala MySQL 100

Data Engineering Podcast

SEPTEMBER 18, 2022

He discusses the challenges of making data collection purposeful and efficient without distracting employees from their primary duties and how business owners can use the provided analytics to support their staff in their duties. Can you describe what OpsAnalitica is and the story behind it? Closing Announcements Thank you for listening!

Hospitality 100

Hospitality Food MongoDB Scala 100

Data Engineering Podcast

AUGUST 28, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Building 100

Building MongoDB Scala MySQL 100

Data Engineering Podcast

JUNE 5, 2022

In this episode Sean Falconer explains the idea of a data privacy vault and how this new architectural element can drastically reduce the potential for making a mistake with how you manage regulated or personally identifiable information. Can you describe what Skyflow is and the story behind it?

Data Security 100

Data Security Metadata MongoDB Scala 100

Data Engineering Podcast

SEPTEMBER 11, 2022

In this episode he shares his journey from building a consumer product to launching a data pipeline service and how his frustrations as a product owner have informed his work at Hevo Data. The Ascend Data Automation Cloud provides a unified platform for data ingestion, transformation, orchestration, and observability.

Data Pipeline 130

Data Pipeline Building MongoDB Scala 130

Data Engineering Podcast

JULY 10, 2022

In this episode Chris Riccomini shares his experiences building and scaling data operations at WePay and LinkedIn, as well as the lessons he has learned working with other teams as they automated their own systems. What are the pieces of data platforms and processing that have been most difficult to scale in an organizational sense?

Data Engineering 100

Data Engineering Data Engineer Engineering MongoDB 100

Data Engineering Podcast

JULY 3, 2022

In this episode they explain how the utility is implemented to run quickly and how you can start using it in your own data workflows to ensure that your data warehouse isn’t missing any records from your source systems. Can you describe what the data diff tool is and the story behind it?

Data Integration 100

Data Integration MongoDB Scala MySQL 100

Data Engineering Podcast

AUGUST 6, 2022

In this episode Frank Liu explains how the open source Milvus vector database is implemented to speed up machine learning development cycles, how to think about proper storage and scaling of these vectors, and how data engineering and machine learning teams can collaborate on the creation and maintenance of these data sets.

Machine Learning 100

Machine Learning Database MySQL PostgreSQL 100

Data Engineering Podcast

OCTOBER 9, 2022

In this episode Purvi Shah, the VP of Enterprise Big Data Platforms at American Express, explains how they have invested in the cloud to power this visibility and the complex suite of integrations they have built and maintained across legacy and modern systems to make it possible. Closing Announcements Thank you for listening!

Food 100

Food MongoDB Scala MySQL 100

Data Engineering Podcast

OCTOBER 2, 2022

In this episode co-founder Martin Sahlen explains the impact that easy access to lineage information can have on the work of data engineers and analysts, and how he and his team have designed their platform to offer that information to engineers and stakeholders in the places that they interact with data.

IT 100

IT Food PostgreSQL MongoDB 100

Data Engineering Podcast

AUGUST 13, 2022

In this episode founder Shayan Mohanty explains how he and his team are bringing software best practices and automation to the world of machine learning data preparation and how it allows data engineers to be involved in the process. Can you describe what Watchful is and the story behind it? What are your core goals at Watchful?

Machine Learning 100

Machine Learning Pipeline-centric Database-centric MongoDB 100

Data Engineering Podcast

SEPTEMBER 4, 2022

In this episode Gopal Erinjippurath discusses the data engineering challenges of building and serving those data sets, and how they are distilling complex climate information into consumable facts so you don’t have to be an expert to understand it. Can you describe what Sust Global is and the story behind it?

MongoDB 100

MongoDB Scala MySQL Machine Learning 100

Lyft Engineering

NOVEMBER 29, 2023

Druid at Lyft Apache Druid is an in-memory, columnar, distributed, open-source data store designed for sub-second queries on real-time and historical data. Druid enables low latency (real-time) data ingestion, flexible data exploration and fast data aggregation resulting in sub-second query latencies.

Kafka 104

Kafka Data Ingestion Datasets Architecture 104

Ascend.io

OCTOBER 4, 2023

Snowpark stands out as a game-changer for data engineers. It empowers them to tap into the familiar terrain of languages like Scala, Java, and Python, but with the unique advantage of not having to move data out of Snowflake. Snowflake is renowned for its vast capabilities, and Snowpark is no exception.

Scala 52

Scala Python Data Ingestion Java 52

Cloudera

APRIL 20, 2021

The developers must understand lower-level languages like Java and Scala and be familiar with the streaming APIs. A modern streaming architecture consists of critical components that provide data ingestion, security and governance, and real-time analytics. What is modern streaming architecture?

Kafka 99

Kafka Hospitality Retail Data Ingestion 99

Cloudera

MARCH 27, 2019

Cloudera Flow Management (CFM) is a no-code data ingestion and management solution powered by Apache NiFi. With a slick user interface, 300+ processors and the NiFi Registry, CFM delivers highly scalable data management and DevOps capabilities to the enterprise. It comes in two flavors – Java and C++.

Management 72

Management Data Ingestion Machine Learning Java 72

Snowflake

AUGUST 16, 2023

Data comes in a continuous manner, and often a separate architecture is required to handle streaming data. What remains challenging is how streaming data is brought together with batch data. That’s why we built Snowpipe Streaming, now generally available to handle row-set data ingestion. Learn more here.

Scala 52

Scala Transportation Kafka Data Lake 52

AltexSoft

AUGUST 29, 2023

In 2010, a transformative concept took root in the realm of data storage and analytics — a data lake. The term was coined by James Dixon , Back-End Java, Data, and Business Intelligence Engineer, and it started a new era in how organizations could store, manage, and analyze their data.

Data Lake 52

Data Lake Architecture IT Amazon Web Services 52

Knowledge Hut

OCTOBER 3, 2023

Additionally, for a job in data engineering, candidates should have actual experience with distributed systems, data pipelines, and related database concepts. First and foremost, the training institute needs to have a solid reputation for offering high-quality instruction in data engineering and cloud computing.

Data Engineering 52

Data Engineering Data Engineer Engineering Cloud Computing 52

ProjectPro

FEBRUARY 21, 2023

Due to the enormous amount of data being generated and used in recent years, there is a high demand for data professionals, such as data engineers, who can perform tasks such as data management, data analysis, data preparation, etc. big data and ETL tools, etc.

Certification 52

Certification Data Engineering Data Engineer Engineering 52

Data Engineering Podcast

NOVEMBER 20, 2022

In this episode CEO and founder Salma Bakouk shares her views on the causes and impacts of "data entropy" and how you can tame it before it leads to failures. The Ascend Data Automation Cloud provides a unified platform for data ingestion, transformation, orchestration, and observability.

Data Lake 130

Data Lake Data Ingestion MongoDB Scala 130

Data Engineering Podcast

JULY 23, 2023

In this episode he explains how investing in high performance and operationally simplified streaming with a familiar API can yield significant benefits for software and data teams together. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles.

Building 130

Building Machine Learning SQL Python 130

Data Engineering Podcast

JUNE 19, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Metadata 130

Metadata MongoDB Scala MySQL 130

Knowledge Hut

AUGUST 18, 2023

Furthermore, the GCP certification helped me gain a deeper understanding of cloud security & data management. To pass the exam, you need to have a detailed understanding of Google Cloud data processing technologies related to data ingestion, storage, processing, & analysis. (d)

Google Cloud 52

Google Cloud Certification Cloud Cloud Computing 52

Data Engineering Podcast

JULY 31, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. How does it address those goals?

Data Analysis 100

Data Analysis MongoDB Scala Algorithm 100

Data Engineering Podcast

MAY 29, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Architecture 100

Architecture Database Data Architecture PostgreSQL 100

Snowflake

NOVEMBER 29, 2023

The data is there, it’s just not FAIR: Findable, Accessible, Interoperable and Reusable. Defining FAIR data and it’s applications for life sciences FAIR was a term coined in 2016 to help define good data management practices within the scientific realm. The principles emphasize machine-actionability (i.e.,

Metadata 86

Metadata Healthcare Medical Data Storage 86

AltexSoft

SEPTEMBER 21, 2023

It is developed in Java and built upon the highly reputable Apache Lucene library. The following are the key advantages that make Elasticsearch a great choice for data search and analytics. Scalability Elasticsearch is built for scalability , enabling businesses to grow without worrying about data management constraints.

Engineering 52

Engineering NoSQL Programming Language Java 52

Snowflake

MARCH 30, 2023

3EJHjvm Once a business need is defined and a minimal viable product ( MVP ) is scoped, the data management phase begins with: Data ingestion: Data is acquired, cleansed, and curated before it is transformed. ML workflow, ubr.to/3EJHjvm

Engineering 82

Engineering Raw Data Data Science Scala 82

AltexSoft

JULY 29, 2022

Apache Hadoop is an open-source Java-based framework that relies on parallel processing and distributed storage for analyzing massive datasets. Developed in 2006 by Doug Cutting and Mike Cafarella to run the web crawler Apache Nutch, it has become a standard for Big Data analytics. What is Hadoop? Hadoop ecosystem evolvement.

Hadoop 59

Hadoop Big Data Google Cloud NoSQL 59

ProjectPro

JANUARY 31, 2023

Define Big Data and Explain the Seven Vs of Big Data. Big Data is a collection of large and complex semi-structured and unstructured data sets that have the potential to deliver actionable insights using traditional data management tools. Data Storage: Spark stores data using the HDFS file system.

Big Data 52

Big Data Hadoop Relational Database AWS 52

ProjectPro

DECEMBER 6, 2016

Pentaho published a whitepaper titled “Hadoop and the Analytic Data Pipeline” that highlights the key categories which need to be focused on - Big Data Ingestion, Transformation, Analytics, Solutions. Source: [link] ) How Trifacta is helping data wranglers in Hadoop, the cloud, and beyond.Zdnet.com, November 4,2016.

Hadoop 52

Hadoop Data Lake BI Big Data 52

ProjectPro

NOVEMBER 15, 2021

It even allows you to build a program that defines the data pipeline using open-source Beam SDKs (Software Development Kits) in any three programming languages: Java, Python, and Go. A Vespa instance is made up of several stateless Java container clusters and one or more data-storing content clusters.

Big Data 52

Big Data Project Metadata Programming Language 52