Data Engineering Podcast

APRIL 14, 2024

In this episode Oren Eini, CEO and creator of RavenDB, explores the nuances of relational vs. non-relational engines, and the strategies for designing a non-relational database. Data lakes are notoriously complex. How have the evolution in data formats (e.g. Go to dataengineeringpodcast.com/dagster today to get started.

Non-relational Database 147

Non-relational Database Relational Database Database Designing 147

Knowledge Hut

MARCH 28, 2024

Data Engineer Career: Overview Currently, with the enormous growth in the volume, variety, and veracity of data generated and the will of large firms to store and analyze their data, data management is a critical aspect of data science. That’s where data engineers are on the go.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Data Engineering Podcast

DECEMBER 11, 2022

Summary One of the most critical aspects of software projects is managing its data. Managing the operational concerns for your database can be complex and expensive, especially if you need to scale to large volumes of data, high traffic, or geographically distributed usage. or any other destination you choose.

Database 100

Database MySQL Data Lake MongoDB 100

Data Engineering Podcast

DECEMBER 4, 2022

In this episode Arjun Narayan explains how the technical barriers to adopting real-time data in your analytics and applications have become surmountable by organizations of all sizes. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows.

Data Lake 100

Data Lake MongoDB MySQL Data Warehouse 100

Data Engineering Weekly

MARCH 3, 2024

The migration enhanced data quality, lineage visibility, performance improvements, cost reductions, and better reliability and scalability, setting a robust foundation for future expansions and onboarding. Part 1: Why did we need to build our own SIEM?

Data Engineering 67

Data Engineering Data Engineer Pipeline-centric Engineering 67

Data Engineering Podcast

OCTOBER 16, 2022

Summary The "data lakehouse" architecture balances the scalability and flexibility of data lakes with the ease of use and transaction support of data warehouses. Mention the podcast to get a free "In Data We Trust World Tour" t-shirt. Can you describe what Dremio is and the story behind it?

Data Lake 100

Data Lake Food MongoDB Scala 100

The Modern Data Company

APRIL 15, 2024

Understands how different pieces of data (books) relate to each other, helping users find related information. Keeps the library orderly, adjusting as new data arrives. Tracks key performance indicators to understand the use and value of the data. Manages information about data access and monitors data quality.

Government 52

Government Metadata Data Data Lake 52

Data Engineering Podcast

AUGUST 20, 2021

Summary Data lakes have been gaining popularity alongside an increase in their sophistication and usability. Despite improvements in performance and data architecture they still require significant knowledge and experience to deploy and manage. Can you describe what Cuelake is and the story behind it?

Building 100

Building Data Lake Data Warehouse SQL 100

Data Engineering Podcast

SEPTEMBER 7, 2020

In this episode he explains how it is designed to allow for querying and combining data where it resides, the use cases that such an architecture unlocks, and the innovative ways that it is being employed at companies across the world. If you hand a book to a new data engineer, what wisdom would you add to it?

Architecture 100

Architecture Data Architecture SQL Engineering 100

Data Engineering Podcast

JANUARY 28, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data.

Data Lake 147

Data Lake Building High Quality Data AWS 147

Data Engineering Podcast

NOVEMBER 6, 2022

In this episode she shares the story behind the project, the details of how it is implemented, and how you can use it for your own data projects. RudderStack helps you build a customer data platform on your warehouse or data lake. How have the design and goals shifted since you started working on the project?

MongoDB 130

MongoDB Scala MySQL Data Lake 130

Data Engineering Podcast

MAY 8, 2022

In this episode Jon Herke shares how TigerGraph customers are taking advantage of those capabilities to achieve meaningful discoveries in their fields, the utilities that it provides for modeling and managing your connected data, and some of his own experiences working with the platform before joining the company.

Database 100

Database Data Lake BI Business Intelligence 100

Data Engineering Podcast

JUNE 12, 2022

In this episode Rod Christensen shares the story behind Aparavi and how you can use it to cut costs and gain value for the long tail of your unstructured data. Acryl Data provides DataHub as an easy to consume SaaS product which has been adopted by several companies. Can you describe what Aparavi is and the story behind it?

Unstructured Data 130

Unstructured Data MongoDB Scala MySQL 130

Data Engineering Podcast

MARCH 27, 2022

Summary Data governance is a practice that requires a high degree of flexibility and collaboration at the organizational and technical levels. The growing prominence of cloud and hybrid environments in data management adds additional stress to an already complex endeavor. What do you have planned for the future of Privacera?

Data Governance 130

Data Governance Government Cloud Building 130

Data Engineering Podcast

SEPTEMBER 11, 2022

In this episode he shares the design of the project and how it fits into your development practices. Your host is Tobias Macey and today I’m interviewing Ananth Packkildurai about Schemata, a modelling framework for decentralised domain-driven ownership of data. Can you describe what Schemata is and the story behind it?

Systems 100

Systems Metadata Building MongoDB 100

Data Engineering Podcast

NOVEMBER 13, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

MongoDB 100

MongoDB Scala MySQL Google Cloud 100

Data Engineering Podcast

FEBRUARY 9, 2020

Summary Designing the structure for your data warehouse is a complex and challenging process. As businesses deal with a growing number of sources and types of information that they need to integrate, they need a data modeling strategy that provides them with flexibility and speed.

Data Lake 100

Data Lake Data Warehouse Hadoop NoSQL 100

Knowledge Hut

NOVEMBER 2, 2023

Implementing Basic IAM Security Difficulty Level: Beginner AWS cloud practitioner projects permit the implementation of Identity and Access Management (IAM) fundamental security controls. Professionals establish IAM users, designate minimal permissions, and integrate IAM with AWS services and applications.

AWS 52

AWS Coding Cloud Project 52

Knowledge Hut

SEPTEMBER 25, 2023

You can learn deeply about the Azure learning journey that leads to the position of an Azure Data Engineer Associate on the Microsoft Azure platform. Who is an Azure Data Engineer? As an Azure Data Engineer, you will be expected to design, implement, and manage data solutions on the Microsoft Azure cloud platform.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

Knowledge Hut

NOVEMBER 17, 2023

They enhance data pipelines, transform data, and guarantee the accuracy, integrity, and compliance of the data. Their job entails Azure data engineer skills like using big data, databases, data lakes, and analytics to help firms make efficient data-driven decisions.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

Data Engineering Podcast

OCTOBER 9, 2022

In this episode Purvi Shah, the VP of Enterprise Big Data Platforms at American Express, explains how they have invested in the cloud to power this visibility and the complex suite of integrations they have built and maintained across legacy and modern systems to make it possible. Closing Announcements Thank you for listening!

Food 100

Food MongoDB Scala MySQL 100

Data Engineering Podcast

DECEMBER 11, 2021

In this episode he explains how it can be used to build rich and useful profiles of your users, the system architecture that powers it, and some of the ways that it is being integrated into an organization’s broader data ecosystem. Start trusting your data with Monte Carlo today! How does the design of Unomi reduce that burden?

Data Warehouse 100

Data Warehouse Raw Data Data Lake BI 100

Data Engineering Podcast

APRIL 16, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. Can you describe what MLOps is?

Data Engineering 100

Data Engineering Data Engineer Engineering IT 100

Data Engineering Podcast

APRIL 10, 2022

Summary Any time that you are storing data about people there are a number of privacy and security considerations that come with it. Privacy engineering is a growing field in data management that focuses on how to protect attributes of personal data so that the containing datasets can be shared safely.

Engineering 100

Engineering Data Lake Data Engineering Data Engineer 100

Data Engineering Podcast

SEPTEMBER 18, 2022

He discusses the challenges of making data collection purposeful and efficient without distracting employees from their primary duties and how business owners can use the provided analytics to support their staff in their duties. RudderStack helps you build a customer data platform on your warehouse or data lake.

Hospitality 100

Hospitality Food MongoDB Scala 100

Knowledge Hut

NOVEMBER 17, 2023

In the post, we will investigate how to become an Azure data engineer, the skills required, the roles and responsibilities of an Azure data engineer, and much more. Who is an Azure Data Engineer? This involves knowing how to manage data partitions, load data into a data warehouse, and speed up query execution.

Data Engineering 52

Data Engineering Data Engineer Engineering Scala 52

Data Engineering Podcast

OCTOBER 2, 2022

In this episode co-founder Martin Sahlen explains the impact that easy access to lineage information can have on the work of data engineers and analysts, and how he and his team have designed their platform to offer that information to engineers and stakeholders in the places that they interact with data.

IT 100

IT Food PostgreSQL MongoDB 100

Data Engineering Podcast

DECEMBER 31, 2023

In this episode Andrey Korchack, CTO of fintech startup Monite, discusses the complexities of designing and implementing a data platform in that sector. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

Designing 130

Designing Data Lake High Quality Data SQL 130

Data Engineering Podcast

SEPTEMBER 4, 2022

In this episode founders Barr Moses and Lior Gavish rejoin the show to reflect on the evolution and adoption of data observability technologies and the capabilities that are being introduced as the broader ecosystem adopts the practices. RudderStack helps you build a customer data platform on your warehouse or data lake.

IT 100

IT Metadata MongoDB MySQL 100

Data Engineering Podcast

DECEMBER 4, 2021

The whole process of hiring is an important organizational skill to cultivate and this is an interesting exploration of the specific challenges involved in finding data professionals. Start trusting your data with Monte Carlo today! Interview Introduction How did you get involved in the area of data management?

Data Warehouse 100

Data Warehouse Data Lake BI Business Intelligence 100

Data Engineering Podcast

OCTOBER 28, 2021

He also explains why he started Decodable to address that limitation and the work that he and his team have done to let data engineers build streaming pipelines entirely in SQL. Start trusting your data with Monte Carlo today! What are the complexities that data engineers encounter when building pipelines on streaming systems?

Data Pipeline 100

Data Pipeline SQL Data Warehouse Data Lake 100

Data Engineering Podcast

DECEMBER 19, 2021

In this episode Pardhu Gunnam and Mars Lan explain how they have designed the architecture and user experience to allow everyone to collaborate on the data lifecycle and provide opportunities for automation and extensible workflows. Start trusting your data with Monte Carlo today! Closing Announcements Thank you for listening!

Systems 100

Systems Building Metadata Data Warehouse 100

Edureka

FEBRUARY 7, 2023

An Azure Data Engineer is responsible for designing, implementing, and maintaining data management and data processing systems on the Microsoft Azure cloud platform. They work with large and complex data sets and are responsible for ensuring that data is stored, processed, and secured efficiently and effectively.

Data Engineering 52

Data Engineering Data Engineer Engineering Programming Language 52

Data Engineering Podcast

AUGUST 27, 2021

Summary The reason that so much time and energy is spent on data integration is because of how our applications are designed. By making the software be the owner of the data that it generates, we have to go through the trouble of extracting the information to then be used elsewhere. What is it used for?

Data Integration 100

Data Integration Architecture Data Warehouse Data Lake 100

Data Engineering Podcast

MAY 1, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Warehouse 100

Data Warehouse Data Lake Architecture Data 100

Data Engineering Podcast

DECEMBER 4, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Business Intelligence 100

Business Intelligence Metadata BI MongoDB 100

Data Engineering Podcast

AUGUST 31, 2020

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management What are the pieces of advice that you wish you had received early in your career of data engineering? If you hand a book to a new data engineer, what wisdom would you add to it?

Data Warehouse 100

Data Warehouse Cloud Building Data Lake 100