Data Engineering Podcast

JULY 22, 2019

Summary The current trend in data management is to centralize the responsibilities of storing and curating the organization’s information to a data engineering team. This organizational pattern is reinforced by the architectural pattern of data lakes as a solution for managing storage and access.

Data Lake 100

Data Lake Hadoop Data Architecture 100

Data Engineering Podcast

NOVEMBER 2, 2020

Summary Data lakes are gaining popularity due to their flexibility and reduced cost of storage. Along with the benefits there are some additional complexities to consider, including how to safely integrate new data sources or test out changes to existing pipelines. When is LakeFS the wrong choice?

Data Lake 100

Data Lake PostgreSQL Datasets Machine Learning 100

Precisely

JANUARY 18, 2024

After all, mainframe technology was firmly established long before many of today’s modern innovations emerged on the scene. Mainframe modernization has accelerated in recent years, breathing new life into mainframe technology and accelerating digital transformation at organizations that rely on these systems.

Technology 52

Technology AWS Cloud Computing Data Lake 52

phData: Data Engineering

APRIL 4, 2023

At phData we try to be at the forefront of new technologies and capabilities that might benefit our customers. Today we want to introduce Fivetran’s support for Amazon S3 with Apache Iceberg, investigate some of the implications of this feature, and learn how it fits into the modern data architecture as a whole.

Data Lake 52

Data Lake Amazon Web Services Data Cleanse Data Warehouse 52

Data Engineering Podcast

NOVEMBER 11, 2018

Summary A data lake can be a highly valuable resource, as long as it is well built and well managed. In this episode Yoni Iny, CTO of Upsolver, discusses the various components that are necessary for a successful data lake project, how the Upsolver platform is architected, and how modern data lakes can benefit your organization.

Data Lake 100

Data Lake Building Kafka Cloud 100

Precisely

JANUARY 19, 2023

After all, mainframe technology was firmly established long before many of today’s modern innovations emerged on the scene. Today’s modernization options are breathing new life into mainframe technology, accelerating digital transformation at organizations that rely on these systems.

Technology 52

Technology AWS Cloud Computing Data Lake 52

Monte Carlo

JUNE 2, 2021

As companies leverage more and more data to drive decision-making and maintain their competitive edge, it’s crucial that this data is accurate and reliable. Over the last few years, data lakes have emerged as a must-have for the modern data stack. What is data observability?

Data Lake 52

Data Lake Machine Learning Unstructured Data Datasets 52

Data Engineering Podcast

MAY 22, 2022

Acryl Data provides DataHub as an easy to consume SaaS product which has been adopted by several companies. Signup for the SaaS product at dataengineeringpodcast.com/acryl RudderStack helps you build a customer data platform on your warehouse or data lake. Stop struggling to speed up your data lake.

Machine Learning 130

Machine Learning Data Engineering Data Engineer Cloud 130

Data Engineering Podcast

FEBRUARY 19, 2023

Because of their complete ownership of your data they constrain the possibilities of what data you can store and how it can be used. Can you describe what Iceberg is and its position in the data lake/lakehouse ecosystem? Can you describe what Iceberg is and its position in the data lake/lakehouse ecosystem?

IT 147

IT Data Lake Metadata Data Warehouse 147

Christophe Blefari

FEBRUARY 2, 2024

Fivetran uses DuckDB as the tech to do file merge in the data lake offering Datacamp uses DuckDB to be able in notebooks to query dataframes in SQL and consider it for teaching SQL — I might have something in the making about this on my side. Lastly they pre-compute statistics on datasets with DuckDB.

Datasets 130

Datasets Data Lake SQL Algorithm 130

Christophe Blefari

JANUARY 20, 2024

Firstly, read the Data Engineering Manifesto , this is not something official in any kind but it greatly depicts all the concepts data engineers daily face. Then here a list of global resources that can help you navigate through the field: The Data Engineer Roadmap — An image with advices and technology names to watch.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

ProjectPro

JULY 22, 2016

For the same cost, organizations can now store 50 times as much data as in a Hadoop data lake than in a data warehouse. Data lake is gaining momentum across various organizations and everyone wants to know how to implement a data lake and why.

Data Lake 40

Data Lake Data Warehouse Hadoop Unstructured Data 40

Towards Data Science

MAY 3, 2023

Data Engineering Learn about slow change dimensions (SCD) and how to implement SCD Type 2 in VDK Photo by Joshua Sortino on Unsplash Data is the backbone of any organization, and in today’s fast-paced world, it is crucial to keep track of its versions. Use VDK to build a data lake and merge multiple sources.

Data Lake 87

Data Lake Data SQL Data Warehouse 87

Data Engineering Podcast

DECEMBER 4, 2022

Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. RudderStack helps you build a customer data platform on your warehouse or data lake. What do you have planned for the future of Materialize and real-time data?

Data Lake 100

Data Lake MongoDB MySQL Data Warehouse 100

Data Engineering Podcast

JANUARY 28, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data.

Data Lake 147

Data Lake Building High Quality Data AWS 147

The Modern Data Company

MAY 10, 2023

In our previous post, The Pros and Cons of Leading Data Management and Storage Solutions , we untangled the differences among data lakes, data warehouses, data lakehouses, data hubs, and data operating systems. Data lakes offer a scalable and cost-effective solution.

Data Management 52

Data Management Data Lake Management Data Warehouse 52

Data Engineering Weekly

MAY 12, 2024

The blog points out that Gen AI is more capable than humans of creating disinformation, especially deep fake technologies. Atlassian emphasizes how crucial it is for its data lake, utilized by over half of its employees, to support continuous growth and adaptability.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

The Modern Data Company

MAY 10, 2023

In our previous post, The Pros and Cons of Leading Data Management and Storage Solutions , we untangled the differences among data lakes, data warehouses, data lakehouses, data hubs, and data operating systems. Data lakes offer a scalable and cost-effective solution.

Data Management 52

Data Management Management Data Lake Data Warehouse 52

The Modern Data Company

MAY 10, 2023

In our previous post, The Pros and Cons of Leading Data Management and Storage Solutions , we untangled the differences among data lakes, data warehouses, data lakehouses, data hubs, and data operating systems. Data lakes offer a scalable and cost-effective solution.

Data Management 52

Data Management Management Data Lake Data Warehouse 52

Knowledge Hut

MARCH 28, 2024

Apart from the demand, pursuing Azure data engineer jobs has numerous advantages, such as high salaries, opportunities for career advancement, and the possibility to work with the most advanced technologies in the field of data innovation. Implement data ingestion, processing, and analysis pipelines for large-scale data sets.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Data Engineering Podcast

DECEMBER 11, 2022

Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. RudderStack helps you build a customer data platform on your warehouse or data lake. RudderStack helps you build a customer data platform on your warehouse or data lake.

Database 100

Database MySQL Data Lake MongoDB 100

Data Engineering Podcast

OCTOBER 16, 2022

Summary The "data lakehouse" architecture balances the scalability and flexibility of data lakes with the ease of use and transaction support of data warehouses. Mention the podcast to get a free "In Data We Trust World Tour" t-shirt. When is Dremio the wrong choice?

Data Lake 100

Data Lake Food MongoDB Scala 100

Knowledge Hut

MARCH 19, 2024

Picture it as a beautiful performance where data and virtualization work together smoothly, creating a story that goes beyond just technology – it's a big change in how businesses work. Background: Siemens, a global technology and engineering company, operates in various sectors, including energy.

Cloud Computing 52

Cloud Computing Cloud Amazon Web Services AWS 52

Data Engineering Podcast

FEBRUARY 25, 2024

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics.

Database 162

Database Technology Data Lake High Quality Data 162

Snowflake

JUNE 20, 2023

In this edition, we’re digging into cybersecurity with Parth Shah, Co-Founder and Head of Product at Dassana , as he discusses the power of operationalizing security data, why you need to consider security data lakes, and how Snowflake gave Dassana an agility upgrade. What inspires you as a founder?

Data Lake 57

Data Lake Business Intelligence Cloud Technology 57

Data Engineering Weekly

MARCH 3, 2024

The migration enhanced data quality, lineage visibility, performance improvements, cost reductions, and better reliability and scalability, setting a robust foundation for future expansions and onboarding. link] All rights reserved ProtoGrowth Inc, India.

Data Engineering 67

Data Engineering Data Engineer Pipeline-centric Engineering 67

Ascend.io

AUGUST 31, 2023

Secondly , the rise of data lakes that catalyzed the transition from ELT to ELT and paved the way for niche paradigms such as Reverse ETL and Zero-ETL. Still, these methods have been overshadowed by EtLT — the predominant approach reshaping today’s data landscape.

Data Lake 52

Data Lake ETL Tools Data Warehouse Data Pipeline 52

Data Engineering Podcast

AUGUST 20, 2021

Summary Data lakes have been gaining popularity alongside an increase in their sophistication and usability. Despite improvements in performance and data architecture they still require significant knowledge and experience to deploy and manage. Can you describe what Cuelake is and the story behind it?

Building 100

Building Data Lake Data Warehouse SQL 100

Knowledge Hut

NOVEMBER 17, 2023

To provide end users with a variety of ready-made models, Azure Data engineers collaborate with Azure AI services built on top of Azure Cognitive Services APIs. The data engineers are responsible for creating conversational chatbots with the Azure Bot Service and automating metric calculations using the Azure Metrics Advisor.

Data Engineering 52

Data Engineering Data Engineer Engineering Scala 52

Cloudera

MARCH 17, 2023

I love the enthusiasm surrounding ChatGPT and the eagerness to learn about modern data architectures such as data lakehouses, data meshes, and data fabrics. ChatGPT is an excellent resource for gaining high-level insights and building awareness of any technology.

Education 61

Education Unstructured Data Data Lake Data Warehouse 61

Christophe Blefari

FEBRUARY 24, 2023

This goes further than being a data-driven enterprise , you have to put in place a framework the puts data measurement at every product choice, resulting in maturity increase. Data Engineers and Kubernetes — A 101 guide about Kubernetes concepts and why you should as a data engineers understand kube basic entities.

Kafka 130

Kafka Data Lake Data Storage Data 130

Snowflake

SEPTEMBER 14, 2023

Companies hampered by legacy data architectures are often plagued by a high total cost of ownership (TCO), an inability to govern data, and a lack of scalability as their data volumes grow. “As Your technology stack should accommodate growth—in data volumes as well as in your business.

Data Architecture 95

Data Architecture Architecture Hadoop Data Warehouse 95

Data Engineering Podcast

SEPTEMBER 7, 2020

In this episode he explains how it is designed to allow for querying and combining data where it resides, the use cases that such an architecture unlocks, and the innovative ways that it is being employed at companies across the world. What are the tradeoffs of using Presto on top of a data lake vs a vertically integrated warehouse solution?

Architecture 100

Architecture Data Architecture SQL Engineering 100

Knowledge Hut

SEPTEMBER 25, 2023

Skill Requirements for Azure Data Engineer Job Description Here are some important skill requirements that you may find in a job description for Azure Data Engineers: 1. Azure Data Engineers work with these and other solutions. They guarantee that the data is efficiently cleaned, converted, and loaded.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

Cloudera

OCTOBER 6, 2022

Accenture : “Digital transformation is the process by which companies embed technologies across their businesses to drive fundamental change.”. Wikipedia : “ Digital transformation (DX ) is the adoption of digital technology by an organization.”. These data lakes house much of the data needed to also support other use cases.

Data Lake 85

Data Lake Machine Learning Data Storage Cloud Computing 85

Cloudera

JANUARY 30, 2024

As we navigate the fourth and fifth industrial revolution, AI technologies are catalyzing a paradigm shift in how products are designed, produced, and optimized. Quality: Use cases like visual inspection, yield optimization, fault detection, and classification are enhanced with AI technologies. Eliminate data silos.

Manufacturing 77

Manufacturing Data Lake Data Science Cloud 77

Data Engineering Weekly

MAY 5, 2024

link] LinkedIn: LakeChime - A Data Trigger Service for Modern Data Lakes LinkedIn points out two critical flaws in a partitioned approach to data management. The granularity of partition creation constrained data consumption.

Data Engineering 67

Data Engineering Data Engineer Engineering Google Cloud 67