Data Engineering Podcast

SEPTEMBER 1, 2021

Summary The Presto project has become the de facto option for building scalable open source analytics in SQL for the data lake. Another area that has been seeing a lot of activity is data lakes and projects to make them more manageable and feature complete (e.g. Hudi, Delta Lake, Iceberg, Nessie, LakeFS, etc.).

Data Lake 100

Data Lake Cloud AWS SQL 100

Cloudera

AUGUST 30, 2022

Modern data architectures. To eliminate or integrate these silos, the public sector needs to adopt robust data management solutions that support modern data architectures (MDAs). Deploying modern data architectures. Lack of sharing hinders the elimination of fraud, waste, and abuse. Forrester ).

Data Architecture 78

Data Architecture Architecture Data Lake NoSQL 78

Striim

FEBRUARY 12, 2024

For instance, analytical data may uncover customer preferences or seasonal product trends, helping teams in strategic planning. This data is commonly gathered through an ETL process and stored in a central location, such as a data lake. This will help achieve important goals by using data effectively.

Data Architecture 52

Data Architecture Architecture Data Data Lake 52

phData: Data Engineering

APRIL 4, 2023

At phData we try to be at the forefront of new technologies and capabilities that might benefit our customers. Today we want to introduce Fivetran’s support for Amazon S3 with Apache Iceberg, investigate some of the implications of this feature, and learn how it fits into the modern data architecture as a whole.

Data Lake 52

Data Lake Amazon Web Services Data Cleanse Data Warehouse 52

Data Engineering Podcast

JUNE 16, 2019

Summary Building and maintaining a data lake is a choose your own adventure of tools, services, and evolving best practices. The flexibility and freedom that data lakes provide allows for generating significant value, but it can also lead to anti-patterns and inconsistent quality in your analytics.

Data Lake 100

Data Lake Lambda Architecture Data Warehouse Hadoop 100

Monte Carlo

FEBRUARY 21, 2023

Today, as data sources become increasingly varied, data management becomes more complex, and agility and scalability become essential traits for data leaders, data fabric is quickly becoming the future of data architecture. If data fabric is the future, how can you get your organization up-to-speed?

Data Architecture 52

Data Architecture Architecture Metadata Unstructured Data 52

Monte Carlo

FEBRUARY 21, 2023

Today, as data sources become increasingly varied, data management becomes more complex, and agility and scalability become essential traits for data leaders, data fabric is quickly becoming the future of data architecture. If data fabric is the future, how can you get your organization up-to-speed?

Data Architecture 52

Data Architecture Architecture Metadata Unstructured Data 52

Data Engineering Podcast

SEPTEMBER 7, 2020

In this episode he explains how it is designed to allow for querying and combining data where it resides, the use cases that such an architecture unlocks, and the innovative ways that it is being employed at companies across the world. What is in store for the future of the Presto project and community?

Architecture 100

Architecture Data Architecture SQL Engineering 100

Data Engineering Podcast

JUNE 17, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake. How has the move to the cloud for data warehousing/data platforms influenced the practice of data modeling?

Data Lake 130

Data Lake Metadata Machine Learning Data Architecture 130

Monte Carlo

APRIL 1, 2021

Over the past few years, data lakes have emerged as a must-have for the modern data stack. But while the technologies powering our access and analysis of data have matured, the mechanics behind understanding this data in a distributed environment have lagged behind.

Data Lake 52

Data Lake Unstructured Data Data Warehouse Metadata 52

Towards Data Science

DECEMBER 4, 2023

Anyways, I wasn’t paying enough attention during university classes, and today I’ll walk you through data layers using —  guess what  —  an example. Business Scenario & Data Architecture Imagine this: next year, a new team on the grid, Red Thunder Racing, will call us (yes, me and you) to set up their new data infrastructure.

Data Engineering 83

Data Engineering Data Engineer Engineering Data Lake 83

Knowledge Hut

MARCH 28, 2024

Apart from the demand, pursuing Azure data engineer jobs has numerous advantages, such as high salaries, opportunities for career advancement, and the possibility to work with the most advanced technologies in the field of data innovation. Develop data models, data governance policies, and data integration strategies.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Towards Data Science

FEBRUARY 15, 2023

In this context, data management in an organization is a key point for the success of its projects involving data. One of the main aspects of correct data management is the definition of a data architecture. What is Delta Lake? The data became useless. The Lakehouse architecture was one of them.

Data Lake 77

Data Lake Data Warehouse Hadoop Data Architecture 77

Cloudera

MAY 9, 2023

New Data Lakehouse Enables Stronger Data Governance SoftBank needed to reduce the number of workloads on its existing platform and decided to adopt Cloudera to build a data lake capable of managing data more effectively. Team members with various Cloudera capabilities provided 24-hour support for upgrade.

Data Security 89

Data Security Telecommunication Data Lake Data Governance 89

Data Engineering Weekly

APRIL 28, 2024

Intuit: The Data Mesh Strategy Behind Intuit’s Global Financial Technology Platform The Data Product Builder platform is becoming increasingly important in enterprise data engineering. It offers more targeted and customized data asset building than the general-purpose data stack.

Data Engineering 52

Data Engineering Data Engineer Engineering Hospitality 52

Data Engineering Podcast

AUGUST 20, 2021

Summary Data lakes have been gaining popularity alongside an increase in their sophistication and usability. Despite improvements in performance and data architecture they still require significant knowledge and experience to deploy and manage. Can you describe what Cuelake is and the story behind it?

Building 100

Building Data Lake Data Warehouse SQL 100

Data Engineering Weekly

MARCH 3, 2024

The migration enhanced data quality, lineage visibility, performance improvements, cost reductions, and better reliability and scalability, setting a robust foundation for future expansions and onboarding.

Data Engineering 67

Data Engineering Data Engineer Pipeline-centric Engineering 67

ProjectPro

JULY 22, 2016

For the same cost, organizations can now store 50 times as much data as in a Hadoop data lake than in a data warehouse. Data lake is gaining momentum across various organizations and everyone wants to know how to implement a data lake and why.

Data Lake 40

Data Lake Data Warehouse Hadoop Unstructured Data 40

Cloudera

NOVEMBER 17, 2022

HBL has re-envisioned itself as a ‘Technology company with a banking license’, as it transforms into the bank of tomorrow – one which empowers its customers through digital enablement. The Solution: CDP Private Cloud brings a next-generation hybrid architecture with cloud-native benefits to HBL’s data platform.

Banking 84

Banking Management Data Lake Professional Services 84

Data Engineering Podcast

SEPTEMBER 23, 2018

Using the metaphor of a museum curator carefully managing the precious resources on display and in the vaults, he discusses the various layers of an enterprise data strategy. Can you walk through the stages of an ideal lifecycle for data within the context of an organizations uses for it?

Data Lake 100

Data Lake Data Warehouse Data Architecture Architecture 100

Cloudera

MARCH 17, 2023

I love the enthusiasm surrounding ChatGPT and the eagerness to learn about modern data architectures such as data lakehouses, data meshes, and data fabrics. ChatGPT is an excellent resource for gaining high-level insights and building awareness of any technology.

Education 62

Education Unstructured Data Data Lake Data Warehouse 62

Knowledge Hut

NOVEMBER 17, 2023

To provide end users with a variety of ready-made models, Azure Data engineers collaborate with Azure AI services built on top of Azure Cognitive Services APIs. The data engineers are responsible for creating conversational chatbots with the Azure Bot Service and automating metric calculations using the Azure Metrics Advisor.

Data Engineering 52

Data Engineering Data Engineer Engineering Scala 52

Cloudera

JANUARY 30, 2024

As we navigate the fourth and fifth industrial revolution, AI technologies are catalyzing a paradigm shift in how products are designed, produced, and optimized. Quality: Use cases like visual inspection, yield optimization, fault detection, and classification are enhanced with AI technologies. Eliminate data silos.

Manufacturing 79

Manufacturing Data Lake Data Science Cloud 79

DataKitchen

NOVEMBER 29, 2021

DataOps adoption continues to expand as a perfect storm of social, economic, and technological factors drive enterprises to invest in process-driven innovation. As a result, enterprises will examine their end-to-end data operations and analytics creation workflows. Data Gets Meshier.

Data Lake 98

Data Lake Manufacturing Architecture Data Architecture 98

Cloudera

SEPTEMBER 15, 2022

Cloudera Contributor: Mark Ramsey, PhD ~ Globally Recognized Chief Data Officer. July brings summer vacations, holiday gatherings, and for the first time in two years, the return of the Massachusetts Institute of Technology (MIT) Chief Data Officer symposium as an in-person event. What is a data fabric?

Data Lake 81

Data Lake Analytics Application Cloud Storage Architecture 81

Databand.ai

AUGUST 30, 2023

A DataOps architecture is the structural foundation that supports the implementation of DataOps principles within an organization. It encompasses the systems, tools, and processes that enable businesses to manage their data more efficiently and effectively. As a result, they can be slow, inefficient, and prone to errors.

Architecture 52

Architecture Data Ingestion Data Governance Data Cleanse 52

Monte Carlo

JANUARY 24, 2024

More data, more access to data, and more regulations mean data governance has become a higher-stakes game. Data Governance Trends The biggest data governance trend isn’t really a trend at all—rather, it’s a state of mind. At the same time, data governance technologies are growing more intelligent.

Data Governance 52

Data Governance Government Data Lake Data Architecture 52

Snowflake

AUGUST 10, 2023

With major clients including Spotify, Puma, Five Guys, and Icelandair, Bynder uses large amounts of data to provide dashboards and open APIs to its customers, as well as vital operational insights to internal users. But when the company started to experience rapid growth, it noticed performance issues with its data architecture. “

Cloud 52

Cloud Building Amazon Web Services BI 52

The Modern Data Company

MARCH 23, 2023

Technical debt is an ongoing issue no one should expect to square away because as technology advances, even today’s top systems will eventually achieve full “legacy” status. Various architectures, from data warehouses to data lakes, have attempted to help solve this problem over the years.

Data Lake 95

Data Lake Data Warehouse Architecture Systems 95

Cloudera

SEPTEMBER 12, 2022

Combining and analyzing both structured and unstructured data is a whole new challenge to come to grips with, let alone doing so across different infrastructures. Both obstacles can be overcome using modern data architectures, specifically data fabric and data lakehouse. Unified data fabric.

Unstructured Data 74

Unstructured Data Data Lake Data Architecture Data 74

Monte Carlo

AUGUST 31, 2023

Data is a priority for your CEO, as it often is for digital-first companies, and she is fluent in the latest and greatest business intelligence tools. Your CTO is excited about migrating to the cloud, and constantly sends your team articles highlighting performance measurements against some of the latest technologies.

Data Lake 52

Data Lake Data Pipeline Unstructured Data Data Warehouse 52

Ascend.io

AUGUST 31, 2023

In the dynamic world of data, many professionals are still fixated on traditional patterns of data warehousing and ETL, even while their organizations are migrating to the cloud and adopting cloud-native data services. Central to this transformation are two shifts.

Data Lake 52

Data Lake ETL Tools Data Warehouse Data Pipeline 52

AltexSoft

FEBRUARY 11, 2023

To get a better understanding of a data architect’s role, let’s clear up what data architecture is. Data architecture is the organization and design of how data is collected, transformed, integrated, stored, and used by a company. Sample of a high-level data architecture blueprint for Azure BI programs.

Data Architect 87

Data Architect Certification Generalist Big Data 87

The Modern Data Company

MARCH 1, 2024

For today’s Chief Data Officers (CDOs) and data teams, the struggle is real. We’re drowning in data yet thirsting for actionable insights. We need a new approach, a paradigm shift that delivers data with the agility and efficiency of a speedboat – enter Data Products.

Database-centric 40

Database-centric Data Lake Government Retail 40

Cloudera

NOVEMBER 27, 2023

Flink is, relatively speaking, a newer technology. However, it continues to gain adoption and inspire new development in the core engine as well as supporting technologies. Flink Forward is a great opportunity to learn about the cutting edge of streaming and stream processing technologies.

Kafka 82

Kafka SQL ETL Tools Data Lake 82

Data Engineering Podcast

NOVEMBER 18, 2019

Summary With the constant evolution of technology for data management it can seem impossible to make an informed decision about whether to build a data warehouse, or a data lake, or just leave your data wherever it currently rests. How does it influence the relevancy of data warehouses or data lakes?

Data Lake 100

Data Lake Scala Data Warehouse Hadoop 100

Knowledge Hut

NOVEMBER 17, 2023

They enhance data pipelines, transform data, and guarantee the accuracy, integrity, and compliance of the data. Their job entails Azure data engineer skills like using big data, databases, data lakes, and analytics to help firms make efficient data-driven decisions.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52