Data Engineering Podcast

FEBRUARY 18, 2024

Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). Data lakes are notoriously complex. Visit: dataengineeringpodcast.com/data-council today. Your first 30 days are free!

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Monte Carlo

APRIL 4, 2023

Data mesh vs data warehouse is an interesting framing because it is not necessarily a binary choice depending on what exactly you mean by data warehouse (more on that later). Despite their differences, however, both approaches require high-quality, reliable data in order to function. What is a Data Mesh?

Data Warehouse 52

Data Warehouse Data Governance Data Architecture 52

RandomTrees

FEBRUARY 6, 2024

Data engineering, the practice of collecting, transforming, and organizing data for analysis, is poised for a significant transformation with the advent of Generative Artificial Intelligence (Gen AI). Generative AI with ETL Pipelines: Generative AI can be used to automate the creation of ETL pipelines.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

Monte Carlo

FEBRUARY 6, 2023

So, you’re planning a cloud data warehouse migration. But be warned, a warehouse migration isn’t for the faint of heart. As you probably already know if you’re reading this, a data warehouse migration is the process of moving data from one warehouse to another. A worthy quest to be sure.

Data Warehouse 52

Data Warehouse AWS Data Validation Data 52

Data Engineering Podcast

JANUARY 21, 2024

In this episode Tasso Argyros, CEO of ActionIQ, gives a summary of the major epochs in database technologies and how he is applying the capabilities of cloud data warehouses to the challenge of building more comprehensive experiences for end-users through a modern customer data platform (CDP).

Data Lake 147

Data Lake High Quality Data NoSQL Data Warehouse 147

Monte Carlo

AUGUST 25, 2023

Different vendors offering data warehouses, data lakes, and now data lakehouses all offer their own distinct advantages and disadvantages for data teams to consider. So let’s get to the bottom of the big question: what kind of data storage layer will provide the strongest foundation for your data platform?

Data Lake 52

Data Lake Data Warehouse Unstructured Data Raw Data 52

Edureka

FEBRUARY 9, 2023

Azure Data Engineering is a rapidly growing field that involves designing, building, and maintaining data processing systems using Microsoft Azure technologies. Contents: What is the role of an Azure Data Engineer?

Data Engineering 52

Data Engineering Data Engineer Engineering Amazon Web Services 52

Data Engineering Weekly

NOVEMBER 26, 2023

It’s the end of the year, and there will be a lot of buzz about what the next five years in data engineering might bring. We saw a fleet of announcements from Data Catalogs tools on how LLM can help to auto-generate documentation [See: How Generative AI Is Making Data Catalogs Smarter ].

Data Engineering 70

Data Engineering Data Engineer Engineering Kafka 70

Data Engineering Podcast

FEBRUARY 4, 2024

In this episode Yingjun Wu explains how it is architected to power analytical workflows on continuous data flows, and the challenges of making it responsive and scalable. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

SQL 173

SQL Data Lake High Quality Data Data Pipeline 173

Knowledge Hut

NOVEMBER 2, 2023

Azure Data engineering projects are complicated and require careful planning and effective team participation for a successful completion. While many technologies are available to help data engineers streamline their workflows and guarantee that each aspect meets its objectives, ensuring that everything works properly takes time.

Data Engineering 52

Data Engineering Data Engineer Coding Project 52

Picnic Engineering

JULY 26, 2022

Data Mesh, or Hub-and-Spoke? and other reflections on building data governance. What are your thoughts on the topic of Data Mesh, and whether to adopt it at Picnic? As a Data Lead, I feel that I don’t have the best seat at the organizational table. What exactly is a “lakeless” Data Warehouse?

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

DareData

JANUARY 30, 2023

Learn the most important data engineering concepts that data scientists should be aware of. As the field of data science and machine learning continues to evolve, it is increasingly evident that data engineering cannot be separated from it. DigDag: An open-source orchestrator for data engineering workflows.

Data Engineering 52

Data Engineering Data Engineer NoSQL Engineering 52

Knowledge Hut

APRIL 25, 2023

The tremendous growth in data generation, then the rise in data engineer jobs - there’s no arguing the fact that the big data industry is at its best pace and you, as an aspiring data engineer, have a lot to learn and make out of it - including some tools! What are Data Engineering Tools?

Data Engineering 52

Data Engineering Data Engineer Engineering Google Cloud 52

Data Science Blog: Data Engineering

MAY 20, 2024

Continuous Integration and Continuous Delivery (CI/CD) for Data Pipelines: It is a Game-Changer with AnalyticsCreator! The need for efficient and reliable data pipelines is paramount in data science and data engineering. It offers full BI-Stack Automation, from source to data warehouse through to frontend.

Data Pipeline 52

Data Pipeline BI Data Lake Data Warehouse 52

Knowledge Hut

JUNE 23, 2023

Data engineers make a tangible difference with their presence in top-notch industries, especially in assisting data scientists in machine learning and deep learning. Let us understand here the complete big data engineer roadmap to lead a successful Data Engineering Learning Path.

Data Engineering 52

Data Engineering Data Engineer Engineering Non-relational Database 52

Data Engineering Weekly

NOVEMBER 13, 2022

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Sign up free to test out the tool today.

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Knowledge Hut

MARCH 7, 2024

Two, it creates a commonality of data definitions, concepts, metadata and the like. The traditional data management and data warehouses, and the sequence of data transformation, extraction and migration- all arise a situation in which there are risks for data to become unsynchronized.

Big Data 98

Big Data Bytes Data Governance Raw Data 98

Monte Carlo

SEPTEMBER 1, 2022

Data engineering is a relatively young profession, even for the tech space. To put it in perspective, front-end engineering has twice the number of years in industry maturity. That’s because not only is data engineering talent in high demand, but the very nature of handling big data necessitates prioritization and focus.

Data Engineering 52

Data Engineering Data Engineer Engineering Datasets 52

Knowledge Hut

SEPTEMBER 21, 2023

Data is the new fuel, and data engineers are the architects who refine and harness its power. As an aspiring data engineer, this is the world I entered a world where I have the privilege of harnessing the power of data to leverage its full potential. Who is an Azure Data Engineer and What Do They Do?

Data Engineering 52

Data Engineering Data Engineer Engineering Cloud Computing 52

phData: Data Engineering

JANUARY 3, 2022

What is Data Engineering? Everything You Need to Know in 2022 Nick Goble January 4, 2022 It’s easy to overlook the amount of data that’s being generated every day — from your smartphone, your Zoom calls, to your Wi-Fi-connected dishwasher. Table of Contents What is Data Engineering What is Data Governance?

Data Engineering 52

Data Engineering Data Engineer Engineering Data Governance 52

Edureka

FEBRUARY 7, 2023

An Azure Data Engineer is responsible for designing, implementing, and maintaining data management and data processing systems on the Microsoft Azure cloud platform. They work with large and complex data sets and are responsible for ensuring that data is stored, processed, and secured efficiently and effectively.

Data Engineering 52

Data Engineering Data Engineer Engineering Programming Language 52

Monte Carlo

OCTOBER 17, 2022

“Privatize” the commons The most successful data governance programs I’ve come across have domain-driven processes with clear lines of ownership and manageable scopes. For example, Clearcover Senior Data Engineering Manager Braun Reyes described how his organization has been successful deploying this type of strategy. “We

Data Warehouse 52

Data Warehouse Government Data Governance Data 52

Databand.ai

DECEMBER 13, 2022

The Top 25 Data Engineering Influencers and Content Creators on LinkedIn Ryan Yackel 2022-12-13 10:23:19 Interested in data engineering? LinkedIn is full of influencers sharing new ideas and sparking conversations on all kinds of topics, and data engineering is no exception. You’ve come to the right place.

Data Engineering 52

Data Engineering Data Engineer Engineering AWS 52

Christophe Blefari

SEPTEMBER 15, 2023

If you're late to the party and you need fresh views on LLMs Daniel wrote an introduction demystifying the Large Language Models and Jesse wrote about LLMs impact from a Data Engineering perspective. How to reduce warehouse costs? — Hugo propose 7 hacks to optimise data warehouse cost. Crazy amounts.

Data Warehouse 130

Data Warehouse Data SQL Python 130

ProjectPro

AUGUST 24, 2021

Nevertheless, that is not the only job in the data world. Data professionals who work with raw data like data engineers, data analysts, machine learning scientists , and machine learning engineers also play a crucial role in any data science project.

Data Engineering 52

Data Engineering Data Engineer Coding Project 52

Towards Data Science

JUNE 8, 2023

Some organizations will attempt to diffuse the responsibility widely across data stewards, data owners, data engineering and governance committees, each owning a fraction of the data value chain. Why is data quality ownership important? The whole needs to be greater than the sum of its parts.

Data Governance 76

Data Governance Government Generalist Data Engineering 76

Monte Carlo

JUNE 6, 2023

Some organizations will attempt to diffuse the responsibility widely across data stewards, data owners, data engineering and governance committees, each owning a fraction of the data value chain. Table of Contents Why is important to answer who is responsible for data quality?

Data Governance 98

Data Governance Government Data Data Engineering 98

Towards Data Science

JULY 24, 2023

While there is no right answer, there is likely a sweet spot for most organizations’ data platforms. Photo by Kelly Sikkema on Unsplash Data engineers discovered the benefits of conscious uncoupling around the same time as Gwyneth Paltrow and Chris Martin in 2014. Take data governance for example.

Metadata 95

Metadata Data Warehouse Data Lake Data Science 95

Data Engineering Podcast

DECEMBER 18, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you're ready to build your next pipeline, or want to test out the projects you hear about on the show, you'll need somewhere to deploy it, so check out our friends at Linode. or any other destination you choose.

Data Lake 100

Data Lake Data Warehouse Data Pipeline MongoDB 100

Data Engineering Podcast

JULY 12, 2021

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management You listen to this show to learn about all of the latest tools, patterns, and practices that power data engineering projects across every domain. Go to dataengineeringpodcast.com/97things today to get your copy!

Designing 100

Designing Data Warehouse Data Governance Consulting 100

Data Engineering Podcast

APRIL 26, 2021

In this episode Maxime Beauchemin discusses how data engineers can use Superset to provide self service access to data and deliver analytics. He digs into how it integrates with your data stack, how you can extend it to fit your use case, and why open source systems are a good choice for your business intelligence.

Business Intelligence 100

Business Intelligence Data Warehouse Hadoop Data Pipeline 100

Data Engineering Podcast

JUNE 11, 2021

He shares some of the common patterns for building pipelines to power business intelligence dashboards, machine learning applications, and data warehouses. Are you bored with writing scripts to move data into SaaS tools like Salesforce, Marketo, or Facebook Ads? No more scripts, just SQL.

Google Cloud 100

Google Cloud Cloud Big Data Ecosystem BI 100

Data Engineering Podcast

JUNE 5, 2022

In this episode Sean Falconer explains the idea of a data privacy vault and how this new architectural element can drastically reduce the potential for making a mistake with how you manage regulated or personally identifiable information. Data teams are increasingly under pressure to deliver. how does it enter an organization)?

Data Security 100

Data Security Metadata MongoDB Scala 100

Striim

NOVEMBER 15, 2023

Through Striim’s breakthrough of AI-ready data streaming, we’re ushering in a new era of analytics and AI, all harmonized under a single data platform on Microsoft Azure. All of these services are served by OneLake, a unified intelligent storage layer that solves the complex problem of decentralized data teams working in silos.

Retail 52

Retail BI Data Warehouse Data Lake 52

Christophe Blefari

JUNE 16, 2023

Why data consumers do not trust your reporting — It is a good illustration of the data journey manifesto. Stakeholders often notice data issues before the data team does. Data warehouses are mutable, this is one of the many root causes proposed by Lucas. Data Documentation 101: Why?

Programming Language 130

Programming Language SQL PostgreSQL Data 130

The Modern Data Company

MAY 8, 2023

Data lakes, data warehouses, data hubs, data lakehouses, and data operating systems are data management and storage solutions designed to meet different needs in data analytics, integration, and processing. However, data warehouses can experience limitations and scalability challenges.

Data Management 52

Data Management Data Lake Management Data Governance 52

The Modern Data Company

MAY 8, 2023

Data lakes, data warehouses, data hubs, data lakehouses, and data operating systems are data management and storage solutions designed to meet different needs in data analytics, integration, and processing. However, data warehouses can experience limitations and scalability challenges.

Data Management 52

Data Management Data Lake Management Data Governance 52