Data Engineering Podcast

DECEMBER 24, 2023

Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. With Materialize, you can!

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Data Engineering Podcast

FEBRUARY 4, 2024

RisingWave is a database engine that was created specifically for stream processing, with S3 as the storage layer. In this episode Yingjun Wu explains how it is architected to power analytical workflows on continuous data flows, and the challenges of making it responsive and scalable. Starburst : ![Starburst

SQL 173

SQL Data Lake High Quality Data Data Pipeline 173

Data Engineering Podcast

NOVEMBER 26, 2023

Summary Building a data platform that is enjoyable and accessible for all of its end users is a substantial challenge. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. Data lakes are notoriously complex.

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Data Engineering Podcast

JANUARY 21, 2024

Summary Databases and analytics architectures have gone through several generational shifts. A substantial amount of the data that is being managed in these systems is related to customers and their interactions with an organization. How have the responsibilities shifted across different roles?

Data Lake 147

Data Lake High Quality Data NoSQL Data Warehouse 147

Data Engineering Podcast

MARCH 31, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Datafold has recently launched data replication testing, providing ongoing validation for source-to-target replication.

Project 130

Project Data Lake High Quality Data Data Workflow 130

Data Engineering Podcast

NOVEMBER 5, 2023

Summary Databases are the core of most applications, but they are often treated as inscrutable black boxes. When an application is slow, there is a good probability that the database needs some attention. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products.

PostgreSQL 147

PostgreSQL Data Lake High Quality Data SQL 147

Data Engineering Podcast

JANUARY 7, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data.

Data Process 147

Data Process Process Data Lake High Quality Data 147

Data Engineering Podcast

DECEMBER 31, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data.

Designing 130

Designing Data Lake High Quality Data SQL 130

Data Engineering Podcast

DECEMBER 17, 2023

Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. With Materialize, you can!

Data Lake 130

Data Lake High Quality Data SQL Architecture 130

Data Engineering Podcast

NOVEMBER 19, 2023

Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. With Materialize, you can!

Project 147

Project Data Lake High Quality Data SQL 147

Data Engineering Podcast

DECEMBER 3, 2023

Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. With Materialize, you can!

Systems 130

Systems Designing Data Lake SQL 130

Data Engineering Podcast

DECEMBER 10, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management You shouldn't have to throw away the database to build with fast-changing data. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products.

Data Lake 130

Data Lake High Quality Data SQL Architecture 130

Data Engineering Podcast

NOVEMBER 12, 2023

If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration. Learn more about Datafold by visiting dataengineeringpodcast.com/datafold Data lakes are notoriously complex. You shouldn't have to throw away the database to build with fast-changing data.

Software Engineer 147

Software Engineer Software Engineering Engineering Data Lake 147

Monte Carlo

FEBRUARY 27, 2024

Organizations need to connect LLMs with their proprietary data and business context to actually create value for their customers and employees. They need robust data pipelines, high-quality data, well-guarded privacy, and cost-effective scalability. Data engineers. Who can deliver?

Data Engineering 59

Data Engineering Data Engineer Engineering High Quality Data 59

Monte Carlo

JANUARY 3, 2024

Let me give you a hint: high-quality proprietary data. It gives the LLM access to that enterprise proprietary data. But RAG development comes with a learning curve, even for your most talented data engineers. Databricks’ new Foundation Model APIs provide instant access to LLMs directly within Databricks.

Technology 113

Technology Database Data Governance Data Engineering 113

U-Next

FEBRUARY 27, 2023

Data normalization is the process of organizing and transforming data to improve its structural integrity, accuracy, and consistency. Data normalization is also an important part of database design. Data normalization is adopted because it helps to ensure that data will be consistent.

IT 98

IT Bytes Database Recruitment 98

Monte Carlo

MARCH 7, 2024

Why prompt engineering isn’t all that and a bag of SQL queries Understand vector databases Create AI differentiation with RAG Find and solve real business problems High-quality data always lives up to the hype What is prompt engineering? Table of Contents Why is prompt engineering important?

Engineering 59

Engineering High Quality Data Database Architecture 59

U-Next

MARCH 7, 2023

Data normalization is the process of organizing and transforming data to improve its structural integrity, accuracy, and consistency. Data normalization is also an important part of database design. Data normalization is adopted because it helps to ensure that data will be consistent.

IT 98

IT Bytes Database Recruitment 98

Databand.ai

AUGUST 30, 2023

Data consistency refers to the state of data in which all copies or instances are the same across all systems and databases. Consistency ensures that data is accurate, up-to-date, and coherent across different database systems, applications, and platforms. Data consistency is essential for various reasons.

Data Integration 52

Data Integration Data Cleanse Data Validation High Quality Data 52

Databand.ai

MAY 30, 2023

Data quality refers to the degree of accuracy, consistency, completeness, reliability, and relevance of the data collected, stored, and used within an organization or a specific context. High-quality data is essential for making well-informed decisions, performing accurate analyses, and developing effective strategies.

Data Cleanse 85

Data Cleanse Datasets Data Governance Data Validation 85

Striim

APRIL 30, 2024

Striim’s Solution Kramp adopted Striim for its powerful, mature real-time data integration, seamlessly connecting diverse databases like Oracle, Microsoft, and Postgres, to ensure continuous, high-quality data replication essential for forecasting and order management.

Google Cloud 52

Google Cloud High Quality Data Business Intelligence Data Warehouse 52

Ripple Engineering

JANUARY 29, 2024

For Ripple's product capabilities, the Payments team of Ripple, for example, ingests millions of transactional records into databases and performs analytics to generate invoices, reports, and other related payment operations.    A lack of a centralized system makes building a single source of high-quality data difficult.

Database-centric 52

Database-centric Pipeline-centric NoSQL High Quality Data 52

Monte Carlo

DECEMBER 22, 2023

Obviously data is a MASSIVE piece of the GenAI puzzle. The moral of this story: data quality will become an essential consideration for generative AI. Organizing Generative AI: 5 Lessons Learned From Data Science Teams So, your business leadership is all in on generative AI?

High Quality Data 52

High Quality Data Data Science Data Engineering Data Engineer 52

Monte Carlo

DECEMBER 22, 2023

Obviously data is a MASSIVE piece of the GenAI puzzle. The moral of this story: data quality will become an essential consideration for generative AI. Organizing Generative AI: 5 Lessons Learned From Data Science Teams So, your business leadership is all in on generative AI?

High Quality Data 52

High Quality Data Data Science Data Engineering Data Engineer 52

Knowledge Hut

APRIL 23, 2024

This guide will help you comprehend big data 4 characteristics to understand all the containing Vs! Volume: Quantity vs Accessibility Volume is the first of the four V's in big data and pertains to the size or magnitude of data being generated, collected, and stored. Consider both structured and unstructured data.

Big Data 52

Big Data Media Datasets Unstructured Data 52

Monte Carlo

JANUARY 5, 2024

It supports ACID transactions and can run fast queries, typically through SQL commands, directly on object storage in the cloud or on-prem on structured and unstructured data. The data lakehouse’s semantic layer also helps to simplify and open data access in an organization.

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

Monte Carlo

JANUARY 5, 2024

It supports ACID transactions and can run fast queries, typically through SQL commands, directly on object storage in the cloud or on-prem on structured and unstructured data. The data lakehouse’s semantic layer also helps to simplify and open data access in an organization.

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

Precisely

MARCH 14, 2023

First, private cloud infrastructure providers like Amazon (AWS), Microsoft (Azure), and Google (GCP) began by offering more cost-effective and elastic resources for fast access to infrastructure. Cloud-native data execution is just the beginning. How could streamlined access to quality data benefit your organization?

Data Integration 52

Data Integration High Quality Data BI Data 52

Precisely

SEPTEMBER 25, 2023

Data validation performs a check against existing values in a database to ensure that they fall within valid parameters. Data enrichment is the process of enhancing your data by appending relevant context from additional sources – improving its overall value, accuracy, and usability. What are the businesses near it?

Data Validation 98

Data Validation Process Raw Data Data Cleanse 98

U-Next

SEPTEMBER 7, 2022

Employee database software is quickly becoming a must-have for HR and people managers. A firm can benefit immensely from maintaining accurate and clean employee data, as it is still a difficult task. . A firm can benefit immensely from maintaining accurate and clean employee data, as it is still a difficult task. .

Data Management 52

Data Management Management Electronics Unstructured Data 52

Monte Carlo

JANUARY 23, 2024

As the machine became smarter with the next generation of data access, we flattened everything to features and columns. RAG involves integrating a real-time database into the LLM’s response generation process, while fine-tuning trains models on targeted datasets to improve domain-specific responses. RAG workflow.

Pipeline-centric 52

Pipeline-centric Database-centric Metadata Unstructured Data 52

Workfall

JULY 4, 2023

With DBT, they weave powerful SQL spells to create data models that capture the essence of their organization’s information. DBT’s superpowers include seamlessly connecting with databases and data warehouses, performing amazing transformations, and effortlessly managing dependencies to ensure high-quality data.

Data Warehouse 69

Data Warehouse SQL PostgreSQL Database 69

Data Engineering Weekly

MARCH 3, 2024

Here is the agenda, 1) Data Application Lifecycle Management - Harish Kumar( Paypal) Hear from the team in PayPal on how they build the data product lifecycle management (DPLM) systems. 3) DataOPS at AstraZeneca The AstraZeneca team talks about data ops best practices internally established and what worked and what didn’t work!!!

Data Engineering 67

Data Engineering Data Engineer Pipeline-centric Engineering 67

Precisely

JANUARY 4, 2024

As the world grows more and more interconnected, the nature of data quality initiatives must adapt to scale. Duplicate records and decaying data quality – especially in customer databases – were the primary concerns for many. But marketplaces and other democratization efforts come with some risks.

Data Governance 73

Data Governance Government Data Integration Data 73

Databand.ai

AUGUST 30, 2023

Data cleansing: Implement corrective measures to address identified issues and improve dataset accuracy levels. Data validation: Ensure new database entries adhere to predefined rules or standards to maintain dataset consistency. Additionally, high-quality data reduces costly errors stemming from inaccurate information.

Building 52

Building Data Cleanse Data Governance Datasets 52

Ascend.io

FEBRUARY 28, 2024

While data engineering and Artificial Intelligence (AI) may seem like distinct fields at first glance, their symbiosis is undeniable. The foundation of any AI system is high-quality data. Here lies the critical role of data engineering: preparing and managing data to feed AI models.

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

Monte Carlo

JUNE 16, 2023

Data quality has broad applications across industries, but its importance and degree of quality required is also contextual to the use case. For example, in marketing, high-quality data can help businesses better understand their customers, allowing them to create more targeted and effective campaigns.

Data Integration 52

Data Integration Hospitality Banking Healthcare 52