Data Engineering Podcast

FEBRUARY 4, 2024

Summary Stream processing systems have long been built with a code-first design, adding SQL as a layer on top of the existing framework. RisingWave is a database engine that was created specifically for stream processing, with S3 as the storage layer. How have the design and goals/scope changed since you first started working on it?

SQL 173

SQL Data Lake High Quality Data Data Pipeline 173

Data Engineering Podcast

DECEMBER 24, 2023

Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. With Materialize, you can!

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Data Engineering Podcast

MARCH 31, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Datafold has recently launched data replication testing, providing ongoing validation for source-to-target replication. How is Elementary designed/implemented?

Project 130

Project Data Lake High Quality Data Data Workflow 130

Monte Carlo

NOVEMBER 8, 2023

Kafka and Vector Database support According to Databricks’ State of Data and AI report , the number of companies using SaaS LLM APIs has grown more than 1300% since November 2022 with a nearly 411% increase in the number of AI models put into production during that same period. Both integrations will be available early 2024.

Kafka 64

Kafka Database High Quality Data Datasets 64

Data Engineering Podcast

DECEMBER 17, 2023

Summary The "modern data stack" promised a scalable, composable data platform that gave everyone the flexibility to use the best tools for every job. Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data.

Data Lake 130

Data Lake High Quality Data SQL Architecture 130

Data Engineering Podcast

NOVEMBER 26, 2023

Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. With Materialize, you can!

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Data Engineering Podcast

NOVEMBER 5, 2023

Summary Databases are the core of most applications, but they are often treated as inscrutable black boxes. When an application is slow, there is a good probability that the database needs some attention. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products.

PostgreSQL 147

PostgreSQL Data Lake High Quality Data SQL 147

Data Engineering Podcast

DECEMBER 10, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management You shouldn't have to throw away the database to build with fast-changing data. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products.

Data Lake 130

Data Lake High Quality Data SQL Architecture 130

Data Engineering Podcast

NOVEMBER 19, 2023

Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack You shouldn't have to throw away the database to build with fast-changing data. It’s the only true SQL streaming database built from the ground up to meet the needs of modern data products. With Materialize, you can!

Project 147

Project Data Lake High Quality Data SQL 147

Data Engineering Podcast

NOVEMBER 12, 2023

If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration. Learn more about Datafold by visiting dataengineeringpodcast.com/datafold Data lakes are notoriously complex. You shouldn't have to throw away the database to build with fast-changing data.

Software Engineer 147

Software Engineer Software Engineering Engineering Data Lake 147

Monte Carlo

FEBRUARY 27, 2024

Organizations need to connect LLMs with their proprietary data and business context to actually create value for their customers and employees. They need robust data pipelines, high-quality data, well-guarded privacy, and cost-effective scalability. Data engineers. Who can deliver?

Data Engineering 59

Data Engineering Data Engineer Engineering High Quality Data 59

Data Engineering Weekly

MAY 19, 2024

Data value depends on the users and the use cases Data quality is multi-dimensional, and high-quality data costs more. An Empirical Evaluation of Columnar Storage Formats paper compares ORC and Parquet formats regarding efficiency, design choices, and what is lacking in that system.

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Monte Carlo

MARCH 7, 2024

Why prompt engineering isn’t all that and a bag of SQL queries Understand vector databases Create AI differentiation with RAG Find and solve real business problems High-quality data always lives up to the hype What is prompt engineering? Prompt engineering is a strategic way to communicate and interact with an LLM.

Engineering 59

Engineering High Quality Data Database Architecture 59

Databand.ai

MAY 30, 2023

Data quality refers to the degree of accuracy, consistency, completeness, reliability, and relevance of the data collected, stored, and used within an organization or a specific context. High-quality data is essential for making well-informed decisions, performing accurate analyses, and developing effective strategies.

Data Cleanse 85

Data Cleanse Datasets Data Governance Data Validation 85

U-Next

FEBRUARY 27, 2023

Data normalization is the process of organizing and transforming data to improve its structural integrity, accuracy, and consistency. Data normalization is also an important part of database design. Data normalization is adopted because it helps to ensure that data will be consistent.

IT 98

IT Bytes Database Recruitment 98

U-Next

MARCH 7, 2023

Data normalization is the process of organizing and transforming data to improve its structural integrity, accuracy, and consistency. Data normalization is also an important part of database design. Data normalization is adopted because it helps to ensure that data will be consistent.

IT 98

IT Bytes Database Recruitment 98

Ripple Engineering

JANUARY 29, 2024

For Ripple's product capabilities, the Payments team of Ripple, for example, ingests millions of transactional records into databases and performs analytics to generate invoices, reports, and other related payment operations.    A lack of a centralized system makes building a single source of high-quality data difficult.

Database-centric 52

Database-centric Pipeline-centric NoSQL High Quality Data 52

Data Engineering Weekly

MARCH 3, 2024

Here is the agenda, 1) Data Application Lifecycle Management - Harish Kumar( Paypal) Hear from the team in PayPal on how they build the data product lifecycle management (DPLM) systems. 3) DataOPS at AstraZeneca The AstraZeneca team talks about data ops best practices internally established and what worked and what didn’t work!!!

Data Engineering 67

Data Engineering Data Engineer Pipeline-centric Engineering 67

Precisely

MARCH 14, 2023

Read Quality data you can depend on – today, tomorrow, and beyond For many years Precisely customers have ensured the accuracy of data across their organizations by leveraging our leading data solutions including Trillium Quality, Spectrum Quality, and Data360 DQ+. What does all this mean for your business?

Data Integration 52

Data Integration High Quality Data BI Data 52

Knowledge Hut

MARCH 5, 2024

Let us look at some of the functions of Data Engineers: They formulate data flows and pipelines Data Engineers create structures and storage databases to store the accumulated data, which requires them to be adept at core technical skills, like design, scripting, automation, programming, big data tools , etc.

Data Engineering 52

Data Engineering Data Engineer Engineering Education 52

Towards Data Science

MARCH 29, 2023

Great reads on modeling, processes, and leadership Photo by Emil Widlund on Unsplash At the very start of my journey in data, I thought I was going to be a data scientist, and my first foray into data was centered on studying statistics and linear algebra, not software engineering or database management.

Data Warehouse 66

Data Warehouse BI Healthcare Database 66

Monte Carlo

JANUARY 10, 2024

Extrinsic data, meanwhile, is more about the context — it’s how your data interacts with the world outside and how it fits into the larger picture of your project or organization. Consider a database that holds customer details. usability) would be about extrinsic data quality.

Data Cleanse 52

Data Cleanse Data Engineering Data Engineer Engineering 52

Data Engineering Podcast

JUNE 29, 2020

This was a great conversation about the complexities of working in a niche domain of data analysis and how to build a pipeline of high quality data from collection to analysis. Can you describe how your overall data management system is architected?

Data Collection 100

Data Collection Management High Quality Data Metadata 100

Monte Carlo

FEBRUARY 20, 2024

Fivetran is the leader in the data ingestion space, known for its ease of use and extensive connector ecosystem. Integrations : They offer a wide array of connectors for databases, SaaS applications, cloud storage solutions, and more, covering both popular and niche data sources. Fivetran Image courtesy of Fivetran.

Data Ingestion 52

Data Ingestion Google Cloud Kafka AWS 52

Knowledge Hut

JANUARY 3, 2024

Data duplication could occur in a large database defined as a single relation. This data repetition could lead to the following: Expanding relationships greatly. Maintaining and updating data is difficult since it necessitates searching through numerous related entries. A few guidelines govern database normalization.

MongoDB 52

MongoDB Database Coding Utilities 52

U-Next

SEPTEMBER 7, 2022

Employee database software is quickly becoming a must-have for HR and people managers. A firm can benefit immensely from maintaining accurate and clean employee data, as it is still a difficult task. . A firm can benefit immensely from maintaining accurate and clean employee data, as it is still a difficult task. .

Data Management 52

Data Management Management Electronics Unstructured Data 52

Databand.ai

AUGUST 30, 2023

Data cleansing: Implement corrective measures to address identified issues and improve dataset accuracy levels. Data validation: Ensure new database entries adhere to predefined rules or standards to maintain dataset consistency. Additionally, high-quality data reduces costly errors stemming from inaccurate information.

Building 52

Building Data Cleanse Data Governance Datasets 52

Monte Carlo

JULY 27, 2023

These specialists are also commonly referred to as data reliability engineers. To be successful in their role, data quality engineers will need to gather data quality requirements (mentioned in 65% of job postings) from relevant stakeholders. Strong analytical and technical skills to address sophisticated issues.

Engineering 52

Engineering Healthcare Scala Data Warehouse 52

Ascend.io

FEBRUARY 28, 2024

While data engineering and Artificial Intelligence (AI) may seem like distinct fields at first glance, their symbiosis is undeniable. The foundation of any AI system is high-quality data. Here lies the critical role of data engineering: preparing and managing data to feed AI models.

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

DataKitchen

JULY 27, 2023

Azure Databricks Delta Live Table s: These provide a more straightforward way to build and manage Data Pipelines for the latest, high-quality data in Delta Lake. Multi-Tool Patterns Indeed, designing patterns involving data pipelines often involves using multiple tools in conjunction, each with its strengths.

Data Pipeline 52

Data Pipeline BI Machine Learning Data Preparation 52

Databand.ai

JULY 5, 2023

Metric collection and visualization: Gathering metrics from various sources, such as logs, APIs, and databases, is crucial for effectively monitoring system performance. Notable Data Observability Tools Here are a few notable data observability tools.

Data Pipeline 52

Data Pipeline Data Lake Data Warehouse Datasets 52

Monte Carlo

JUNE 16, 2023

Data quality has broad applications across industries, but its importance and degree of quality required is also contextual to the use case. For example, in marketing, high-quality data can help businesses better understand their customers, allowing them to create more targeted and effective campaigns.

Data Integration 52

Data Integration Hospitality Banking Healthcare 52

Knowledge Hut

JUNE 28, 2023

Reporting, querying, and analyzing structured data to generate actionable insights. Data Sources Diverse and vast data sources, including structured, unstructured, and semi-structured data. Structured data from databases, data warehouses, and operational systems.

Data Mining 52

Data Mining Business Intelligence BI Datasets 52

DataKitchen

NOVEMBER 4, 2021

It’s too hard to change our IT data product. Can we create high-quality data in an “answer-ready” format that can address many scenarios, all with minimal keyboarding? . “I I get cut off at the knees from a data perspective, and I am getting handed a sandwich of sorts and not a good one!”.

Process 98

Process Data Process Pharmaceutical Data Lake 98

Monte Carlo

AUGUST 30, 2023

When you run dbt test, dbt will execute these SQL tests against your models in your transformed database. You will also need to securely store and provide dbt with the necessary credentials to access your target database. With pipelines, the environment starts clean every time. Run your dbt models and tests.

SQL 52

SQL Datasets Database High Quality Data 52

Towards Data Science

JUNE 16, 2023

In the following examples, we’ll be using Looker, but most modern BI tools enable usage-based reporting in some form (Lightdash also has built in Usage Analytics , Tableau Cloud offers Admin Insights , and Mode’s Discovery Database offers access to usage data, just to name a few). Here are some processes to build in quality by design.

BI 76

BI Data ETL Tools Machine Learning 76

Maxime Beauchemin

JANUARY 20, 2017

Data modeling is changing Typical data modeling techniques — like the star schema  — which defined our approach to data modeling for the analytics workloads typically associated with data warehouses, are less relevant than they once were. Those systems have been taught to normalize the data for storage on their own.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130