Data Engineering Digest

How Data Engineering Teams Power Machine Learning With Feature Platforms

Data Engineering Podcast

JULY 2, 2023

Summary Feature engineering is a crucial aspect of the machine learning workflow. In this episode Razi Raziuddin shares how data engineering teams can support the machine learning workflow through the development and support of systems that empower data scientists and ML engineers to build and maintain their own features.

Machine Learning

Machine Learning Data Engineering Data Engineer Engineering

Platform Engineering: Predictions and Prospects in 2023 & Beyond

Workfall

FEBRUARY 14, 2023

Reading Time: 7 minutes Platform Engineering has received a lot of attention, but there is some misunderstanding about what it is and, perhaps more importantly, how it differs from more well-known disciplines like SRE and DevOps. Platform Engineering is the rebranded DevOps or it is the next stage of DevOps evolution?

Engineering

Engineering Software Engineer Software Engineering AWS

Qdrant: Open-Source Vector Search Engine with Managed Cloud Platform

KDnuggets

FEBRUARY 9, 2023

Qdrant open-source vector similarity search engine is now available in the cloud. The cloud platform for Qdrant offers business users cost-efficient, fully managed service in addition to powerful features of their open-source vector search database.

Cloud

Cloud Engineering Management Database

Webinars

Peak Performance: Continuous Testing & Evaluation of LLM-Based Applications

From Developer Experience to Product Experience: How a Shared Focus Fuels Product Success

Understanding User Needs and Satisfying Them

Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You Need to Know

How To Get Promoted In Product Management

MORE WEBINARS

Life Happens in Real Time, Not in Batches: Choosing a Data Streaming Platform and Stream Processing Engine

Confluent

OCTOBER 12, 2023

Learn about the key capabilities of a data streaming platform and what factors to consider when choosing a stream processing engine like Apache Flink® to fuel use cases with real-time data.

Process

Process Engineering Data

New Study: 2018 State of Embedded Analytics Report

Why do some embedded analytics projects succeed while others fail? We surveyed 500+ application teams embedding analytics to find out which analytics features actually move the needle. Read the 6th annual State of Embedded Analytics Report to discover new best practices. Brought to you by Logi Analytics.

Project

What Is A DataOps Engineer? Responsibilities + How A DataOps Platform Facilitates The Role

Meltano

OCTOBER 5, 2022

DataOps engineers. Get Started With DataOps In this guide, we’ll explain all you need to know about a DataOps engineer, including their key responsibilities, and how these experts use a DataOps platform, like Meltano , to optimize internal processes. What Is A DataOps Engineer? Designing data engineering assets.

Engineering

Engineering Raw Data Data Pipeline ETL Tools

Employee Spotlight: Dipayan Bhowmick, Director of Engineering, Platform and Data Observability

Acceldata

DECEMBER 5, 2022

Meet Dipayan Bhowmich, Director of Engineering at Acceldata focusing on Platform and Data Observability.

Engineering

Engineering Data

Meet Magpie: The End-to-End Data Engineering Platform (VIDEO)

Silectis

DECEMBER 15, 2020

If you’ve been following along with Silectis over the past couple of years, you are familiar with our data engineering platform, Magpie. You’re aware of the many outcomes it puts at the fingertips of data engineers, and teams of data practitioners more largely. If you’re new around here, not to worry. Ready to work smarter?

Data Engineering

Data Engineering Data Engineer Engineering ETL Tools

Data Lineage Now Available with Silectis Magpie Data Engineering Platform

Silectis

FEBRUARY 3, 2021

We’re excited to share that Silectis has released a new suite of automated data lineage features within Magpie, the end-to-end data engineering platform. METADATA OBJECT LINEAGE Metadata objects in the Magpie platform include things like data sources, schemas, tables, and jobs.

Data Engineering

Data Engineering Data Engineer Engineering Data Lake

Monetizing Analytics Features: Why Data Visualizations Will Never Be Enough

Think your customers will pay more for data visualizations in your application? Five years ago they may have. But today, dashboards and visualizations have become table stakes. Discover which features will differentiate your application and maximize the ROI of your embedded analytics. Brought to you by Logi Analytics.

Data

Using Trino And Iceberg As The Foundation Of Your Data Lakehouse

Data Engineering Podcast

FEBRUARY 18, 2024

In this episode Dain Sundstrom, CTO of Starburst, explains how the combination of the Trino query engine and the Iceberg table format offer the ease of use and execution speed of data warehouses with the infinite storage and scalability of data lakes. Go to dataengineeringpodcast.com/dagster today to get started. Your first 30 days are free!

Data Lake

Data Lake High Quality Data Data Warehouse Google Cloud

Making Email Better With AI At Shortwave

Data Engineering Podcast

APRIL 21, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Can you describe how you have architected the Shortwave platform? Go to dataengineeringpodcast.com/dagster today to get started.

Data Lake

Data Lake High Quality Data Data Pipeline Machine Learning

Barking Up The Wrong GPTree: Building Better AI With A Cognitive Approach

Data Engineering Podcast

MAY 5, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Go to dataengineeringpodcast.com/dagster today to get started. Your first 30 days are free! Data lakes are notoriously complex.

Building

Building Data Lake High Quality Data Machine Learning

Shutting Down My Job Board for Software Engineering Positions After 2.5 Years

The Pragmatic Engineer

APRIL 9, 2024

I started The Pragmatic Engineer Job Board in October 2021. Context My initial motivation with the job board was to add a kind of "helpful monetization" to both my blog, and The Pragmatic Engineer Newsletter. On top of the ability to post jobs, they had access to The Pragmatic Engineer Talent Collective.

Software Engineer

Software Engineer Software Engineering Engineering Recruitment

Data News — Week 24.11

Christophe Blefari

MARCH 15, 2024

Databricks invests in Mistral AI — Mistral successfully positioned as the main OpenAI rival by being integrated in all major data platforms (Azure, Snowflake previously). Starting today it will be supported by 3 majors platforms: Kaggle, HuggingFace and OpenML. I'm speechless. This is Croissant.

Metadata

Metadata Datasets Data Data Warehouse

Data Engineering Best Practices: How LinkedIn Scales Its Analytical Data Platform to One Exabyte and Beyond

Acceldata

FEBRUARY 9, 2023

Learn how LinkedIn uses data engineering best practices to scale its data. platform.

Data Engineering

Data Engineering Data Engineer Engineering IT

Establish A Single Source Of Truth For Your Data Consumers With A Semantic Layer

Data Engineering Podcast

APRIL 7, 2024

Summary Maintaining a single source of truth for your data is the biggest challenge in data engineering. In this episode Artyom Keydunov, creator of Cube, discusses the evolution and applications of the semantic layer as a component of your data platform, and how Cube provides speed and cost optimization for your data consumers.

Data Lake

Data Lake High Quality Data BI Data Workflow

Designing A Non-Relational Database Engine

Data Engineering Podcast

APRIL 14, 2024

The default association with the term "database" is relational engines, but non-relational engines are also used quite widely. In this episode Oren Eini, CEO and creator of RavenDB, explores the nuances of relational vs. non-relational engines, and the strategies for designing a non-relational database.

Non-relational Database

Non-relational Database Relational Database Database Designing

Build Your Second Brain One Piece At A Time

Data Engineering Podcast

APRIL 28, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Support Data Engineering Podcast Summary Generative AI promises to accelerate the productivity of human collaborators.

Building

Building Data Lake High Quality Data Machine Learning

The “10x engineer:" 50 years ago and now

The Pragmatic Engineer

MARCH 12, 2024

👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. CI/CD : running automated tests on all changes, and deploying code to production automatically.

Engineering

Engineering Programming Language Hospitality Programming

Ship Smarter Not Harder With Declarative And Collaborative Data Orchestration On Dagster+

Data Engineering Podcast

MARCH 24, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. What are the notable enhancements beyond the Dagster Core project that this updated platform provides? Your first 30 days are free!

Data Lake

Data Lake High Quality Data Hadoop Data Pipeline

Find Out About The Technology Behind The Latest PFAD In Analytical Database Development

Data Engineering Podcast

FEBRUARY 25, 2024

Summary Building a database engine requires a substantial amount of engineering effort and time investment. When Paul Dix decided to re-write the InfluxDB engine he found the Apache Arrow ecosystem ready and waiting with useful building blocks to accelerate the process. Your first 30 days are free!

Database

Database Technology Data Lake High Quality Data

Data Engineering: How Spotify Upgraded its Data Orchestration Platform

Acceldata

DECEMBER 5, 2022

PlatformsLearn how Spotify leverages data orchestration to run 20,000 batch data pipelines defined in more than 1,000 repositories every day.

Data Engineering

Data Engineering Data Engineer IT Engineering

Tackling Real Time Streaming Data With SQL Using RisingWave

Data Engineering Podcast

FEBRUARY 4, 2024

RisingWave is a database engine that was created specifically for stream processing, with S3 as the storage layer. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Can you describe what RisingWave is and the story behind it?

SQL

SQL Data Lake High Quality Data Data Pipeline

Platform Engineering and Third Generation Microservices in Dublin

Zalando Engineering

MAY 25, 2017

There’s ongoing collaboration with our colleagues in Germany and Finland - recently, over twenty data science researchers and engineers from Dublin attended a two-day internal conference in Berlin. The part we talk about less is building more and more critical elements of the Zalando Platform. Our platform work started small.

Engineering

Engineering Data Science Machine Learning Software Engineer

Adding Anomaly Detection And Observability To Your dbt Projects Is Elementary

Data Engineering Podcast

MARCH 31, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Trusted by teams of all sizes, including Comcast and Doordash, Starburst is a data lake analytics platform that delivers the adaptability and flexibility a lakehouse ecosystem promises.

Project

Project Data Lake High Quality Data Data Workflow

Reconciling The Data In Your Databases With Datafold

Data Engineering Podcast

MARCH 17, 2024

Summary A significant portion of data workflows involve storing and processing information in database engines. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines.

Database

Database Data Lake High Quality Data Data Workflow

Data News — Week 24.14

Christophe Blefari

APRIL 5, 2024

Looks cool to be able to switch Polars engines like this. Survey about query engines used by companies — Data Council happened recently. A talker did a survey during his keynote about the query engines used by the audience and Spark is still leading to BigQuery/Snowflake/Athena. When is the data model finished?

SQL

SQL Data Python Datasets

When And How To Conduct An AI Program

Data Engineering Podcast

MARCH 3, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Go to dataengineeringpodcast.com/dagster today to get started. Your first 30 days are free! Data lakes are notoriously complex.

Programming

Programming Data Lake High Quality Data Data Pipeline

Data Sharing Across Business And Platform Boundaries

Data Engineering Podcast

FEBRUARY 11, 2024

There are also numerous technical considerations to be made, particularly if the producer and consumer of the data aren't using the same platforms. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

Data Lake

Data Lake High Quality Data Government Data Pipeline

Version Your Data Lakehouse Like Your Software With Nessie

Data Engineering Podcast

MARCH 10, 2024

The primary purpose of the catalog is to inform the query engine of what data exists and where, but the Nessie project aims to go beyond that simple utility. Trusted by teams of all sizes, including Comcast and Doordash, Starburst is a data lake analytics platform that delivers the adaptability and flexibility a lakehouse ecosystem promises.

Data Lake

Data Lake High Quality Data Data Pipeline Architecture

Twitter vs Instagram Threads: two different approaches to throttling

The Pragmatic Engineer

JULY 6, 2023

Originally published 6 July 2023 👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. If you’re not yet a full subscriber, you missed this week’s deep-dive on What a senior engineer is at Big Tech. We cover one out of six topics in today’s subscriber-only The Scoop issue.

Google Cloud

Google Cloud Media Cloud Utilities

Going from Developer to CEO: Chronosphere

The Pragmatic Engineer

OCTOBER 10, 2023

👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. We covered more on this topic in the article How Uber built its observability platform.

Software Engineer

Software Engineer Software Engineering Architecture Media

Realtime Data Applications Made Easier With Meroxa

Data Engineering Podcast

APRIL 23, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake. How have the focus and goals of the platform and company evolved over the past 2 years? Who are the target customers for Meroxa?

Data Lake

Data Lake Kafka Machine Learning Data Warehouse

Troubleshooting Kafka In Production

Data Engineering Podcast

DECEMBER 24, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. Trusted by teams of all sizes, including Comcast and Doordash, Starburst is a data lake analytics platform that delivers the adaptability and flexibility a lakehouse ecosystem promises.

Kafka

Kafka Data Lake High Quality Data SQL

Addressing The Challenges Of Component Integration In Data Platform Architectures

Data Engineering Podcast

NOVEMBER 26, 2023

Summary Building a data platform that is enjoyable and accessible for all of its end users is a substantial challenge. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. With Materialize, you can! Data lakes are notoriously complex.

Architecture

Architecture Data Lake High Quality Data SQL

Modern Customer Data Platform Principles

Data Engineering Podcast

JANUARY 21, 2024

In this episode Tasso Argyros, CEO of ActionIQ, gives a summary of the major epochs in database technologies and how he is applying the capabilities of cloud data warehouses to the challenge of building more comprehensive experiences for end-users through a modern customer data platform (CDP).

Data Lake

Data Lake High Quality Data NoSQL Data Warehouse

Is the “AI developer”a threat to jobs – or a marketing stunt?

The Pragmatic Engineer

MAY 1, 2024

This article was published on 14 March 2024 in The Pragmatic Engineer, for subscribers. Subscribe to The Pragmatic Engineer to stay up-to-date on what is happening with software engineering, Big Tech, and startups. And to get timely analysis like this into your inbox, early. I suspect the media campaign is overreaching, though.

Software Engineer

Software Engineer Software Engineering Programming Language Media

Build A Data Lake For Your Security Logs With Scanner

Data Engineering Podcast

JANUARY 28, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex. Trusted by teams of all sizes, including Comcast and Doordash, Starburst is a data lake analytics platform that delivers the adaptability and flexibility a lakehouse ecosystem promises.

Data Lake

Data Lake Building High Quality Data AWS

An Overview Of The Sate Of Data Orchestration In An Increasingly Complex Data Ecosystem

Data Engineering Podcast

SEPTEMBER 10, 2023

This offers a single location for managing visibility and error handling so that data platform engineers can manage complexity. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. With Materialize, you can!

BI

BI SQL Data Machine Learning

Is the “AI developer”a threat to jobs – or a marketing stunt?

The Pragmatic Engineer

MARCH 19, 2024

👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover topics related to Big Tech and startups through the lens of engineering managers and senior engineers. In this article, we cover one out of three topics from last week’s subscriber-only The Pulse issue.

Software Engineer

Software Engineer Software Engineering Programming Language Media

Datadog’s $65M/year customer mystery solved

The Pragmatic Engineer

MAY 11, 2023

👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. I wanted to find the truth, so I tracked down software engineers at the company. I can confirm this, having talked with both current and former software engineers at Coinbase who shared details of what happened. And I got my answer: Yes.

AWS

AWS Software Engineer Software Engineering Google Cloud

What Data Engineers Really Do?

Analytics Vidhya

JUNE 25, 2023

In a data-driven world, behind-the-scenes heroes like data engineers play a crucial role in ensuring smooth data flow. A data engineer investigates the issue, identifies a glitch in the e-commerce platform’s data funnel, and swiftly implements seamless data pipelines. appeared first on Analytics Vidhya.

Data Engineering

Data Engineering Data Engineer Engineering Data Pipeline

Pushing The Limits Of Scalability And User Experience For Data Processing WIth Jignesh Patel

Data Engineering Podcast

JANUARY 7, 2024

Unfortunately, the volumes of data that are being generated continue to double, requiring further advancements in the platform capabilities to keep up. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data lakes are notoriously complex.

Data Process

Data Process Process Data Lake High Quality Data

How Data Engineering Teams Power Machine Learning With Feature Platforms

Platform Engineering: Predictions and Prospects in 2023 & Beyond

Webinars

Trending Sources

Qdrant: Open-Source Vector Search Engine with Managed Cloud Platform

Webinars

Life Happens in Real Time, Not in Batches: Choosing a Data Streaming Platform and Stream Processing Engine

New Study: 2018 State of Embedded Analytics Report

What Is A DataOps Engineer? Responsibilities + How A DataOps Platform Facilitates The Role

Employee Spotlight: Dipayan Bhowmick, Director of Engineering, Platform and Data Observability

Meet Magpie: The End-to-End Data Engineering Platform (VIDEO)

Data Lineage Now Available with Silectis Magpie Data Engineering Platform

Monetizing Analytics Features: Why Data Visualizations Will Never Be Enough

Using Trino And Iceberg As The Foundation Of Your Data Lakehouse

Making Email Better With AI At Shortwave

Barking Up The Wrong GPTree: Building Better AI With A Cognitive Approach

Shutting Down My Job Board for Software Engineering Positions After 2.5 Years

Data News — Week 24.11

Data Engineering Best Practices: How LinkedIn Scales Its Analytical Data Platform to One Exabyte and Beyond

Establish A Single Source Of Truth For Your Data Consumers With A Semantic Layer

Designing A Non-Relational Database Engine

Build Your Second Brain One Piece At A Time

The “10x engineer:" 50 years ago and now

Ship Smarter Not Harder With Declarative And Collaborative Data Orchestration On Dagster+

Find Out About The Technology Behind The Latest PFAD In Analytical Database Development

Data Engineering: How Spotify Upgraded its Data Orchestration Platform

Tackling Real Time Streaming Data With SQL Using RisingWave

Platform Engineering and Third Generation Microservices in Dublin

Adding Anomaly Detection And Observability To Your dbt Projects Is Elementary

Reconciling The Data In Your Databases With Datafold

Data News — Week 24.14

When And How To Conduct An AI Program

Data Sharing Across Business And Platform Boundaries

Version Your Data Lakehouse Like Your Software With Nessie

Twitter vs Instagram Threads: two different approaches to throttling

Going from Developer to CEO: Chronosphere

Realtime Data Applications Made Easier With Meroxa

Troubleshooting Kafka In Production

Addressing The Challenges Of Component Integration In Data Platform Architectures

Modern Customer Data Platform Principles

Is the “AI developer”a threat to jobs – or a marketing stunt?

Build A Data Lake For Your Security Logs With Scanner

An Overview Of The Sate Of Data Orchestration In An Increasingly Complex Data Ecosystem

Is the “AI developer”a threat to jobs – or a marketing stunt?

Datadog’s $65M/year customer mystery solved

What Data Engineers Really Do?

Pushing The Limits Of Scalability And User Experience For Data Processing WIth Jignesh Patel

Stay Connected