Data Engineering Podcast

FEBRUARY 18, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Join in with the event for the global data community, Data Council Austin. Don't miss out on their only event this year!

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Snowflake

APRIL 17, 2024

In today’s data-driven world, developer productivity is essential for organizations to build effective and reliable products, accelerate time to value, and fuel ongoing innovation. We’re excited to share more innovations soon, making data even more accessible for all.

Data Pipeline 100

Data Pipeline Python Data Engineering Data Engineer 100

Precisely

DECEMBER 28, 2023

Enterprise technology is having a watershed moment; no longer do we access information once a week, or even once a day. This article explores what streaming data pipelines are, how they work, and how to build this data pipeline architecture. To build the streaming data pipeline, you’ll initialize the in-memory framework.

Data Pipeline 91

Data Pipeline Building Kafka Big Data 91

dbt Developer Hub

NOVEMBER 3, 2022

Let’s discuss how to convert events from an event-driven microservice architecture into relational tables in a warehouse like Snowflake. Quality problems lead to first responders unable to check into disaster sites or parents unable to access ESA funds. So our solution was to start using an intentional contract: Events.

Kafka 52

Kafka ETL Tools BI Database 52

Workfall

SEPTEMBER 26, 2023

Reading Time: 9 minutes In this blog, we will cover: What are Server-Sent Events? Why Stream Data Using Server-Sent Events (SSE)? Hands-On Conclusion What are Server-Sent Events? These events can carry information like notifications, updates, or alerts. This makes it accessible to a wider range of developers and projects.

Python 85

Python Bytes Coding Data 85

Data Engineering Podcast

AUGUST 10, 2020

Summary Event based data is a rich source of information for analytics, unless none of the event structures are consistent. The team at Iteratively are building a platform to manage the end to end flow of collaboration around what events are needed, how to structure the attributes, and how they are captured.

Data Collection 100

Data Collection Data Engineering Data Engineer Education 100

Towards Data Science

FEBRUARY 9, 2024

How to build a modern, scalable data platform to power your analytics and data science projects (updated) Table of Contents: What’s changed? Orchestration I mentioned modularity as a core concept of building a modern data platform in my 2021 article, but I failed to emphasize the importance of data orchestration.

Building 82

Building Transportation Data Lake Metadata 82

Pinterest Engineering

JANUARY 4, 2024

It is written in C++ and offers bindings for several programming languages, making it accessible for developers in different environments. In order to build a distributed and replicated service using RocksDB, we built a real time replicator library: Rocksplicator. RocksDB is a single node key value store.

Database 76

Database Building Datasets Relational Database 76

LinkedIn Engineering

AUGUST 3, 2023

As I worked closely with iOS, I wanted to learn everything I could about the UI and dedicated my time to coding files and researching how to build apps. While browsing one day, I found an opportunity at LinkedIn for a software engineer on the events team. At my previous company, I didn’t think much about accessibility.

Building 68

Building Software Engineer Software Engineering Computer Science 68

Snowflake

OCTOBER 2, 2023

To help customers overcome these challenges, RudderStack and Snowflake recently launched Profiles , a new product that allows every data team to build a customer 360 directly in their Snowflake Data Cloud environment. Now teams can leverage their existing data engineering tools and workflows to build their customer 360.

Cloud 96

Cloud Building Insurance Data Engineering 96

Snowflake

DECEMBER 18, 2023

Data clean rooms have emerged as the technology to meet this need, enabling interoperability where multiple parties can collaborate on and analyze sensitive data in a governed way without exposing direct access to the underlying data and business logic. Snowflake’s acquisition of Samooha is subject to customary closing conditions.

Cloud 112

Cloud Building Entertainment Government 112

Rockset

MARCH 18, 2024

For instance, GPT-4’s cutoff date was April 2023, so it would not be aware of any events or developments happening beyond that point of time. Building a real-time, contextual and trustworthy knowledge base for AI applications revolves around RAG pipelines. What are the challenges building RAG pipelines? What is RAG?

Cloud 64

Cloud Building Metadata Kafka 64

Data Engineering Podcast

NOVEMBER 26, 2023

Summary Building a data platform that is enjoyable and accessible for all of its end users is a substantial challenge. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Data Engineering Podcast

APRIL 23, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management RudderStack helps you build a customer data platform on your warehouse or data lake. What are the shifts that have made them more accessible to a wider variety of teams? RudderStack also supports real-time use cases.

Data Lake 277

Data Lake Kafka Machine Learning Data Warehouse 277

Snowflake

DECEMBER 19, 2023

But for many organizations, building this understanding is more akin to solving an ever-growing jigsaw puzzle (with no easy edge pieces!) It’s a potentially cumbersome and time-consuming process that too often requires moving or sharing access to sensitive customer data. than reading data insights from a beautiful dashboard.

Building 77

Building Pipeline-centric Database-centric Digital Media 77

Data Engineering Podcast

MAY 25, 2021

To eliminate this impedance mismatch Edo Liberty founded Pinecone to build database that works natively with vectors. In this episode he explains how this technology will allow teams to accelerate the speed of innovation, how vectors make it possible to build more advanced search functionality, and how Pinecone is architected.

Database 100

Database Building Data Warehouse Machine Learning 100

Data Engineering Podcast

FEBRUARY 25, 2024

Summary Building a database engine requires a substantial amount of engineering effort and time investment. Over the decades of research and development into building these software systems there are a number of common components that are shared across implementations. Don't miss out on our only event this year!

Database 162

Database Technology Data Lake High Quality Data 162

Data Engineering Podcast

MARCH 3, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Join us at the top event for the global data community, Data Council Austin. Don't miss out on our only event this year!

Programming 162

Programming Data Lake High Quality Data Data Pipeline 162

Data Engineering Podcast

MARCH 27, 2022

In this episode Balaji Ganesan shares how his experiences building and maintaining Ranger in previous roles helped him understand the needs of organizations and engineers as they define and evolve their data governance policies and practices. Can you describe what Privacera is and the story behind it?

Data Governance 130

Data Governance Government Cloud Building 130

Data Engineering Podcast

MARCH 17, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Join us at the top event for the global data community, Data Council Austin. Don't miss out on our only event this year!

Database 147

Database Data Lake High Quality Data Data Workflow 147

Data Engineering Podcast

MARCH 10, 2024

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Dagster offers a new approach to building and running data platforms and data pipelines. Join us at the top event for the global data community, Data Council Austin. Don't miss out on our only event this year!

Data Lake 147

Data Lake High Quality Data Data Pipeline Architecture 147

Snowflake

JUNE 28, 2023

External Network Access (PrPr) – Allows users to seamlessly connect to external endpoints from their Snowpark code (UDFs/UDTFs and Stored procedures) while maintaining high security and governance. This is great for building Snowpark apps/integration that requires execution of Snowpark codes that does not have to be persisted.

Python 52

Python Accessible Accessibility Pipeline-centric 52

Data Engineering Podcast

DECEMBER 24, 2023

Summary Kafka has become a ubiquitous technology, offering a simple method for coordinating events and data across different systems. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.

Kafka 245

Kafka Data Lake High Quality Data SQL 245

Monte Carlo

APRIL 24, 2023

In an evolving data landscape, the explosion of new tooling solutions—from cloud-based transforms to data observability —has made the question of “build versus buy” increasingly important for data leaders. Check out Part 1 of the build vs buy guide to catch up. But answering the build versus buy question isn’t easy.

Data Pipeline 52

Data Pipeline Building Data Ingestion BI 52

Workfall

DECEMBER 19, 2023

By signing in, users gain access to an enhanced dashboard, opening the door to a world of suggested apps, including the “Haiku Creator,” “Weird Tour Guide,” and “Podcast Generator,” each providing a hands-on opportunity to explore PartyRock’s versatile capabilities. Let’s dive in!

Building 81

Building AWS Accessible Accessibility 81

Rockset

OCTOBER 4, 2022

Event streams, such as clickstreams, IoT data and other time series data, are common sources of data into these apps. The broad adoption of Apache Kafka has helped make these event streams more accessible. In this post, we compare two options for real-time analytics on event and CDC streams: Rockset and ClickHouse.

MySQL 52

MySQL Kafka Aggregated Data Architecture 52

Data Engineering Podcast

AUGUST 26, 2019

Summary Data engineers are responsible for building tools and platforms to power the workflows of other members of the business. Each group of users has their own set of requirements for the way that they access and interact with those platforms depending on the insights they are trying to gather.

Building 100

Building Data Analytics Media Data Engineering 100

Knowledge Hut

MAY 3, 2024

Kafka offers better fault tolerance because of its event-driven processing. Processing Type Kafka analyses events as they often take place. Stream processing is highly beneficial if the events you wish to track are happening frequently and close together in time. A continuous processing model is an outcome.

Kafka 98

Kafka Scala Java Amazon Web Services 98

Netflix Tech

SEPTEMBER 28, 2021

The Revenue and Growth Tools (RGT) team decided to set off on a journey to build tools in an abstract manner to have solutions readily available within our organization. The traditional approach of bundling dependencies at build time lacks the flexibility we need to empower our stakeholders.

Building 96

Building Metadata Designing Coding 96

Data Engineering Podcast

FEBRUARY 3, 2020

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. Why might an organization want to build one?

Building 100

Building Media AWS Data 100

Knowledge Hut

MAY 2, 2024

Whatever your goal is, whether it’s to be a Data Scientist, Data Analyst, or Machine Learning Engineer, your primary area of focus should be on “Mathematics” Math is the basic building block to solve all the Business and Data driven applications in the real-world scenario.

Machine Learning 98

Machine Learning Algorithm Datasets Deep Learning 98

Data Engineering Podcast

JULY 27, 2020

In this episode he shares his approach to testing complex systems, the common challenges that are faced by engineers who build them, and why it is important to understand their limitations. With their open-source foundation, fixed pricing, and unlimited volume, they are enterprise ready, but accessible to everyone.

Systems 100

Systems Building Scala Java 100

Snowflake

AUGUST 8, 2023

In 2022, the company realized that in order to monitor, analyze, counter, and prevent security threats, it needed both a modern data lake as well as a security information and event management (SIEM) solution. Account takeover: A third party gains access to an account and uses it for fraud, cyberattacks, and other malicious activities.

Data Lake 52

Data Lake Programming IT Building 52

Rockset

FEBRUARY 24, 2022

Event-based architectures have been gaining popularity for some time. With increased adoption has come a flood of options for aggregating and analyzing events. Which databases are optimized for ingesting streaming events and analyzing them in real time? Supporting separation of data access (read/write) mechanisms like CQRS.

AWS 52

AWS Amazon Web Services Kafka SQL 52

Data Engineering Podcast

SEPTEMBER 2, 2019

In order to stay current we need access to deeply technical presentations that aren’t burdened by extraneous marketing. In this episode Pete discusses his motivation for starting these events, how they serve to bring the data community together, and the observations that he has made about the direction that we are moving.

Building 100

Building Data Database Design Media 100

LinkedIn Engineering

DECEMBER 20, 2023

At LinkedIn, trust is the cornerstone for building meaningful connections and professional relationships. Let’s look into the critical modules that are needed to build this type of system. This includes taking measures such as issuing warnings, restricting access, or suspending accounts as necessary. Espresso , Venice , Rest.li

Building 91

Building Algorithm Kafka Machine Learning 91

The Pragmatic Engineer

APRIL 11, 2024

It’s common enough for banks to contract with development agencies to build and maintain their services. There are still ongoing slowdowns in bank transfers, payments with debit and prepaid cards and in accessing and using online services, including Internet Banking, Smart Business Sella and apps.

Banking 210

Banking Utilities Database Engineering 210