Data Engineering Podcast

JUNE 30, 2024

How does the focus on data assets/data products shift your approach to observability as compared to a table/pipeline centric approach? How does the focus on data assets/data products shift your approach to observability as compared to a table/pipeline centric approach?

Pipeline-centric 147

Pipeline-centric Engineering Data Lake High Quality Data 147

Snowflake

APRIL 17, 2024

This traditional SQL-centric approach often challenged data engineers working in a Python environment, requiring context-switching and limiting the full potential of Python’s rich libraries and frameworks. The post Snowflake’s New Python API Empowers Data Engineers to Build Modern Data Pipelines with Ease appeared first on Snowflake.

Data Pipeline 124

Data Pipeline Python Data Engineering Data Engineer 124

DataKitchen

NOVEMBER 5, 2024

We have also seen a fourth layer, the Platinum layer , in companies’ proposals that extend the Data pipeline to OneLake and Microsoft Fabric. The need to copy data across layers, manage different schemas, and address data latency issues can complicate data pipelines. However, this architecture is not without its challenges.

Architecture 78

Architecture Raw Data Pipeline-centric Data Ingestion 78

Data Engineering Weekly

NOVEMBER 3, 2024

The blog emphasizes the importance of starting with a clear client focus to avoid over-engineering and ensure user-centric development. impactdatasummit.com Thumbtack: What we learned building an ML infrastructure team at Thumbtack Thumbtack shares valuable insights from building its ML infrastructure team.

Data Engineering 64

Data Engineering Data Engineer Engineering Pipeline-centric 64

Cloudera

JULY 13, 2021

CDP Data Engineering offers an all-inclusive toolset that enables data pipeline orchestration, automation, advanced monitoring, visual profiling, and a comprehensive management toolset for streamlining ETL processes and making complex data actionable across your analytic teams. . A key aspect of ETL or ELT pipelines is automation.

Data Engineering 127

Data Engineering Data Engineer Engineering Pipeline-centric 127

Snowflake

APRIL 8, 2024

The list of Top 10 semi-finalists is a perfect example: we have use cases for cybersecurity, gen AI, food safety, restaurant chain pricing, quantitative trading analytics, geospatial data, sales pipeline measurement, marketing tech and healthcare. Our sincere thanks go out to everyone who participated in this year’s competition.

Pipeline-centric 131

Pipeline-centric Food Healthcare Unstructured Data 131

Scott Logic

JUNE 19, 2025

While purists correctly note that Medallion isn’t truly an “architecture,” this layered approach provides a practical pattern for managing data lifecycle and quality across complex pipelines.

Data Lake 59

Data Lake Pipeline-centric Raw Data Architecture 59

Data Engineering Weekly

MARCH 30, 2025

One thing that stands out to me is As AI-driven data workflows increase in scale and become more complex, modern data stack tools such as drag-and-drop ETL solutions are too brittle, expensive, and inefficient for dealing with the higher volume and scale of pipeline and orchestration approaches.

Data Engineering 59

Data Engineering Data Engineer Engineering Pipeline-centric 59

Edureka

APRIL 22, 2025

Ideal for: Business-centric workflows involving fabric Snowflake = environments with a lot of developers and data engineers 2. Ideal for: Fabric: Microsoft-centric organizations Snowflake: Multi-cloud flexibility seekers 3. Cloud support Microsoft Fabric: Works only on Microsoft Azure.

BI 52

BI Pipeline-centric Data Lake Google Cloud 52

Data Engineering Podcast

MAY 7, 2023

However, that's also something we're re-thinking with our warehouse-centric strategy. Rudderstack]([link] RudderStack provides all your customer data pipelines in one platform. You can collect, transform, and route data across your entire stack with its event streaming, ETL, and reverse ETL pipelines.

Pipeline-centric 130

Pipeline-centric Data Lake Machine Learning Data Warehouse 130

Towards Data Science

JUNE 27, 2023

Although I wasn’t aware of all the hype, the Data-Centric AI Community promptly came to the rescue: The 2.0 There is nothing worst for a data flow than wrong typesets , especially within a data-centric AI paradigm. In the new release, users can rest to sure that their pipelines won’t break if they’re using pandas 2.0,

Pipeline-centric 98

Pipeline-centric Data Science Machine Learning Education 98

Cloudera

SEPTEMBER 27, 2024

At EVOLVE in Singapore, the Manila Electric Company, Meralco , won the Cloudera 2024 Data Impact Award in the Leadership and Transformation category for its customer-centric and data-driven transformation. But, what is the ultimate impact of all this effort and investment on each of us in our daily lives?

Pipeline-centric 90

Pipeline-centric Pharmaceutical Telecommunication Data Analytics 90

Data Engineering Weekly

JULY 28, 2024

Adopting LLM in SQL-centric workflow is particularly interesting since companies increasingly try text-2-SQL to boost data usage. Pipeline breakpoint feature. I like testing people on their practical knowledge rather than artificial coding challenges. Swiggy recently wrote about its internal platform, Hermes, a text-to-SQL solution.

Data Engineering 111

Data Engineering Data Engineer Engineering Database-centric 111

ProjectPro

JUNE 6, 2025

This project builds a comprehensive ETL and analytics pipeline, from ingestion to visualization, using Google Cloud Platform. Tech Stack: Python, PySpark, Mage, Looker, GCP- BigQuery Skills Deveoped: Building ETL pipelines using PySpark and Mage. End-to-end analytics pipeline design. Interactive dashboards creation in Looker.

Data Engineering 59

Data Engineering Data Engineer Project Engineering 59

Netflix Tech

NOVEMBER 3, 2023

When a user wants to leverage Data Mesh to move and transform data, they start by creating a new Data Mesh pipeline. The pipeline is composed of individual “Processors” that are connected by Kafka topics. Furthermore, many pipelines needed to be composed of multiple Processors. Overview of the SQL Processor workflow.

SQL 112

SQL Pipeline-centric Kafka Data 112

Data Engineering Podcast

APRIL 7, 2019

Summary How much time do you spend maintaining your data pipeline? How does the data-centric approach of DataCoral differ from the way that other platforms think about processing information? How does the data-centric approach of DataCoral differ from the way that other platforms think about processing information?

Data Pipeline 100

Data Pipeline Pipeline-centric Database-centric AWS 100

The Pragmatic Engineer

JANUARY 5, 2023

The first response has been frustration because of the chaos a breach like this causes: At a scaleup I talked with, infrastructure teams shut down all pipelines in order to replace secrets. Our customers are some of the most innovative, engineering-centric businesses on the planet, and helping them do great work will continue to be our focus.”

Pipeline-centric 130

Pipeline-centric Database-centric Coding Accessible 130

ProjectPro

JUNE 6, 2025

Read this dbt (data build tool) Snowflake tutorial blog to leverage the combined potential of dbt, the ultimate data transformation tool, and Snowflake, the scalable cloud data warehouse, to create efficient data pipelines. Snowflake dbt Tutorial- Getting Started with Snowflake dbt Integration How To Build Snowflake dbt Data Pipelines?

Pipeline-centric 40

Pipeline-centric Database-centric Raw Data Data Warehouse 40

François Nguyen

MARCH 22, 2021

You are starting to be an operation or technology centric data team. How to build Data Products or never call me Data Pipeline any more You have this interesting schema in her second article on Data Mesh by Zhamak Dehghani : “Data mesh introduces the concept of data product as its architectural quantum.

Technology 130

Technology Architecture Google Cloud Metadata 130

Data Engineering Podcast

AUGUST 13, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. Data stacks are becoming more and more complex.

Machine Learning 100

Machine Learning Pipeline-centric Database-centric MongoDB 100

ProjectPro

JUNE 6, 2025

It has become the go-to toolkit for AI developers looking to build Retrieval-Augmented Generation (RAG) pipelines , tool integrations, and sequential reasoning workflows. LangChain excels in providing linear, pipeline-based structures that streamline straightforward tasks.

Pipeline-centric 40

Pipeline-centric Database-centric Architecture Database 40

Christophe Blefari

SEPTEMBER 13, 2024

NVidia released Eagle a vision-centric multimodal LLM — Look at the example in the Github repo, given an image and a user input the LLM is able to answer things like "Describe the image in detail" or "Which car in the picture is more aerodynamic" based on a drawing.

Pipeline-centric 100

Pipeline-centric Data Python Data Science 100

Cloudera

SEPTEMBER 17, 2020

For modern data engineers using Apache Spark, DE offers an all-inclusive toolset that enables data pipeline orchestration, automation, advanced monitoring, visual troubleshooting, and a comprehensive management toolset for streamlining ETL processes and making complex data actionable across your analytic teams. Job Deployment Made Simple.

Data Pipeline 104

Data Pipeline Data Engineering Data Engineer Engineering 104

Christophe Blefari

APRIL 8, 2023

At the same time Maxime Beauchemin wrote a post about Entity-Centric data modeling. This week I discovered SQLMesh , a all-in-one data pipelines tool. I hope he will fill the gaps. In the first part he treats about the history of modeling and the main concepts. When it comes to modeling it's hard not to mention dbt.

Pipeline-centric 130

Pipeline-centric Database-centric Algorithm Data 130

Christophe Blefari

APRIL 8, 2023

At the same time Maxime Beauchemin wrote a post about Entity-Centric data modeling. This week I discovered SQLMesh , a all-in-one data pipelines tool. I hope he will fill the gaps. In the first part he treats about the history of modeling and the main concepts. When it comes to modeling it's hard not to mention dbt.

Pipeline-centric 130

Pipeline-centric Database-centric Algorithm Data 130

Netflix Tech

JANUARY 10, 2024

The Netflix video processing pipeline went live with the launch of our streaming service in 2007. By integrating with studio content systems, we enabled the pipeline to leverage rich metadata from the creative side and create more engaging member experiences like interactive storytelling.

Process 101

Process Pipeline-centric Media Metadata 101

ProjectPro

JUNE 6, 2025

Azure SQL Data Warehouse Projects Azure Data Factory and Databricks End-to-End Project Build Streaming Data Pipeline using Azure Stream Analytics Learn Real-Time Data Ingestion with Azure Purview Teradata Teradata is a prominent data warehousing and analytics platform renowned for efficiently managing and analyzing vast datasets.

Data Storage 58

Data Storage PostgreSQL Data Warehouse AWS 58

ProjectPro

JUNE 6, 2025

Pydantic AI agents offer a robust, Python-centric framework designed to streamline the development of AI-driven applications. In contrast, PydanticAI is a Python-centric framework focused on type safety, structured response validation, and integration with multiple LLMs. Source: www.linkedin.com/ Why Use Pydantic AI Agent Framework?

Building 40

Building Pipeline-centric Database-centric Data Validation 40

Cloudera

JANUARY 24, 2023

Next, it needed to enhance the company’s customer-centric approach for a needs-based alignment of products and services. We are positive that our continuing partnership with Cloudera and Blutech Consulting will be foundational to our customer-centric approach, considerably improving our customer responsiveness,” he said.

Banking 80

Banking Data Analytics Pipeline-centric IT 80

Snowflake

DECEMBER 19, 2023

Sometimes they need feedback on touchpoints very quickly, while other pipelines don’t need as much acceleration. Acadia, a digital media agency, wanted to accelerate end-to-end pipeline for its clients while also enhancing security for clients’ PII. One conversation quickly coming to the forefront is first-party data.

Building 104

Building Pipeline-centric Database-centric Digital Media 104

Monte Carlo

JANUARY 21, 2025

Key Themes Data-Driven Decision-Making : Learn how to build a data-centric culture that drives better outcomes. If youre attending, be sure to stop by Booth #219 to learn more about how data observability can enable your team to build AI-ready pipelines. Its a unique blend of business and technical expertise under one roof.

Data Analytics 52

Data Analytics Pipeline-centric Food Data Lake 52

ProjectPro

JUNE 6, 2025

Data scientists can practice Kubernetes projects to gain proficiency in deploying and managing data pipelines across cloud providers or on-premises infrastructure. This project can handle unexpected failures and keep the data flowing smoothly using Kubernetes for fault-tolerant data pipelines.

Data Science 40

Data Science Project Pipeline-centric Healthcare 40

ProjectPro

JUNE 6, 2025

Synapse Analytics Offerings : Synapse Analytics tools provide a suite of advanced analytics services: Synapse Data Warehousing: A scalable data warehousing solution designed around lake-centric architecture, allowing independent scaling of compute and storage resources. Gain Expertise Using Microsoft Fabric with ProjectPro!

Database-centric 40

Database-centric BI Pipeline-centric Data Lake 40

Cloudera

APRIL 22, 2021

Look at details of volumes/buckets/keys/containers/pipelines/datanodes. Given a file, find out what nodes/pipeline is it part of. Seamlessly scale the architecture to thousands of nodes with a single pane of glass management using Cisco Application Centric Infrastructure (ACI).

Pipeline-centric 101

Pipeline-centric Data Lake Hadoop Metadata 101

Data Engineering Weekly

JUNE 2, 2024

The resulting solution was SnowPatrol, an OSS app that alerts on anomalous Snowflake usage, powered by ML Airflow pipelines. link] Adevinta: How we moved from local scripts and spreadsheets shared by email to Data Products Data Product Thinking Shaping the data management to build a reliable, customer-centric data application.

Data Engineering 104

Data Engineering Data Engineer Engineering Pipeline-centric 104

Cloudera

DECEMBER 13, 2023

To enable LGIM to better utilize its wealth of data, LGIM required a centralized platform that made internal data discovery easy for all teams and could securely integrate external partners and third-party outsourced data pipelines. To realize this cohesive data vision, LGIM adopted Cloudera Data Platform (CDP) Public Cloud.

Pipeline-centric 89

Pipeline-centric Professional Services BI Datasets 89

Confluent

SEPTEMBER 25, 2019

Of course, this is not to imply that companies will become only software (there are still plenty of people in even the most software-centric companies), just that the full scope of the business is captured in an integrated software defined process. Here, the bank loan business division has essentially become software.

Database-centric 56

Database-centric Kafka Pipeline-centric Retail 56