Ascend.io

APRIL 22, 2024

Plus, we’ll offer a teaser of what’s to come: an exclusive live demo that you simply cannot miss. The Evolution of Data Engineering Over the past decade, we’ve seen tremendous change in the data engineering landscape. It’s time we stop Data Plumbing, and get back to Data Engineering. Let’s dive in!

Project 52

Project Metadata Data Pipeline Data Engineering 52

Christophe Blefari

APRIL 12, 2024

LLM DataGen — A small demo of a LLM based on Gemma that generates JSONL based on a given name. BigQuery releases data canvas — This is a large open canva (like count.co ) in which you can write SQL queries, assisted by Gemini, and link them in a DAG fashion.

BI 130

BI SQL Data Coding 130

Confluent

NOVEMBER 20, 2020

Building data pipelines isn’t always straightforward. The gap between the shiny “hello world” examples of demos and the gritty reality of messy data and imperfect formats is sometimes all too […].

Cloud 139

Cloud Data Pipeline Data Building 139

DataKitchen

FEBRUARY 23, 2024

Webinar: Beyond Data Observability: Personalization DataKitchen DataOps Observability Problem Statement White Paper: ‘Taming Chaos’ Technical Product Overview Four-minute online demo Detailed Product: Documentation Webinar: Data Observability Demo Day DataKitchen DataOps TestGen Problem Statement White Paper: ‘Mystery Box Full Of Data Errors’ (..)

Data Validation 48

Data Validation Data Data Pipeline Data Solutions 48

Grouparoo

MAY 11, 2021

We'll make the assumption that Dylan already has a server available, so I'm just going to demo this on my local machine for simplicity. The instructions and concepts easily transfer to wherever you want to run Grouparoo. Integration Steps First, we've got to do some setup.

Data Pipeline 52

Data Pipeline Data Project Utilities 52

Snowflake

APRIL 8, 2024

Many entries also used Snowpark , taking advantage of the ability to work in the code they prefer to develop data pipelines, ML models and apps, then execute in Snowflake. Our sincere thanks go out to everyone who participated in this year’s competition.

Pipeline-centric 120

Pipeline-centric Food Healthcare Unstructured Data 120

Data Engineering Podcast

DECEMBER 4, 2022

Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days or even weeks. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.

Data Lake 100

Data Lake MongoDB MySQL Data Warehouse 100

Data Engineering Podcast

DECEMBER 11, 2022

Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days or even weeks. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.

Database 100

Database MySQL Data Lake MongoDB 100

Data Engineering Podcast

OCTOBER 30, 2022

Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days or even weeks. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.

Engineering 100

Engineering MongoDB Scala MySQL 100

Monte Carlo

JUNE 6, 2023

But is there more to generative AI than a fancy demo on Twitter? And how will it impact data? How generative AI will disrupt data With the advent of OpenAI’s DALL-E and ChatGPT, large language models became much more useful to the vast majority of humans. Request a demo today. Let’s assess. Will you join us?

Data Engineering 84

Data Engineering Data Engineer Engineering Business Intelligence 84

Data Engineering Podcast

JULY 17, 2022

Go to dataengineeringpodcast.com/atlan today to learn more about how Atlan’s active metadata platform is helping pioneering data teams like Postman, Plaid, WeWork & Unilever achieve extraordinary things with metadata and escape the chaos. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Data Management 130

Data Management Management Metadata MongoDB 130

Data Engineering Podcast

NOVEMBER 6, 2022

Their SDKs make event streaming from any app or website easy, and their extensive library of integrations enable you to automatically send data to hundreds of downstream tools. Sign up free at dataengineeringpodcast.com/rudder Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

MongoDB 130

MongoDB Scala MySQL Data Lake 130

Databand.ai

AUGUST 30, 2023

Besides these categories, specialized solutions tailored specifically for particular domains or use cases also exist, such as extract, transform and load (ETL) tools for managing data pipelines, data integration tools for combining information from disparate sources or systems and more.

Data Cleanse 52

Data Cleanse Data Pipeline Datasets Data Validation 52

Data Engineering Podcast

JULY 10, 2022

Go to dataengineeringpodcast.com/atlan today to learn more about how Atlan’s active metadata platform is helping pioneering data teams like Postman, Plaid, WeWork & Unilever achieve extraordinary things with metadata and escape the chaos. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Metadata 100

Metadata MongoDB MySQL Data Pipeline 100

The Modern Data Company

JANUARY 31, 2023

This post highlights exactly how our founders taught us to think differently about data and why it matters. Here are the cornerstones of this new paradigm: Data ownership is a construct Data pipelines should be accessible to everyone Data products should adapt to the organization, not vice versa.

Retail 98

Retail Healthcare Data Architecture Data Pipeline 98

Data Engineering Podcast

JUNE 19, 2022

Go to dataengineeringpodcast.com/atlan today to learn more about how Atlan’s active metadata platform is helping pioneering data teams like Postman, Plaid, WeWork & Unilever achieve extraordinary things with metadata and escape the chaos. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Metadata 130

Metadata Unstructured Data MongoDB MySQL 130

Snowflake

DECEMBER 21, 2023

In this edition, Patch.tech Co-Founder and CPO Whelan Boyd talks about how frustration with clogged data pipelines sparked the idea for Patch’s code packages, which allow engineers to distribute data sets with all the built-in elements that analysts and developers need to create apps. Snowflake’s ecosystem is unmatched.

Software Engineer 84

Software Engineer Software Engineering Database Data Pipeline 84

Monte Carlo

AUGUST 22, 2023

Slow running data pipelines cost data teams time, money, and goodwill. They utilize excess compute, cause data quality issues, and create a poor user experience for data consumers that must wait in exasperation for data to return, dashboards to load, and AI models to update.

Metadata 52

Metadata Data Pipeline Data Warehouse Utilities 52

Monte Carlo

NOVEMBER 9, 2023

And in many ways, LLMs are going to make data engineers more valuable – and that’s exciting! Still, it’s one thing to show your boss a cool demo of a data discovery tool or text-to-SQL generator – it’s another thing to use it with your company’s proprietary data, or even more concerning, customer data.

Unstructured Data 86

Unstructured Data Database Data Pipeline Architecture 86

Precisely

SEPTEMBER 12, 2023

As a proud member of the Connect with Confluent program , we help organizations going through digital transformation and IT infrastructure modernization break down data silos and power their streaming data pipelines with trusted data. Let’s cover some additional information to know before attending.

Kafka 52

Kafka Data Integration Data Pipeline Datasets 52

Data Engineering Podcast

JULY 3, 2022

Go to dataengineeringpodcast.com/atlan today to learn more about how Atlan’s active metadata platform is helping pioneering data teams like Postman, Plaid, WeWork & Unilever achieve extraordinary things with metadata and escape the chaos. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Architecture 100

Architecture Metadata MongoDB MySQL 100

Monte Carlo

MAY 26, 2023

A better measurement is the data downtime formula (above), which more comprehensively measures the amount of time the data was inaccurate, missing, or otherwise erroneous. A proactive approach for measuring data freshness is to create service level agreements or SLAs for specific data pipelines.

Data Pipeline 98

Data Pipeline Data Data Warehouse Machine Learning 98

Ascend.io

DECEMBER 19, 2022

We hope the real-time demonstrations of Ascend automating data pipelines were a real treat—a long with the special edition T-Shirt designed specifically for the show (picture of our founder and CEO rocking the t-shirt below). Thank you to the hundreds of AWS re:Invent attendees who stopped by our booth!

Data Ingestion 52

Data Ingestion Data Pipeline Metadata AWS 52

Workfall

SEPTEMBER 18, 2023

It’s designed to be user-friendly, customizable, and extensible, making it a valuable tool for data engineers, analysts, and data-driven organizations looking to streamline their data pipelines. With its drag-and-drop interface, creating data pipelines becomes as easy as arranging blocks in a puzzle.

Data Pipeline 93

Data Pipeline Raw Data Data Schemas Healthcare 93

Christophe Blefari

APRIL 8, 2023

In the recent years dbt simplified and revolutionised the tooling to create data models. This week I discovered SQLMesh , a all-in-one data pipelines tool. In their demo you can use prompt to search over images or timeseries. dbt, as of today, is the leading framework. But alternatives are coming. Roboto AI raises $4.8m

Pipeline-centric 130

Pipeline-centric Database-centric Algorithm Data 130

Christophe Blefari

APRIL 8, 2023

In the recent years dbt simplified and revolutionised the tooling to create data models. This week I discovered SQLMesh , a all-in-one data pipelines tool. In their demo you can use prompt to search over images or timeseries. dbt, as of today, is the leading framework. But alternatives are coming. Roboto AI raises $4.8m

Pipeline-centric 130

Pipeline-centric Database-centric Algorithm Data 130

Data Engineering Podcast

NOVEMBER 13, 2022

Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days or even weeks. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.

MongoDB 100

MongoDB Scala MySQL Google Cloud 100

Data Engineering Podcast

FEBRUARY 24, 2020

Are you spending too much time maintaining your data pipeline? Snowplow empowers your business with a real-time event data pipeline running in your own cloud account without the hassle of maintenance. Set up a demo and mention you’re a listener for a special offer!

IT 100

IT Data Lake Data Pipeline Media 100

Monte Carlo

MAY 2, 2023

Respondents averaged 642 tables across their data lake, lakehouse, or warehouse environments. Respondents averaged 290 manually-written tests across their data pipelines. Check out the full report , including commentary and reactions from nearly a dozen industry-leading data executives.

Data Lake 80

Data Lake Data Software Engineer Software Engineering 80

Ascend.io

MAY 23, 2023

The data engineering world is full of tips and tricks on how to handle specific patterns that recur with every data pipeline. In general, the root of the orphaned data problem lies in the interim storage of data between the individual processing steps that make up a data pipeline.

Datasets 52

Datasets Data Pipeline Metadata Database 52

Monte Carlo

OCTOBER 4, 2022

In November of 2020, O’Reilly Media first approached us with the idea to author Data Quality Fundamentals: A Practitioner’s Guide to Building More Trustworthy Data Pipelines. It was an inflection point for a fledgling company that had only just begun to establish the category of data observability.

High Quality Data 52

High Quality Data Data Pipeline Media Data 52

Snowflake

JUNE 27, 2023

Common examples include container images used to kick-off ML training on GPUs or a step in a data pipeline which may be running any language, framework or library inside a container. Check out the demo video and sign up for the waitlist to deploy Hex in Snowflake. LLM Providers. Astronomer.

Scala 58

Scala Programming Language Data Science Python 58

Towards Data Science

JUNE 19, 2023

I use Compose daily at work and for my personal projects to build and manage my data pipelines, and today I’d like to show how you can build your ETL projects with Compose. Inspired by Docker Compose, this solution came in the form of a project called Meerschaum Compose. mrsm compose run Registers the pipes and syncs them one-by-one.

PostgreSQL 83

PostgreSQL SQL Python Project 83

Data Engineering Podcast

JUNE 5, 2022

Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days or even weeks. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.

Data Security 100

Data Security Metadata MongoDB Scala 100

Datakin

SEPTEMBER 13, 2021

It runs through the lineage graph, performance metrics, data quality, and data pipeline diffing with the Compare tab. View our instant demo system right now, or sign up for Datakin to see it with your own datasets and jobs. The best way to understand what Datakin does is to see it. You can see it here: Want to learn more?

Metadata 52

Metadata Datasets Data Pipeline Systems 52

The Modern Data Company

JULY 31, 2023

By omitting the complexity and debt of data pipelines resulting from multiple fragmented tools in the ‘MAD Ecosystem,’ these loops can be efficiently maintained Process Angle : Administering Data Products involves a combination of technology, culture, and process. Read more about our partnership.

Government 98

Government Data Governance Architecture Data Pipeline 98

Data Engineering Weekly

JULY 2, 2023

Please send your talk proposal today, and let's collaborate to make this the most exciting, insightful, and interactive Data Eng & AI conference yet! LinkedIn write about Hoptimator for auto generated Flink pipeline with multiple stages of systems. Please watch out for this space for the conference registration page soon!!!

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52