Data Engineering Podcast

JUNE 12, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking all of that information into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Go to dataengineeringpodcast.com/atlan today to learn more about how you can take advantage of active metadata and escape the chaos.

Metadata 100

Metadata Unstructured Data Business Intelligence MongoDB 100

Ascend.io

DECEMBER 19, 2022

We hope the real-time demonstrations of Ascend automating data pipelines were a real treat—a long with the special edition T-Shirt designed specifically for the show (picture of our founder and CEO rocking the t-shirt below). Instead, it is a Sankey diagram driven by the same dynamic metadata that runs the Ascend control plane.

Data Ingestion 52

Data Ingestion Data Pipeline Metadata AWS 52

Towards Data Science

JUNE 19, 2023

I use Compose daily at work and for my personal projects to build and manage my data pipelines, and today I’d like to show how you can build your ETL projects with Compose. This may be as simple as a SQL query or may include custom keys for use in your plugins. mrsm compose run Registers the pipes and syncs them one-by-one.

PostgreSQL 74

PostgreSQL SQL Python Project 74

Data Engineering Podcast

DECEMBER 16, 2019

Summary Building clean datasets with reliable and reproducible ingestion pipelines is completely useless if it’s not possible to find them and understand their provenance. The solution to discoverability and tracking of data lineage is to incorporate a metadata repository into your data platform.

Metadata 100

Metadata PostgreSQL Datasets Data Warehouse 100

Data Engineering Weekly

JULY 2, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork, and SQL grunt work out of building complete customer profiles, so you can quickly ship actionable, enriched data to every downstream team. Write SQL queries without learning SQL? See how it works today.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Silectis

DECEMBER 15, 2020

A Unified Data Engineering Solution Your journey with Magpie starts with a seamless integration with AWS, GCP, or Azure. Magpie is an enterprise-ready solution built on the powerful Apache Spark, but with language support for SQL, Python, R, and Scala. The time to identify more insights out of huge sets of data. Or your team?

Data Engineering 52

Data Engineering Data Engineer Engineering ETL Tools 52

Monte Carlo

JANUARY 23, 2024

Gen AI can whip up serviceable code in moments — making it much faster to build and test data pipelines. Today’s LLMs can already process enormous amounts of unstructured data, automating much of the monotonous work of data science. But what does that mean for the roles of data engineers and data scientists going forward?

Pipeline-centric 52

Pipeline-centric Database-centric Metadata Unstructured Data 52

Rockset

AUGUST 26, 2022

When a change is detected, the data is ingested into a data lake hosted on Amazon S3. There, the data is converted, tagged with metadata, cleaned, and de-duplicated in preparation for queries. They demoed but quickly eliminated Snowflake. Data is uploaded to Rockset from S3 every 3-5 minutes.

IT 52

IT Food Data Lake Aggregated Data 52

Rockset

APRIL 18, 2023

In real-time ML, the freshness of the features, the serving latency, and the uptime and availability of the data pipeline and model matter. Turn events into real-time features: Rockset turns events into features in real time with SQL ingest transformations. 0 0 Ok, we have the imported data.

Machine Learning 52

Machine Learning Datasets Database Data Pipeline 52

Monte Carlo

AUGUST 7, 2023

Prevention: Finally, data observability also provides mechanisms to prevent data issues from happening in the first place, like placing circuit breaking in pipelines and creating visibility around the impact code changes would make on data, among other proactive measures of preventing bad data from entering your pipelines in the first place.

Coding 52

Coding Data Software Engineer Software Engineering 52

Monte Carlo

APRIL 20, 2023

Here is a list of 17 valuable data lineage use cases that can help improve data quality, incident management, democratization, system modernization, and compliance. I can surface ownership metadata and alert the relevant owners to make sure the appropriate changes are made so these breakages never happen.

Data Warehouse 52

Data Warehouse BI Data Government 52

Monte Carlo

JANUARY 23, 2023

Data lineage gives you the where but not always the why. Data teams must now determine if it is: A systems issue : Did an Airflow job not run? A code issue : Did someone modify a SQL query or dbt model that mucked everything up? A data issue : Did a third-party send us garbage data filled with NULLs and other nonsense?

IT 52

IT BI Government Data Governance 52

AltexSoft

OCTOBER 8, 2021

To make a more informed decision, enquire about demos and do your own in-depth research. Oracle Data Integrator, IBM InfoSphere, Snaplogic, Xplenty, and. The prevailing part of users claim that it is quite easy to configure and manage data flows with Oracle’s graphical tools. Data profiling and cleansing. Ease of use.

Data Integration 52

Data Integration Hadoop Data Warehouse Data Lake 52

AltexSoft

MARCH 30, 2023

Source: The Data Team’s Guide to the Databricks Lakehouse Platform Integrating with Apache Spark and other analytics engines, Delta Lake supports both batch and stream data processing. Besides that, it’s fully compatible with various data ingestion and ETL tools.

Scala 64

Scala Data Lake BI Google Cloud 64