Ascend.io

APRIL 22, 2024

This initiative is more than just an upgrade; it’s a reimagining of what a Data Automation Platform can be: dynamic, extensible, and highly intelligent. A unified platform that combines a powerful metadata core, an extensible plugin architecture, DataAware automation, and multiple AI Assistants.

Project 52

Project Metadata Data Pipeline Data Engineering 52

Data Engineering Weekly

MAY 16, 2023

Like the staging environment, Fronting Kafka receives all the events without validation. A streaming consumer, often implemented in stream processing frameworks like Flink or Spark, consumes the events from the fronting Kafka and runs through data contract validation. Event Routers typically don’t alter the payload.

Engineering 110

Engineering Kafka Data Pipeline Data Warehouse 110

Workfall

JUNE 12, 2023

Moreover, it facilitates the implementation of microservices architectures and event-driven systems, automating reactions to data changes without manual intervention. It captures incremental changes from transactional databases or other sources, efficiently loading them into data warehouses or data lakes.

Telecommunication 80

Telecommunication Metadata Healthcare Finance 80

Data Engineering Podcast

MARCH 27, 2022

Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. Acryl]([link] The modern data stack needs a reimagined metadata management platform.

Data Governance 130

Data Governance Government Cloud Building 130

Data Engineering Podcast

JULY 10, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Metadata 100

Metadata MongoDB MySQL Data Pipeline 100

François Nguyen

MARCH 22, 2021

TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. With this 3rd platform generation, you have more real time data analytics and a cost reduction because it is easier to manage this infrastructure in the cloud thanks to managed services. What you have to code is this workflow !

Architecture 130

Architecture Technology Google Cloud Metadata 130

Data Engineering Podcast

FEBRUARY 20, 2022

RudderStack’s smart customer data pipeline is warehouse-first. It builds your customer data warehouse and your identity graph on your data warehouse, with support for Snowflake, Google BigQuery, Amazon Redshift, and more. RudderStack’s smart customer data pipeline is warehouse-first.

Systems 100

Systems Software Engineer Software Engineering Data Warehouse 100

Cloudera

JULY 14, 2023

While these instructions are carried out for Cloudera Data Platform (CDP), Cloudera Data Engineering, and Cloudera Data Warehouse, one can extrapolate them easily to other services and other use cases as well. In all the use cases we are trying to migrate a table named “events.”

Metadata 70

Metadata Data Warehouse Big Data Ecosystem Java 70

Data Engineering Podcast

DECEMBER 16, 2019

The solution to discoverability and tracking of data lineage is to incorporate a metadata repository into your data platform. The metadata repository serves as a data catalog and a means of reporting on the health and status of your datasets when it is properly integrated into the rest of your tools.

Metadata 100

Metadata PostgreSQL Datasets Data Warehouse 100

Data Engineering Podcast

MARCH 22, 2021

Datafold also helps automate regression testing of ETL code with its Data Diff feature that instantly shows how a change in ETL or BI code affects the produced data, both on a statistical level and down to individual rows and values. RudderStack’s smart customer data pipeline is warehouse-first.

Data Warehouse 100

Data Warehouse Metadata Data Lake Hadoop 100

Data Engineering Podcast

JULY 5, 2021

RudderStack’s smart customer data pipeline is warehouse-first. It builds your customer data warehouse and your identity graph on your data warehouse, with support for Snowflake, Google BigQuery, Amazon Redshift, and more. How does SaasGlue manage metadata propagation throughout the execution graph?

Systems 100

Systems Management Data Warehouse Programming Language 100

Cloudera

AUGUST 31, 2021

Cloudera and Accenture demonstrate strength in their relationship with an accelerator called the Smart Data Transition Toolkit for migration of legacy data warehouses into Cloudera Data Platform. Accenture’s Smart Data Transition Toolkit . Are you looking for your data warehouse to support the hybrid multi-cloud?

Data Warehouse 82

Data Warehouse Database-centric Metadata Cloud 82

Netflix Tech

MARCH 7, 2024

Once we have discovered the Parquet files to be processed, MetaflowDataFrame takes over: it downloads data using Metaflow’s high-throughput S3 client directly to the process’ memory, which often outperforms reading of local files. Explainer flow is event-triggered by an upstream flow, such Model A, B, C flows in the illustration.

Systems 90

Systems Media Machine Learning Data Warehouse 90

Data Engineering Podcast

APRIL 24, 2022

Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. How do you structure the log events and metadata to provide detail and context for data applications?

Machine Learning 100

Machine Learning Systems Data Lake Metadata 100

Data Engineering Podcast

JUNE 8, 2021

RudderStack’s smart customer data pipeline is warehouse-first. It builds your customer data warehouse and your identity graph on your data warehouse, with support for Snowflake, Google BigQuery, Amazon Redshift, and more. RudderStack’s smart customer data pipeline is warehouse-first.

Data Warehouse 100

Data Warehouse Hadoop Metadata Architecture 100

Data Engineering Podcast

MAY 3, 2021

They explain how to think about your data systems in a holistic and maintainable fashion, the security challenges that threaten to derail your efforts, and the power of using metadata as the foundation of everything that you do. RudderStack’s smart customer data pipeline is warehouse-first.

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100

Data Engineering Podcast

APRIL 5, 2021

Datafold also helps automate regression testing of ETL code with its Data Diff feature that instantly shows how a change in ETL or BI code affects the produced data, both on a statistical level and down to individual rows and values. RudderStack’s smart customer data pipeline is warehouse-first.

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100

Monte Carlo

MAY 18, 2023

A question to ask is how will these data lineage source solutions account for dbt models, Fivetran connectors, and their relationship to tables? If our organization decided to pack up and migrate from one table format or data warehouse to another, would all the work we put into maintaining our open source data lineage solution for naught?

Metadata 64

Metadata Data Lake Government Data Governance 64

Maxime Beauchemin

JANUARY 20, 2017

Data modeling is changing Typical data modeling techniques — like the star schema  — which defined our approach to data modeling for the analytics workloads typically associated with data warehouses, are less relevant than they once were.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130

AltexSoft

AUGUST 29, 2023

Instead of relying on traditional hierarchical structures and predefined schemas, as in the case of data warehouses, a data lake utilizes a flat architecture. This structure is made efficient by data engineering practices that include object storage. Data warehouse vs. data lake in a nutshell.

Data Lake 52

Data Lake Architecture IT Amazon Web Services 52

ProjectPro

FEBRUARY 16, 2023

With so much riding on the efficiency of ETL processes for data engineering teams, it is essential to take a deep dive into the complex world of ETL on AWS to take your data management to the next level. ETL has typically been carried out utilizing data warehouses and on-premise ETL tools.

AWS 52

AWS Data Management ETL Tools Management 52

Rockset

APRIL 26, 2023

Offline feature store : Detecting anomalies requires historical data in order to have a baseline for comparisons. This data tends to be slow changing and is stored in an offline feature store. This could be a cloud data warehouse, a data lake, or a database. The database has two primary jobs.

Architecture 52

Architecture Transportation Data Lake Insurance 52

Monte Carlo

AUGUST 29, 2022

To do this, you need data monitors that drill “ deep ” into the data using both machine learning and user-defined rules, as well as metadata monitors to scale “ broadly ” across every production table in your environment and to be fully integrated across your stack.

IT 52

IT Metadata Data Warehouse BI 52

Databand.ai

JUNE 28, 2023

This might include processes like data extraction from different sources, data cleansing, data transformation (like aggregation), and loading the data into a database or a data warehouse. Data storage and delivery : Observability continues into the storage and delivery phase.

Data Pipeline 98

Data Pipeline Data Engineering Data Engineer Engineering 98

Cloudera

SEPTEMBER 17, 2020

By enabling their event analysts to monitor and analyze events in real time, as well as directly in their data visualization tool, and also rate and give feedback to the system interactively, they increased their data to insight productivity by a factor of 10. . Our solution: Cloudera Data Visualization.

Unstructured Data 81

Unstructured Data Pharmaceutical Data Warehouse MySQL 81

Data Engineering Podcast

AUGUST 6, 2022

Summary The optimal format for storage and retrieval of data is dependent on how it is going to be used. For analytical systems there are decades of investment in data warehouses and various modeling techniques. What are some of the data management considerations that are introduced by vector databases?

Machine Learning 100

Machine Learning Database MySQL PostgreSQL 100

dbt Developer Hub

JANUARY 24, 2023

Ensures developers can work quickly by shortening the CI feedback loop Reduce costs in your data warehouse by running only what's been modified I think we can all agree that increasing developer productivity while simultaneously reducing costs is an outcome that every company strives for. A query method on the metadata property.

Metadata 52

Metadata Python Data Warehouse Cloud 52

Grouparoo

OCTOBER 6, 2021

This provided a nice overview of the breadth of topics that are relevant to data engineering including data warehouses/lakes, pipelines, metadata, security, compliance, quality, and working with other teams. For example, grouping the ones about metadata, discoverability, and column naming might have made a lot of sense.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Striim

JANUARY 4, 2023

Companies need to analyze large volumes of datasets, leading to an increase in data producers and consumers within their IT infrastructures. These companies collect data from production applications and B2B SaaS tools (e.g., This data makes its way into a data repository, like a data warehouse (e.g.,

PostgreSQL 52

PostgreSQL Data Warehouse Data Lake Data 52

Hepta Analytics

FEBRUARY 14, 2022

This week, we got to think about our data ingestion design. We looked at the following: How do we ingest – ETL vs ELT Where do we store the data – Data lake vs data warehouse Which tool to we use to ingest – cronjob vs workflow engine NOTE : This weeks task requires good internet speed and good compute.

Data Ingestion 52

Data Ingestion Data Engineering Data Engineer Engineering 52

Afterpay Tech

SEPTEMBER 6, 2022

This pattern also enables us to move away from the dependency on the binlog replication for sending data to the data warehouse. This way, Kafka event publication becomes silent and is decoupled from the main payments flow to not interfere with or introduce extra latency to the customer experience.

Kafka 52

Kafka AWS Metadata Data Warehouse 52

Cloudera

SEPTEMBER 15, 2022

July brings summer vacations, holiday gatherings, and for the first time in two years, the return of the Massachusetts Institute of Technology (MIT) Chief Data Officer symposium as an in-person event. A key area of focus for the symposium this year was the design and deployment of modern data platforms. What is a data fabric?

Data Lake 79

Data Lake Analytics Application Cloud Storage Architecture 79

Netflix Tech

DECEMBER 21, 2020

By Anupom Syam Background At Netflix, our current data warehouse contains hundreds of Petabytes of data stored in AWS S3 , and each day we ingest and create additional Petabytes. Some of the optimizations are prerequisites for a high-performance data warehouse.

Data Warehouse 81

Data Warehouse Metadata Algorithm Data 81

Monte Carlo

SEPTEMBER 20, 2022

The complexity of Prefect’s data pipelines underscores the value—and challenges—of achieving data reliability at scale. But a growing company means growing data needs. After some initial cheerleading, Dylan convinced the Prefect team to stand up a Monte Carlo instance for their BigQuery data warehouse.

Big Data 52

Big Data Data Warehouse Data Data Governance 52

Data Engineering Weekly

NOVEMBER 6, 2022

link] Pinterest: How Pinterest Leverages Realtime User Actions in Recommendation to Boost Homefeed Engagement Volume Looping in real-time user interaction events with the recommendation engine can significantly improve the user experience. Every data processing engine has one metadata store to integrate.

Data Engineering 40

Data Engineering Data Engineer Engineering Machine Learning 40

Rockset

AUGUST 2, 2023

The Windward Maritime AI platform Lastly, Windward wanted to move their entire platform from batch-based data infrastructure to streaming. This transition can support new use cases that require a faster way to analyze events that was not needed until now. They used MongoDB as their metadata store to capture vessel and company data.

Database-centric 52

Database-centric PostgreSQL Transportation Insurance 52

Snowflake

NOVEMBER 2, 2023

Over the years, the technology landscape for data management has given rise to various architecture patterns, each thoughtfully designed to cater to specific use cases and requirements. These patterns include both centralized storage patterns like data warehouse , data lake and data lakehouse , and distributed patterns such as data mesh.

Data Lake 93

Data Lake Data Warehouse Cloud Unstructured Data 93