Data Engineering Podcast

NOVEMBER 26, 2023

In this episode Tobias Macey shares his thoughts on the challenges that he is facing as he prepares to build the next set of architectural layers for his data platform to enable a larger audience to start accessing the data being managed by his team. Developing event-driven pipelines is going to be a lot easier - Meet Functions!

Architecture 147

Architecture Data Lake High Quality Data SQL 147

The Modern Data Company

JANUARY 22, 2024

This mission culminates in the esteemed recognition of honorable mention in Gartner’s 2023 Magic Quadrant for Data Integration, showcasing commitment to excellence and industry leadership in the data-driven era. Data engineering excellence Modern offers robust solutions for building, managing, and operationalizing data pipelines.

Data Integration 98

Data Integration Metadata Government Unstructured Data 98

Monte Carlo

JUNE 14, 2023

In this post, we will help you quickly level up your overall knowledge of data pipeline architecture by reviewing: Table of Contents What is data pipeline architecture? Why is data pipeline architecture important? What is data pipeline architecture? Why is data pipeline architecture important?

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

The Modern Data Company

JANUARY 22, 2024

This mission culminates in the esteemed recognition of honorable mention in Gartner’s 2023 Magic Quadrant for Data Integration, showcasing commitment to excellence and industry leadership in the data-driven era. Data engineering excellence Modern offers robust solutions for building, managing, and operationalizing data pipelines.

Data Integration 87

Data Integration Metadata Government Unstructured Data 87

Data Engineering Podcast

JULY 10, 2022

In this episode Inbar Yogev and Lior Winner share the journey that they and their teams at Riskified have been on for their data platform. They also discuss how they have established a guild system for training and supporting data professionals in the organization. Atlan is the metadata hub for your data ecosystem.

Metadata 100

Metadata MongoDB MySQL Data Pipeline 100

François Nguyen

MARCH 22, 2021

TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. How do we build data products ? How can we interoperate between the data domains ? We want interoperability for any data stored versus we have to think how to store the data in a specific node to optimize the processing.

Technology 130

Technology Architecture Google Cloud Metadata 130

Data Engineering Podcast

JUNE 5, 2022

In this episode Sean Falconer explains the idea of a data privacy vault and how this new architectural element can drastically reduce the potential for making a mistake with how you manage regulated or personally identifiable information. Atlan is the metadata hub for your data ecosystem. how does it enter an organization)?

Data Security 100

Data Security Metadata MongoDB Scala 100

Monte Carlo

MARCH 13, 2024

Let’s dig in and explore the landscape of the top so-called “data quality tools” — what they are, what they’re not, and whether they’re the right first step towards more reliable data. Governance helps companies set important standards and achieve higher levels of data security, data accessibility, and data quality.

Data Governance 52

Data Governance Government Metadata Data 52

Databand.ai

JULY 19, 2023

However, their importance has grown significantly in recent years due to the increasing complexity of data architectures and the growing need for data governance and compliance. In this article: Why Are Data Lineage Tools Important? It provides context for data, making it easier to understand and manage.

Pipeline-centric 52

Pipeline-centric Data Governance Metadata Government 52

Silectis

DECEMBER 15, 2020

If you’ve been following along with Silectis over the past couple of years, you are familiar with our data engineering platform, Magpie. You’re aware of the many outcomes it puts at the fingertips of data engineers, and teams of data practitioners more largely. If you’re new around here, not to worry. Or your team?

Data Engineering 52

Data Engineering Data Engineer Engineering ETL Tools 52

Ascend.io

APRIL 7, 2023

With the monolithic architectures most organizations have today, business users are stuck, constantly waiting for new data pipelines to be built or amended based on their requests. Data engineers aren’t huge fans of this paradigm either. So, in theory, data fabric generates ROI by: Giving users autonomy.

Metadata 52

Metadata Datasets Data Governance Government 52

Databand.ai

AUGUST 30, 2023

Unified DataOps represents a fresh approach to managing and synchronizing data operations across several domains, including data engineering, data science, DevOps, and analytics. Technical Challenges Choosing appropriate tools and technologies is critical for streamlining data workflows across the organization.

Data Governance 40

Data Governance Data Cleanse Government Data Pipeline 40

Snowflake

DECEMBER 12, 2023

To give customers flexibility for how they fit Snowflake into their data architecture, Iceberg Tables can be configured to use either Snowflake or an external service such as AWS Glue as the table’s catalog to track metadata, with an easy, one-line SQL command to convert the table’s catalog to Snowflake in a metadata-only operation.

Metadata 90

Metadata Python AWS Government 90

Monte Carlo

JANUARY 16, 2024

Factor in the advertising strategies, media production, partner programming, audience analytics…and you’re looking at an ocean of data that would fill even the deepest trench (we’d like a television show about that too, please!). So how does Fox’s data strategy support these complex data workflows? Image from Castor.

Government 52

Government High Quality Data Data Governance Entertainment 52

Databand.ai

AUGUST 30, 2023

Each type of tool plays a specific role in the DataOps process, helping organizations manage and optimize their data pipelines more effectively. Data Integration Data integration is the process of collecting, transforming, and consolidating data from various sources.

Data Cleanse 52

Data Cleanse Data Pipeline Data Ingestion Data Validation 52

Databand.ai

AUGUST 30, 2023

Poor data quality: The lack of automation and data governance in legacy architectures can lead to data quality issues, such as incomplete, inaccurate, or duplicate data. This requires implementing robust data integration tools and practices, such as data validation, data cleansing, and metadata management.

Architecture 52

Architecture Data Ingestion Data Governance Data Cleanse 52

AltexSoft

FEBRUARY 11, 2023

Data architecture is the organization and design of how data is collected, transformed, integrated, stored, and used by a company. What is the main difference between a data architect and a data engineer? By the way, we have a video dedicated to the data engineering working principles.

Data Architect 87

Data Architect Certification Generalist Big Data 87

Monte Carlo

NOVEMBER 14, 2023

Databricks announced that Delta tables metadata will also be compatible with the Iceberg format, and Snowflake has also been moving aggressively to integrate with Iceberg. How data observability can increase data trust in Iceberg tables What is Apache Iceberg? How Apache Iceberg tables structure metadata.

Data Lake 64

Data Lake Metadata Data Warehouse SQL 64

Data Engineering Podcast

APRIL 5, 2021

She explains how the design of the platform is informed by the needs of managing data projects for large and small teams across her previous roles, how it integrates with your existing systems, and how it can work to bring everyone onto the same page. RudderStack’s smart customer data pipeline is warehouse-first.

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100

Monte Carlo

MAY 18, 2023

Data lineage tools are like detectives that help data professionals quickly sort through the tangled webs of interdependencies that make up the modern data stack. Data lineage tools help you uncover and map the complex relationships between data elements across various sources, systems, and processes.

Metadata 64

Metadata Data Lake Government Data Governance 64

ProjectPro

AUGUST 24, 2021

Nevertheless, that is not the only job in the data world. Data professionals who work with raw data like data engineers, data analysts, machine learning scientists , and machine learning engineers also play a crucial role in any data science project.

Data Engineering 52

Data Engineering Data Engineer Coding Project 52

Data Engineering Podcast

NOVEMBER 9, 2020

He also shares the internal architecture, how he approached the design to make it accessible and easy to use, and how it autodiscovers the schemas and metadata for your source systems. If you hand a book to a new data engineer, what wisdom would you add to it?

Building 100

Building PostgreSQL BI Metadata 100

François Nguyen

MARCH 7, 2021

In this article, Juan Sequada gives maybe one of the best definition of Data Mesh ” It is paradigm shift towards a distributed architecture that attempts to find an ideal balance between centralization and decentralization of metadata and data management.” The best image to use is the chicken and the pig fable.

Government 130

Government Data Governance Data Metadata 130

Monte Carlo

DECEMBER 8, 2023

Enter: data provenance and data lineage. Data lineage is a visual tool that tracks the movement and transformations of data through various systems, processes, and applications. Data provenance is the record of metadata from data’s original sources, providing the historical context and authenticity of data.

Metadata 52

Metadata Data Data Warehouse Government 52

Monte Carlo

DECEMBER 8, 2023

Enter: data provenance and data lineage. Data lineage is a visual tool that tracks the movement and transformations of data through various systems, processes, and applications. Data provenance is the record of metadata from data’s original sources, providing the historical context and authenticity of data.

Metadata 52

Metadata Data Data Warehouse Government 52

Monte Carlo

APRIL 7, 2021

Data pipelines can break for a million different reasons, and there isn’t a one-size-fits all approach to understanding how or why. Here are five critical steps data engineers must take to conduct engineering root cause analysis for data quality issues. You get the idea. Here’s what you do.

Data Engineering 40

Data Engineering Data Engineer Engineering Datasets 40

Cloudera

OCTOBER 7, 2021

Application Logic: Application logic refers to the type of data processing, and can be anything from analytical or operational systems to data pipelines that ingest data inputs, apply transformations based on some business logic and produce data outputs.

Architecture 124

Architecture Metadata Government Kafka 124

Monte Carlo

AUGUST 19, 2023

The data platform manages the collection, normalization, transformation, and application of data for a given data product—from business insights and dashboards to ML and AI engineering. So, with all that ambiguity, where’s a data engineer to start?Even

Building 52

Building BI Data Lake Data Governance 52

Databand.ai

JULY 28, 2022

In general, it makes it possible to identify errors in data, reduce the risk associated with system and process changes, and increase trust in data. All of these are essential at a time when data plays such an integral role in business outcomes and decision-making. This technique focuses directly on the data (vs.

Metadata 52

Metadata Data Lake Datasets Data Warehouse 52

Monte Carlo

SEPTEMBER 12, 2022

The field of data engineering has been growing at a breakneck pace. New frameworks, new challenges, and new technologies are constantly shifting how engineers think about their work and their roles within their organizations. Plus, he was one of the first data engineers at Facebook and Airbnb.

BI 52

BI Data Governance ETL Tools Data Warehouse 52

Cloudera

MARCH 23, 2022

Through Cloudera’s contributions, we have extended support for Hive and Impala, delivering on the vision of a data architecture for multi-function analytics from large scale data engineering (DE) workloads and stream processing (DF) to fast BI and querying (within DW) and machine learning (ML). . 3: Open Performance.

Metadata 90

Metadata Data Architecture BI Machine Learning 90

Monte Carlo

JANUARY 12, 2024

The concept of big data – complicated datasets that are too dense for traditional computing setups to deal with – is nothing new. But what is new, or still developing at least, is the extent to which data engineers can manage, data scientists can experiment, and data analysts can analyze this treasure trove of raw business insights.

Big Data 59

Big Data Data Analytics Data Science Data Lake 59

Snowflake

JUNE 27, 2023

Supporting mission criticality with enhanced native data governance, new Snowflake UIs, a growing compliance footprint, and updated cross-cloud business continuity At Snowflake, we’re committed to providing best-in-class native data governance features for customers entrusting our platform with their data.

Data Governance 57

Data Governance Unstructured Data Government SQL 57

Towards Data Science

MAY 19, 2023

Business users are unable to find and access data assets critical to their workflows. Data engineers spend countless hours troubleshooting broken pipelines. The data team is constantly burning out and has a high employee turnover. Is it the responsibility of the data product managers?

Pipeline-centric 105

Pipeline-centric Data Software Engineer Software Engineering 105

DataKitchen

JANUARY 5, 2021

Execution of this mission requires the contribution of several groups: data center/IT, data engineering, data science, data visualization, and data governance. Each of the roles mentioned above views the world through a preferred set of tools: Data Center/IT – Servers, storage, software.

Data Governance 59

Data Governance Government Metadata Data Science 59

Monte Carlo

AUGUST 25, 2023

One advantage of data warehouses is their integrated nature. As fully managed solutions, data warehouses are designed to offer ease of construction and operation. A warehouse can be a one-stop solution, where metadata, storage, and compute components come from the same place and are under the orchestration of a single vendor.

Data Lake 52

Data Lake Data Warehouse Unstructured Data Raw Data 52

Monte Carlo

JANUARY 23, 2023

Historically, data lakes tend to be preferable for companies with large volumes of data, data science, and AI and ML training development—supported by a healthy team of data engineers. So, what is a data mesh and what makes it different from a data lake? Image courtesy of Monte Carlo Data.)

Data Lake 52

Data Lake Unstructured Data Business Intelligence Architecture 52