Data Engineering Weekly

DECEMBER 29, 2022

Data Catalog as a passive web portal to display metadata requires significant rethinking to adopt modern data workflow, not just adding “modern” in its prefix. I know that is an expensive statement to make😊 To be fair, I’m a big fan of data catalogs, or metadata management , to be precise.

Metadata 62

Metadata Data Warehouse ETL Tools Data Workflow 62

Data Engineering Podcast

JANUARY 30, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Data Engineering 100

Data Engineering Data Engineer Engineering Python 100

Databand.ai

AUGUST 30, 2023

DataOps is a collaborative approach to data management that combines the agility of DevOps with the power of data analytics. It aims to streamline data ingestion, processing, and analytics by automating and integrating various data workflows.

Architecture 52

Architecture Data Ingestion Data Cleanse Data Governance 52

Data Engineering Podcast

APRIL 5, 2021

She explains how the design of the platform is informed by the needs of managing data projects for large and small teams across her previous roles, how it integrates with your existing systems, and how it can work to bring everyone onto the same page. What portions of the data workflow is Atlan responsible for?

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100

Data Engineering Podcast

MAY 3, 2021

Summary The Data industry is changing rapidly, and one of the most active areas of growth is automation of data workflows. Taking cues from the DevOps movement of the past decade data professionals are orienting around the concept of DataOps. How does this differ from "business as usual" in the data industry?

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100

Data Engineering Podcast

AUGUST 5, 2019

As organizations grow and data sources proliferate it becomes difficult to keep track of everything, particularly for analysts and data scientists who are not involved with the collection and management of that information. Finding the data that you need is tricky, and Amundsen will help you solve that problem.

PostgreSQL 100

PostgreSQL MongoDB Metadata Media 100

Data Engineering Weekly

OCTOBER 30, 2022

Editor’s Note: The current state of the Data Catalog The results are out for our poll on the current state of the Data Catalogs. The highlights are that 59% of folks think data catalogs are sometimes helpful. We saw in the Data Catalog poll how far it has to go to be helpful and active within a data workflow.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Ingestion 52

Knowledge Hut

SEPTEMBER 29, 2023

Why Should You Get an Azure Data Engineer Certification? Becoming an Azure data engineer allows you to seamlessly blend the roles of a data analyst and a data scientist. One of the pivotal responsibilities is managing data workflows and pipelines, a core aspect of a data engineer's role.

Certification 52

Certification Data Engineering Data Engineer Engineering 52

Data Engineering Podcast

JUNE 16, 2024

In this episode Kevin Liu shares some of the interesting features that they have built by combining those technologies, as well as the challenges that they face in supporting the myriad workloads that are thrown at this layer of their data platform. Can you describe what role Trino and Iceberg play in Stripe's data architecture?

Data Lake 147

Data Lake High Quality Data Metadata Python 147

RandomTrees

SEPTEMBER 17, 2024

In the realm of big data and AI, managing and securing data assets efficiently is crucial. Databricks addresses this challenge with Unity Catalog, a comprehensive governance solution designed to streamline and secure data management across Databricks workspaces. Advantages of the Unity Catalog 1.

Data Governance 52

Data Governance Government Metadata Machine Learning 52

Data Engineering Podcast

SEPTEMBER 17, 2023

Linked data technologies provide a means of tightly coupling metadata with raw information. In this episode Brian Platz explains how JSON-LD can be used as a shared representation of linked data for building semantic data products. What is the overlap between knowledge graphs and "linked data products"?

Building 189

Building BI SQL Python 189

Data Engineering Podcast

DECEMBER 18, 2022

In this episode Abe Gong brings his experiences with the Great Expectations project and community to discuss the technical and organizational considerations involved in implementing these constraints to your data workflows. Atlan is the metadata hub for your data ecosystem. Missing data? Stale dashboards?

Metadata 130

Metadata Business Intelligence Data Lake BI 130

Monte Carlo

MAY 14, 2024

At its core, a table format is a sophisticated metadata layer that defines, organizes, and interprets multiple underlying data files. Table formats incorporate aspects like columns, rows, data types, and relationships, but can also include information about the structure of the data itself.

Data Lake 52

Data Lake Metadata Hadoop Data Governance 52

Data Engineering Podcast

NOVEMBER 26, 2023

One of the core complexities that needs to be addressed is the fractal set of integrations that need to be managed across the individual components. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. Data lakes are notoriously complex.

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Databand.ai

AUGUST 30, 2023

DataOps tools are software solutions designed to simplify and streamline the various aspects of data management and analytics, such as data ingestion, data transformation, data quality management, data cataloging, and data orchestration. In this article: Why Are DataOps Tools Important?

Data Cleanse 52

Data Cleanse Data Pipeline Data Ingestion Data Validation 52

Data Engineering Weekly

JANUARY 15, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. 2023 predictions from the panel are; Unified metadata becomes kingmaker.

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

Ascend.io

DECEMBER 11, 2023

This includes but is not limited to, job process sequencing, metadata synchronization, cataloging of data processing results, and managing the interdependencies of various data tasks. The goal is to minimize manual intervention and maximize the efficiency of data use. So, why is data orchestration a big deal?

Data Workflow 52

Data Workflow Data Pipeline Data Lake Data 52

Databand.ai

AUGUST 30, 2023

Unified DataOps represents a fresh approach to managing and synchronizing data operations across several domains, including data engineering, data science, DevOps, and analytics. Technical Challenges Choosing appropriate tools and technologies is critical for streamlining data workflows across the organization.

Data Governance 40

Data Governance Data Cleanse Government Data Pipeline 40

AltexSoft

JUNE 7, 2021

It doesn’t belong to the master-slave paradigm, being responsible for loading data into the cluster, describing how the data must be processed, and retrieving the output. Each Hadoop cluster has three functional layers: storage layer created by Hadoop’s native file system — HDFS, resource management layer represented by YARN, and.

Big Data Tools 98

Big Data Tools Hadoop Big Data Database-centric 98

AltexSoft

SEPTEMBER 21, 2023

From those home-made beginnings as Compass, Elasticsearch has matured into one of the leading enterprise search engines, standing among the top 10 most popular database management systems globally according to the Stack Overflow 2023 Developer Survey. Kibana is a data visualization tool that serves as the user interface for the Elastic Stack.

Engineering 52

Engineering NoSQL Programming Language Java 52

Data Engineering Podcast

SEPTEMBER 18, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Building 100

Building Metadata MongoDB MySQL 100

Databand.ai

DECEMBER 29, 2022

He believes in making data do the work through proper data management based on strategic rationale and business alignment. On LinkedIn, he posts frequently about big data, master data, data science, data management, and data storytelling.

BI 52

BI Consulting Data Science Data Governance 52

AltexSoft

MARCH 14, 2023

As the volume and complexity of data continue to grow, organizations seek faster, more efficient, and cost-effective ways to manage and analyze data. In recent years, cloud-based data warehouses have revolutionized data processing with their advanced massively parallel processing (MPP) capabilities and SQL support.

IT 59

IT Data Warehouse Data Governance Data Lake 59