Data Engineering Podcast

JUNE 5, 2022

In this episode Sean Falconer explains the idea of a data privacy vault and how this new architectural element can drastically reduce the potential for making a mistake with how you manage regulated or personally identifiable information. Atlan is the metadata hub for your data ecosystem.

Data Security 100

Data Security Metadata MongoDB Scala 100

Data Engineering Podcast

JULY 10, 2022

They also discuss how they have established a guild system for training and supporting data professionals in the organization. Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities.

Metadata 100

Metadata MongoDB MySQL Data Pipeline 100

Databand.ai

AUGUST 30, 2023

This includes defining roles and responsibilities related to managing datasets and setting guidelines for metadata management. Data profiling: Regularly analyze dataset content to identify inconsistencies or errors. Some common data quality metrics include accuracy, completeness, consistency, timeliness, and uniqueness.

Building 52

Building Data Cleanse Data Governance Datasets 52

Monte Carlo

FEBRUARY 21, 2023

Enter data fabric: a data management architecture designed to serve the needs of the business, not just those of data engineers. A data fabric is an architecture and associated data products that provide consistent capabilities across a variety of endpoints spanning multiple cloud environments.

Data Architecture 52

Data Architecture Architecture Metadata Unstructured Data 52

Monte Carlo

FEBRUARY 21, 2023

Enter data fabric: a data management architecture designed to serve the needs of the business, not just those of data engineers. A data fabric is an architecture and associated data products that provide consistent capabilities across a variety of endpoints spanning multiple cloud environments.

Data Architecture 52

Data Architecture Architecture Metadata Unstructured Data 52

The Modern Data Company

APRIL 15, 2024

Understands how different pieces of data (books) relate to each other, helping users find related information. Keeps the library orderly, adjusting as new data arrives. Tracks key performance indicators to understand the use and value of the data. Manages information about data access and monitors data quality.

Government 52

Government Metadata Data Data Lake 52

phData: Data Engineering

SEPTEMBER 19, 2023

With many data modeling methodologies and processes available, choosing the right approach can be daunting. This blog will guide you through the best data modeling methodologies and processes for your data lake, helping you make informed decisions and optimize your data management practices. What is a Data Lake?

Data Lake 52

Data Lake Process Metadata Data Warehouse 52

AltexSoft

FEBRUARY 11, 2023

In the article, we explore the role of a data architect, discuss the responsibilities and required skills, and share what kind of companies may need such a specialist. What is a data architect? To get a better understanding of a data architect’s role, let’s clear up what data architecture is.

Data Architect 87

Data Architect Certification Generalist Big Data 87

Databand.ai

AUGUST 30, 2023

DataOps is a collaborative approach to data management that combines the agility of DevOps with the power of data analytics. It aims to streamline data ingestion, processing, and analytics by automating and integrating various data workflows.

Architecture 52

Architecture Data Ingestion Data Governance Data Cleanse 52

Data Engineering Podcast

APRIL 5, 2021

She explains how the design of the platform is informed by the needs of managing data projects for large and small teams across her previous roles, how it integrates with your existing systems, and how it can work to bring everyone onto the same page. What portions of the data workflow is Atlan responsible for?

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100

Data Engineering Podcast

NOVEMBER 9, 2020

He also shares the internal architecture, how he approached the design to make it accessible and easy to use, and how it autodiscovers the schemas and metadata for your source systems. Are you bogged down by having to manually manage data access controls, repeatedly move and copy data, and create audit reports to prove compliance?

Building 100

Building PostgreSQL BI Metadata 100

AltexSoft

AUGUST 29, 2023

The goal is to provide a comprehensive guide that can be a navigational tool for all specialists plotting their course in today’s data-driven world. What is a data lake? A data lake is a centralized repository designed to hold vast volumes of data in its native, raw format — be it structured, semi-structured, or unstructured.

Data Lake 52

Data Lake Architecture IT Amazon Web Services 52

Data Engineering Podcast

SEPTEMBER 23, 2018

He also explains which layers are useful for the different members of the business, and which pitfalls to look out for along the path to a mature and flexible data platform. How do you define data curation? How does the size and maturity of a company affect the ways that they architect and interact with their data systems?

Data Lake 100

Data Lake Data Warehouse Data Architecture Architecture 100

Data Engineering Podcast

JUNE 19, 2022

Summary Metadata is the lifeblood of your data platform, providing information about what is happening in your systems. In order to level up their value a new trend of active metadata is being implemented, allowing use cases like keeping BI reports up to date, auto-scaling your warehouses, and automated data governance.

Metadata 130

Metadata MongoDB Scala MySQL 130

phData: Data Engineering

DECEMBER 16, 2021

Perhaps one of the most crucial elements in deciding a data catalog that’s right for your business is exploring how your tool is designed when it comes to infrastructure. As a result, it is important to know what is happening to your data. Is your data just being scanned, or is it being moved?

Metadata 52

Metadata Datasets ETL Tools Cloud 52

AltexSoft

MARCH 2, 2023

As the amount of enterprise data continues to surge, businesses are increasingly recognizing the importance of data governance — the framework for managing an organization’s data assets for accuracy, consistency, security, and effective use. What is data governance? billion in 2020 to $5.28

Data Governance 52

Data Governance Government Healthcare Programming 52

Monte Carlo

MAY 14, 2024

At its core, a table format is a sophisticated metadata layer that defines, organizes, and interprets multiple underlying data files. Table formats incorporate aspects like columns, rows, data types, and relationships, but can also include information about the structure of the data itself.

Data Lake 52

Data Lake Metadata Hadoop Data Governance 52

dbt Developer Hub

JULY 2, 2023

Data Vault 2.0 is a data modeling technique designed to help scale large data warehousing projects. So why Data Vault? The Data Vault data model is designed using a few well-applied principles, and in practice, pools source data so it is available for use by all downstream consumers.

Cloud 52

Cloud Data Warehouse Data BI 52

The Modern Data Company

MARCH 7, 2023

Data governance can be a powerful agent in scaling the use and distribution of trusted data throughout the company. If you missed it, make sure to catch up on Part 1 – Data Timeliness. What Is Data Taxonomy? Data that is properly classified, catalogued, and tagged is usually well-governed data.

Data Governance 52

Data Governance Government Metadata Datasets 52

François Nguyen

MARCH 7, 2021

In this article, Juan Sequada gives maybe one of the best definition of Data Mesh ” It is paradigm shift towards a distributed architecture that attempts to find an ideal balance between centralization and decentralization of metadata and data management.”

Government 130

Government Data Governance Data Metadata 130

Data Engineering Podcast

NOVEMBER 26, 2023

In this episode Tobias Macey shares his thoughts on the challenges that he is facing as he prepares to build the next set of architectural layers for his data platform to enable a larger audience to start accessing the data being managed by his team. Closing Announcements Thank you for listening!

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Databand.ai

AUGUST 30, 2023

DataOps , short for data operations, is an emerging discipline that focuses on improving the collaboration, integration, and automation of data processes across an organization. By improving data quality, organizations can generate more accurate and reliable insights from their data analytics processes.

Data Cleanse 52

Data Cleanse Data Pipeline Data Ingestion Data Validation 52

Databand.ai

AUGUST 30, 2023

Integrating these principles with data operation-specific requirements creates a more agile atmosphere that supports faster development cycles while maintaining high quality standards. Organizations need to establish data governance policies, processes, and procedures, as well as assign roles and responsibilities for data governance.

Data Governance 40

Data Governance Data Cleanse Government Data Pipeline 40

Cloudera

JUNE 29, 2023

To improve the way they model and manage risk, institutions must modernize their data management and data governance practices. Up your liquidity risk management game Historically, technological limitations made it difficult for financial institutions to accurately forecast and manage liquidity risk.

Data Architecture 83

Data Architecture Architecture Management Banking 83

Monte Carlo

NOVEMBER 14, 2023

Databricks announced that Delta tables metadata will also be compatible with the Iceberg format, and Snowflake has also been moving aggressively to integrate with Iceberg. It is designed to be easily queryable with SQL even for large analytic tables (we’re talking petabytes of data). How Apache Iceberg tables structure metadata.

Data Lake 64

Data Lake Metadata Data Warehouse SQL 64

Cloudera

MARCH 23, 2022

Iceberg is designed to be open and engine agnostic allowing datasets to be shared. The table information (such as schema, partition) is stored as part of the metadata (manifest) file separately, making it easier for applications to quickly integrate with the tables and the storage formats of their choice. Simplify data management .

Metadata 90

Metadata Data Architecture BI Machine Learning 90

Towards Data Science

MAY 19, 2023

Tech Target , defines a data silo as a repository of data controlled by one department or business unit and, therefore, not wholly or easily accessible by other departments within the same organisation. Absence or poor adoption of company-wide guidelines surrounding the creation and deployment of data products.

Pipeline-centric 105

Pipeline-centric Data Software Engineer Software Engineering 105

Precisely

JUNE 22, 2023

Historically, data problems were addressed using point solutions that solve specific data challenges. These types of solutions address data quality, master data management, and data governance, very often in isolation from one another. This new approach is designed to eliminate functional silos.

Data Integration 52

Data Integration Banking Data Governance Government 52

AltexSoft

AUGUST 22, 2022

Some would say that it’s not a big deal, however, these mixed environments have resulted in the complexities of managing disjointed data and business processes. With these challenges in enterprise data management, there has to be an approach to overcoming them, right? The solution is called a data fabric.

Architecture 52

Architecture Metadata Data Lake Machine Learning 52

Ascend.io

JULY 19, 2023

Process: Building Robust Systems from Data Governance to Product Publishing Just as a well-oiled machine needs precise processes, a successful data mesh needs to establish and adhere to processes regarding data governance, data curation, and data product publishing.

Data Governance 40

Data Governance Government Metadata Data 40

AltexSoft

NOVEMBER 10, 2021

Well, there’s a new phenomenon in data management that received the name of a data lakehouse. The pun being obvious, there’s more to that than just a new term: Data lakehouses combine the best features of both data lakes and data warehouses and this post will explain this all. Data warehouse.

Architecture 52

Architecture Data Lake Data Warehouse Metadata 52

Monte Carlo

AUGUST 25, 2023

This same structure aids in maintaining data quality and simplifies how users interact with and understand the data. One advantage of data warehouses is their integrated nature. As fully managed solutions, data warehouses are designed to offer ease of construction and operation.

Data Lake 52

Data Lake Data Warehouse Unstructured Data Raw Data 52

The Modern Data Company

JANUARY 17, 2023

It can improve the data quality, discoverability, and observability of the data in your CDP. A Data Operating System Can Save Your CDP DataOS is the only operating system on the market designed to work with every tool, app, and data source within your data ecosystem. The result?

Metadata 59

Metadata Government Data Lake Data 59

Snowflake

MARCH 30, 2023

3EJHjvm Once a business need is defined and a minimal viable product ( MVP ) is scoped, the data management phase begins with: Data ingestion: Data is acquired, cleansed, and curated before it is transformed. Data governance As a data management framework, feature stores must consider data privacy and data governance.

Engineering 83

Engineering Raw Data Data Science Scala 83

AltexSoft

NOVEMBER 23, 2021

All enterprise data is available through a single virtual layer for different users and a variety of use cases. They can design and perform whatever reports and analysis they need without worrying about a data format or where it resides. It’s not all rosy in the kingdom of data virtualization though.

Process 69

Process Data Lake Metadata Data Warehouse 69

Monte Carlo

AUGUST 30, 2023

You can enable the storage of test failures in your `dbt_project.yml` file by adding the following configuration: ```yaml test: +store_failures: true ``` Keep in mind that retaining these records does incur more storage cost and management resources, so make sure to clean the stored failure tables regularly. Also, remember data governance.

SQL 52

SQL Datasets Database High Quality Data 52

phData: Data Engineering

AUGUST 4, 2023

Database Storage The Snowflake architecture’s database storage layer organizes data into multiple tiny partitions, which are compressed and optimized internally. Snowflake stores and manages data in the cloud using a shared disk approach, which simplifies data management.

Cloud Storage 52

Cloud Storage Google Cloud Amazon Web Services Data Storage 52