Databand.ai

JULY 19, 2023

Data lineage tools are not a new concept. However, their importance has grown significantly in recent years due to the increasing complexity of data architectures and the growing need for data governance and compliance. In this article: Why Are Data Lineage Tools Important?

Pipeline-centric 52

Pipeline-centric Data Governance Metadata Government 52

Data Engineering Podcast

JUNE 5, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking all of that information into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Go to dataengineeringpodcast.com/atlan today to learn more about how you can take advantage of active metadata and escape the chaos.

Data Security 100

Data Security Metadata MongoDB Scala 100

Data Engineering Podcast

JULY 10, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Metadata 100

Metadata MongoDB MySQL Data Pipeline 100

phData: Data Engineering

SEPTEMBER 19, 2023

This data can be structured, semi-structured, or unstructured and comes from various sources such as databases, IoT devices, log files, etc. What are Data Modeling Methodologies, and Why Are They Important for a Data Lake? They provide a framework for organizing and representing data elements, attributes, and relationships.

Data Lake 52

Data Lake Process Metadata Data Warehouse 52

Monte Carlo

FEBRUARY 21, 2023

Despite its prevalence, data can be messy, siloed, ungovernable, and inaccessible—especially to the non-technical employees who rely on it. Enter data fabric: a data management architecture designed to serve the needs of the business, not just those of data engineers. Table of Contents What is a data fabric?

Data Architecture 52

Data Architecture Architecture Metadata Unstructured Data 52

Monte Carlo

FEBRUARY 21, 2023

Despite its prevalence, data can be messy, siloed, ungovernable, and inaccessible—especially to the non-technical employees who rely on it. Enter data fabric: a data management architecture designed to serve the needs of the business, not just those of data engineers. Table of Contents What is a data fabric?

Data Architecture 52

Data Architecture Architecture Metadata Unstructured Data 52

Monte Carlo

MAY 18, 2023

Whether you’re a data scientist, data engineer, or business analyst, keeping track of your data’s origin, transformation, and movement is crucial for maintaining transparency, enforcing data governance, and ensuring data quality. Why do you need data lineage?

Metadata 64

Metadata Data Lake Government Data Governance 64

Databand.ai

AUGUST 30, 2023

This includes defining roles and responsibilities related to managing datasets and setting guidelines for metadata management. Data profiling: Regularly analyze dataset content to identify inconsistencies or errors. Some common data quality metrics include accuracy, completeness, consistency, timeliness, and uniqueness.

Building 52

Building Data Cleanse Data Governance Datasets 52

AltexSoft

FEBRUARY 11, 2023

In the article, we explore the role of a data architect, discuss the responsibilities and required skills, and share what kind of companies may need such a specialist. What is a data architect? To get a better understanding of a data architect’s role, let’s clear up what data architecture is.

Data Architect 87

Data Architect Certification Generalist Big Data 87

Databand.ai

AUGUST 30, 2023

Poor data quality: The lack of automation and data governance in legacy architectures can lead to data quality issues, such as incomplete, inaccurate, or duplicate data. They include the various databases, applications, APIs, and external systems from which data is collected and ingested.

Architecture 52

Architecture Data Ingestion Data Governance Data Cleanse 52

The Modern Data Company

APRIL 15, 2024

The term “Data Product” has become a buzzword, often misused or overstretched. But at its core, a Data Product is much more than just data. This blend ensures that a Data Product is informative, actionable, and adaptable to various needs.

Government 52

Government Metadata Data Data Lake 52

Data Engineering Podcast

APRIL 5, 2021

She explains how the design of the platform is informed by the needs of managing data projects for large and small teams across her previous roles, how it integrates with your existing systems, and how it can work to bring everyone onto the same page. What types of data assets (e.g. is Atlan designed to understand?

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100

Data Engineering Podcast

NOVEMBER 9, 2020

In this episode Grant Seward explains how he built Tree Schema to be an easy to use and cost effective option for organizations to build their data catalogs. He also shares the internal architecture, how he approached the design to make it accessible and easy to use, and how it autodiscovers the schemas and metadata for your source systems.

Building 100

Building PostgreSQL BI Metadata 100

Data Engineering Podcast

JUNE 19, 2022

Summary Metadata is the lifeblood of your data platform, providing information about what is happening in your systems. In order to level up their value a new trend of active metadata is being implemented, allowing use cases like keeping BI reports up to date, auto-scaling your warehouses, and automated data governance.

Metadata 130

Metadata MongoDB Scala MySQL 130

Snowflake

FEBRUARY 2, 2023

Or worse, until minor technical annoyances become major problems, and their fixes steal development time that could be spent improving performance or designing new features that could ultimately increase revenue and decrease costs. Learn more about how data.world’s cloud-native enterprise data catalog optimizes your cloud data migration.

Cloud 85

Cloud Data Data Governance Government 85

Cloudera

AUGUST 26, 2021

Ozone natively provides Amazon S3 and Hadoop Filesystem compatible endpoints in addition to its own native object store API endpoint and is designed to work seamlessly with enterprise scale data warehousing, machine learning and streaming workloads. Data ingestion through ‘s3’. Ozone Namespace Overview.

Data Science 107

Data Science Cloud Hadoop Metadata 107

Lyft Engineering

OCTOBER 3, 2023

High-level Concepts Check Definition   — The user-written configuration file defining the data quality contract and who to notify if it is breached. Check Result— The numeric measurement of data quality at a point in time, a boolean pass/fail value, and metadata about this run. It can be a fixed threshold or a statistical one.

Big Data 84

Big Data Metadata Data Warehouse Data 84

AltexSoft

AUGUST 29, 2023

The goal is to provide a comprehensive guide that can be a navigational tool for all specialists plotting their course in today’s data-driven world. What is a data lake? A data lake is a centralized repository designed to hold vast volumes of data in its native, raw format — be it structured, semi-structured, or unstructured.

Data Lake 52

Data Lake Architecture IT Amazon Web Services 52

phData: Data Engineering

DECEMBER 16, 2021

Perhaps one of the most crucial elements in deciding a data catalog that’s right for your business is exploring how your tool is designed when it comes to infrastructure. As a result, it is important to know what is happening to your data. Is your data just being scanned, or is it being moved?

Metadata 52

Metadata Datasets ETL Tools Cloud 52

Data Engineering Podcast

SEPTEMBER 23, 2018

Can you walk through the stages of an ideal lifecycle for data within the context of an organizations uses for it? What are some of the common mistakes that are made when designing a data architecture and how do they lead to failure?

Data Lake 100

Data Lake Data Warehouse Data Architecture Architecture 100

ThoughtSpot

AUGUST 23, 2023

The ThoughtSpot layup For the last 11 years, we have been working towards our vision: allowing business users to get data insights through an intuitive interface, with minimal effort; and meanwhile, giving data teams robust control over the semantics of data, governance, and security controls.

BI 87

BI Metadata Data Warehouse Data Governance 87

Silectis

DECEMBER 15, 2020

Integrated Security & Governance Features Ensuring that your data is protected and secure is one of our top priorities. We’ve designed Magpie to keep you from putting out fires like broken data pipelines with the built-in features mentioned above. Learn more about the technology behind Magpie here. Or your team?

Data Engineering 52

Data Engineering Data Engineer Engineering ETL Tools 52

LinkedIn Engineering

JUNE 2, 2023

Sharing economic data and commentary with media such as CNBC, The Wall Street Journal, NBC News, Financial Times, etc. Discoverability: Consumers must be able to easily discover and access the correct data sources for their needs, whether it is stored in a centralized repository or scattered throughout the wider data landscape.

Building 75

Building Banking Datasets Media 75

Monte Carlo

SEPTEMBER 20, 2022

Prefect understood the importance of data quality, but with a team of two data engineers and the needs of a much larger organization, setting up comprehensive data quality coverage was proving to be unattainable. Scaling data governance. This left Dylan’s team with a gap to fill.

Big Data 52

Big Data Data Warehouse Data Data Governance 52

Monte Carlo

MARCH 16, 2023

But in recent years, Randy and his colleagues have tackled one of their biggest challenges yet: building a data mesh , a type of decentralized data platform architecture that leverages a domain-driven, self-serve design. How do we instill the sense of quality and timeliness in the data?”

Software Engineer 52

Software Engineer Software Engineering Data Governance Government 52

Cloudera

AUGUST 4, 2021

And if data security tops IT concerns, data governance should be their second priority. Not only is it critical to protect data, but data governance is also the foundation for data-driven businesses and maximizing value from data analytics. But it’s still not easy. But it’s still not easy.

Data Governance 82

Data Governance Government Metadata Data Security 82

Monte Carlo

MARCH 16, 2023

But in recent years, Randy and his colleagues have tackled one of their biggest challenges yet: building a data mesh , a type of decentralized data platform architecture that leverages a domain-driven, self-serve design. How do we instill the sense of quality and timeliness in the data?”

Software Engineer 40

Software Engineer Software Engineering Data Governance Government 40

AltexSoft

MARCH 2, 2023

As the amount of enterprise data continues to surge, businesses are increasingly recognizing the importance of data governance — the framework for managing an organization’s data assets for accuracy, consistency, security, and effective use. Projections show that the data governance market will expand from $1.81

Data Governance 52

Data Governance Government Healthcare Programming 52

Cloudera

SEPTEMBER 15, 2022

The gathering in 2022 marked the sixteenth year for top data and analytics professionals to come to the MIT campus to explore current and future trends. A key area of focus for the symposium this year was the design and deployment of modern data platforms. See below. .

Data Lake 78

Data Lake Analytics Application Cloud Storage Architecture 78

The Modern Data Company

MARCH 7, 2023

Data governance can be a powerful agent in scaling the use and distribution of trusted data throughout the company. If you missed it, make sure to catch up on Part 1 – Data Timeliness. What Is Data Taxonomy? Data that is properly classified, catalogued, and tagged is usually well-governed data.

Data Governance 52

Data Governance Government Metadata Datasets 52

Cloudera

AUGUST 21, 2020

The governance aspect is perhaps even more important and businesses need to be able to understand where the data comes from. Data lineage, personally identifiable information or PPI and metadata all fall under a broad data governance banner which is critically important in terms of what needs to be protected and mapped out.

Banking 59

Banking Kafka Cloud Storage Government 59

dbt Developer Hub

JULY 2, 2023

Data Vault 2.0 is a data modeling technique designed to help scale large data warehousing projects. So why Data Vault? The Data Vault data model is designed using a few well-applied principles, and in practice, pools source data so it is available for use by all downstream consumers.

Cloud 52

Cloud Data Warehouse Data BI 52

Precisely

JANUARY 25, 2024

To achieve that, IT leaders must understand the various methods and tools available to them, and how to design a manageable roadmap that balances costs, risks, and benefits. You must carefully consider various mainframe functions, including security, system logs, metadata, and COBOL copybooks when moving to the new cloud platform.

Metadata 69

Metadata Data Governance Relational Database Government 69

Monte Carlo

MARCH 14, 2023

“We had some new use cases around data science that were hard to fit into our existing org. Up to this point, the BlaBlaCar data organization was designed around skillsets. They maintained a team of data analysts, a team of data engineers, and a team of data scientists.

Transportation 40

Transportation Government Google Cloud Architecture 40

François Nguyen

MARCH 7, 2021

In this article, Juan Sequada gives maybe one of the best definition of Data Mesh ” It is paradigm shift towards a distributed architecture that attempts to find an ideal balance between centralization and decentralization of metadata and data management.”

Government 130

Government Data Governance Data Metadata 130

Data Engineering Podcast

NOVEMBER 26, 2023

Trusted by the teams at Comcast and Doordash, Starburst delivers the adaptability and flexibility a lakehouse ecosystem promises, while providing a single point of access for your data and all your data governance allowing you to discover, transform, govern, and secure all in one place.

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Monte Carlo

NOVEMBER 14, 2023

Databricks announced that Delta tables metadata will also be compatible with the Iceberg format, and Snowflake has also been moving aggressively to integrate with Iceberg. It is designed to be easily queryable with SQL even for large analytic tables (we’re talking petabytes of data). How Apache Iceberg tables structure metadata.

Data Lake 64

Data Lake Metadata Data Warehouse SQL 64