Cloudera

SEPTEMBER 8, 2022

The management of data assets in multiple clouds is introducing new data governance requirements, and it is both useful and instructive to have a view from the TM Forum to help navigate the changes. . What’s new in data governance for telco? What about data fabric?

Telecommunication 77

Telecommunication Data Governance Government Portfolio 77

Data Engineering Podcast

JULY 12, 2021

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management You listen to this show to learn about all of the latest tools, patterns, and practices that power data engineering projects across every domain. Can you start by giving your definition of the modern data stack?

Designing 100

Designing Data Warehouse Data Governance Consulting 100

Knowledge Hut

MARCH 7, 2024

Two, it creates a commonality of data definitions, concepts, metadata and the like. The traditional data management and data warehouses, and the sequence of data transformation, extraction and migration- all arise a situation in which there are risks for data to become unsynchronized.

Big Data 80

Big Data Bytes Data Governance Raw Data 80

Knowledge Hut

MARCH 28, 2024

Data Engineer Career: Overview Currently, with the enormous growth in the volume, variety, and veracity of data generated and the will of large firms to store and analyze their data, data management is a critical aspect of data science. That’s where data engineers are on the go.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Data Engineering Weekly

MARCH 3, 2024

The article explains the challenges with frequent updates in the vanilla data lake setup and how Hudi allows for faster writes and guarantees atomicity, consistency, isolation, and durability (ACID). Part 1: Why did we need to build our own SIEM?

Data Engineering 67

Data Engineering Data Engineer Pipeline-centric Engineering 67

Databand.ai

JULY 19, 2023

Data lineage tools are not a new concept. However, their importance has grown significantly in recent years due to the increasing complexity of data architectures and the growing need for data governance and compliance. In this article: Why Are Data Lineage Tools Important?

Pipeline-centric 52

Pipeline-centric Data Governance Metadata Government 52

Data Engineering Podcast

APRIL 14, 2024

In this episode Oren Eini, CEO and creator of RavenDB, explores the nuances of relational vs. non-relational engines, and the strategies for designing a non-relational database. How have the evolution in data formats (e.g. When designing and building a database, what are the initial set of questions that need to be answered?

Non-relational Database 147

Non-relational Database Relational Database Database Designing 147

The Modern Data Company

FEBRUARY 15, 2023

Maintaining a clear strategy for gathering, storing, and labeling data is challenging. Limited resources: Data management has always been resource intensive, but not all organizations can maintain a full data team. Without suitable resources for company-wide data management, it’s easier to fall behind.

Government 98

Government Data Governance Raw Data Data 98

Towards Data Science

JUNE 8, 2023

This post will focus on the most common team ownership models including: data engineering, data reliability engineering, analytics engineering, data quality analysts, and data governance teams. Why is data quality ownership important? The governance team treats every team output as a data product.

Data Governance 78

Data Governance Government Generalist Data Engineering 78

Databand.ai

AUGUST 30, 2023

This includes defining roles and responsibilities related to managing datasets and setting guidelines for metadata management. To do this, you’ll need to demonstrate the value of data quality in terms of improved decision-making, increased operational efficiency, and reduced risk.

Building 52

Building Data Cleanse Data Governance Datasets 52

Databand.ai

AUGUST 30, 2023

Data Testing Tools: Key Capabilities and 6 Tools You Should Know Helen Soloveichik August 30, 2023 What Are Data Testing Tools? Data testing tools are software applications designed to assist data engineers and other professionals in validating, analyzing, and maintaining data quality.

Data Cleanse 52

Data Cleanse Data Validation Data Pipeline Datasets 52

Databand.ai

MAY 30, 2023

Striking a balance between these two aspects ensures that you have relevant, actionable insights from your data. Strategies for Improving Data Quality 1. Establish Data Governance Policies Creating data governance policies ensures uniformity in handling and managing data throughout your organization.

Data Cleanse 85

Data Cleanse Datasets Data Governance Data Validation 85

Monte Carlo

JUNE 6, 2023

This post will focus on the most common team ownership models including: data engineering, data reliability engineering, analytics engineering, data quality analysts, and data governance teams. Table of Contents Why is important to answer who is responsible for data quality?

Data Governance 98

Data Governance Government Data Data Engineering 98

Databand.ai

AUGUST 30, 2023

DataOps is a collaborative approach to data management that combines the agility of DevOps with the power of data analytics. It aims to streamline data ingestion, processing, and analytics by automating and integrating various data workflows.

Architecture 52

Architecture Data Ingestion Data Governance Data Cleanse 52

Data Engineering Podcast

JUNE 5, 2022

In this episode Sean Falconer explains the idea of a data privacy vault and how this new architectural element can drastically reduce the potential for making a mistake with how you manage regulated or personally identifiable information. How have the design and goals of the system changed or evolved over time?

Data Security 100

Data Security Metadata MongoDB Scala 100

The Modern Data Company

JULY 18, 2023

Data Products Data products encompass several key aspects that contribute to their effectiveness and value in addressing data challenges and delivering actionable insights. These aspects ensure that data products are well-designed, user-centric, and aligned with business goals.

Data Governance 69

Data Governance High Quality Data Government Data 69

Cloudera

JUNE 22, 2023

It ensures compliance with regulatory requirements while shifting non-sensitive data and workloads to the cloud. Its built-in intelligence automates common data management and data integration tasks, improves the overall effectiveness of data governance, and permits a holistic view of data across the cloud and on-premises environments.

Data Cleanse 66

Data Cleanse Data Governance Unstructured Data Cloud Storage 66

Monte Carlo

JANUARY 10, 2024

In this article, we present six intrinsic data quality techniques that serve as both compass and map in the quest to refine the inner beauty of your data. Data Profiling 2. Data Cleansing 3. Data Validation 4. Data Auditing 5. Data Governance 6. This is known as data governance.

Data Cleanse 52

Data Cleanse Data Engineering Data Engineer Engineering 52

Ascend.io

SEPTEMBER 5, 2023

In the fast-evolving landscape of cloud data solutions, Snowflake has consistently been at the forefront of innovation, offering enterprises sophisticated tools to optimize their data management. Customers can now process their native code inside Snowflake, essentially bringing the compute to the data.

IT 59

IT Scala Java Programming Language 59

Data Engineering Podcast

DECEMBER 31, 2023

Summary Working with financial data requires a high degree of rigor due to the numerous regulations and the risks involved in security breaches. In this episode Andrey Korchack, CTO of fintech startup Monite, discusses the complexities of designing and implementing a data platform in that sector.

Designing 130

Designing Data Lake High Quality Data SQL 130

The Modern Data Company

APRIL 15, 2024

Understands how different pieces of data (books) relate to each other, helping users find related information. Keeps the library orderly, adjusting as new data arrives. Tracks key performance indicators to understand the use and value of the data. Manages information about data access and monitors data quality.

Government 52

Government Metadata Data Data Lake 52

The Modern Data Company

JULY 31, 2023

Unified governance, particularly challenging in fragmented data stacks with numerous tools and data assets, is addressed through the infrastructure piece of Data Products. Thoughtworks, with its expertise in data strategies, has led successful data transformation initiatives.

Government 98

Government Data Governance Architecture Data Pipeline 98

The Modern Data Company

JULY 18, 2023

Data Products Data products encompass several key aspects that contribute to their effectiveness and value in addressing data challenges and delivering actionable insights. These aspects ensure that data products are well-designed, user-centric, and aligned with business goals.

Data Governance 52

Data Governance High Quality Data Government Data 52

The Modern Data Company

JULY 18, 2023

Data Products Data products encompass several key aspects that contribute to their effectiveness and value in addressing data challenges and delivering actionable insights. These aspects ensure that data products are well-designed, user-centric, and aligned with business goals.

Data Governance 52

Data Governance High Quality Data Government Data 52

Monte Carlo

FEBRUARY 21, 2023

Enter data fabric: a data management architecture designed to serve the needs of the business, not just those of data engineers. A data fabric is an architecture and associated data products that provide consistent capabilities across a variety of endpoints spanning multiple cloud environments.

Data Architecture 52

Data Architecture Architecture Metadata Unstructured Data 52

Monte Carlo

FEBRUARY 21, 2023

Enter data fabric: a data management architecture designed to serve the needs of the business, not just those of data engineers. A data fabric is an architecture and associated data products that provide consistent capabilities across a variety of endpoints spanning multiple cloud environments.

Data Architecture 52

Data Architecture Architecture Metadata Unstructured Data 52

AltexSoft

FEBRUARY 11, 2023

In the article, we explore the role of a data architect, discuss the responsibilities and required skills, and share what kind of companies may need such a specialist. What is a data architect? To get a better understanding of a data architect’s role, let’s clear up what data architecture is.

Data Architect 87

Data Architect Certification Generalist Big Data 87

Data Engineering Podcast

JULY 10, 2022

They also discuss how they have established a guild system for training and supporting data professionals in the organization. Can you describe the role of data at Riskified? Who/what are the primary consumers of the data that you are responsible for? What is the composition of your data roles? What does Riskified do?

Metadata 100

Metadata MongoDB MySQL Data Pipeline 100

Data Engineering Podcast

APRIL 5, 2021

She explains how the design of the platform is informed by the needs of managing data projects for large and small teams across her previous roles, how it integrates with your existing systems, and how it can work to bring everyone onto the same page. What portions of the data workflow is Atlan responsible for?

Data Warehouse 100

Data Warehouse Data Pipeline BI Metadata 100

Data Engineering Podcast

SEPTEMBER 21, 2020

Summary Data engineering is a constantly growing and evolving discipline. There are always new tools, systems, and design patterns to learn, which leads to a great deal of confusion for newcomers. If you hand a book to a new data engineer, what wisdom would you add to it? Which ones are you happy about?

Data Engineering 100

Data Engineering Data Engineer Engineering AWS 100

Databand.ai

JUNE 21, 2023

Eric Jones June 21, 2023 What Are Data Integrity Tools? Data integrity tools are software applications or systems designed to ensure the accuracy, consistency, and reliability of data stored in databases, spreadsheets, or other data storage systems. Data integrity tools are also crucial for regulatory compliance.

Data Integration 52

Data Integration Data Cleanse Data Validation Telecommunication 52

Data Engineering Podcast

NOVEMBER 9, 2020

He also shares the internal architecture, how he approached the design to make it accessible and easy to use, and how it autodiscovers the schemas and metadata for your source systems. Are you bogged down by having to manually manage data access controls, repeatedly move and copy data, and create audit reports to prove compliance?

Building 100

Building PostgreSQL BI Metadata 100

The Modern Data Company

FEBRUARY 2, 2024

This efficiency doesn’t just speed things up; it fundamentally changes how swiftly and effectively you can harness the power of your data assets. These data products are not mere collections of data; they are ready-to-use, trustworthy resources that fundamentally change the game.

Raw Data 75

Raw Data Utilities Government Data 75

Knowledge Hut

MARCH 26, 2024

Growing Technology Landscape: With the rapid advancements in technology, there's an increasing demand for professionals who can design and implement complex IT solutions. Role Level: Typically a senior-level position requiring extensive experience in architecture design and implementation across various domains and industries.

Amazon Web Services 52

Amazon Web Services Google Cloud Computer Science AWS 52

Data Engineering Podcast

OCTOBER 15, 2021

Summary The binding element of all data work is the metadata graph that is generated by all of the workflows that produce the assets used by teams across the organization. The DataHub project was created as a way to bring order to the scale of LinkedIn’s data needs. Can you describe what Acryl Data is and the story behind it?

Metadata 100

Metadata BI Data Warehouse Government 100

Data Engineering Podcast

DECEMBER 3, 2023

Andrei Tserakhau has dedicated his careeer to this problem, and in this episode he shares the lessons that he has learned and the work he is doing on his most recent data transfer system at DoubleCloud. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles.

Systems 130

Systems Designing Data Lake SQL 130

phData: Data Engineering

SEPTEMBER 19, 2023

With many data modeling methodologies and processes available, choosing the right approach can be daunting. This blog will guide you through the best data modeling methodologies and processes for your data lake, helping you make informed decisions and optimize your data management practices. What is a Data Lake?

Data Lake 52

Data Lake Process Metadata Data Warehouse 52