Data Engineering Podcast

JULY 31, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Because of its centrality to your data systems it is valuable for debugging, governance, understanding context, and myriad other purposes.

IT 100

IT Metadata MongoDB MySQL 100

Netflix Tech

MARCH 10, 2021

ConsoleMe: A Central Control Plane for AWS Permissions and Access By Curtis Castrapel , Patrick Sanders , and Hee Won Kim At AWS re:Invent 2020, we open sourced two new tools for managing multi-account AWS permissions and access. They are the one-stop-shop for cloud permissions and access. This happened for us at Netflix.

AWS 100

AWS Accessible Accessibility Cloud 100

Data Engineering Podcast

AUGUST 28, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. And don’t forget to thank them for their continued support of this show! Email hosts@dataengineeringpodcast.com ) with your story.

Data Engineering 100

Data Engineering Data Engineer MongoDB Metadata 100

Cloudera

JUNE 7, 2022

Common security, governance, metadata, replication, and automation enable CDP to operate as an integrated system. As big data has evolved to cloud data and now to hybrid data, it has gotten more complex for businesses to access, use and create value from it. Clearly, hybrid data presents a massive opportunity and a tough challenge.

IT 108

IT Unstructured Data Data Architecture Government 108

Precisely

MAY 13, 2024

Integrated data catalog for metadata support As you build out your IT ecosystem, it’s important to leverage tools that have the capabilities to support forward-looking use cases. It synthesizes all the metadata around your organization’s data assets and arranges the information into a simple, easy-to-understand format.

Data Integration 52

Data Integration Metadata Amazon Web Services Data Governance 52

Data Council

JANUARY 21, 2021

Storing Cold Metadata with Alki (Dropbox) Dropbox shared insights into Alki , the petabyte-scale metadata store it designed for infrequently accessed metadata (“cold data”). Here's our January 2021 roundup of links from across the web that could be relevant to you: 1.

Metadata 52

Metadata AWS Cloud Database 52

Cloudera

APRIL 18, 2024

In those discussions, it was clear that everyone understood the need to treat data estates more cohesively as a whole—that means bringing more attention to security, data governance, and metadata management, the latter of which has become increasingly popular. Do I trust my data to be ready for use in AI?

Metadata 105

Metadata Government Data Governance Data 105

AltexSoft

AUGUST 29, 2023

Such an object storage model allows metadata tagging and incorporating unique identifiers, streamlining data retrieval and enhancing performance. The term was coined by James Dixon , Back-End Java, Data, and Business Intelligence Engineer, and it started a new era in how organizations could store, manage, and analyze their data.

Data Lake 52

Data Lake Architecture IT Amazon Web Services 52

Data Engineering Podcast

NOVEMBER 13, 2022

Shane Gibson co-founded AgileData to make analytics accessible to companies of all sizes. Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities.

Building 130

Building Metadata MongoDB MySQL 130

Ascend.io

NOVEMBER 28, 2023

This includes job process sequencing, metadata synchronization, cataloging of data processing results, triggering data loads and quality checks, detecting when one task is done and triggering another, and setting the timing of scripts and system commands. In contrast, data pipeline orchestration is a more targeted approach.

Data Pipeline 52

Data Pipeline IT Data Metadata 52

Data Engineering Podcast

OCTOBER 30, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Summary Business intelligence has grown beyond its initial manifestation as dashboards and reports.

Business Intelligence 130

Business Intelligence Metadata MongoDB MySQL 130

Data Engineering Podcast

DECEMBER 4, 2022

The typical conception of how it is accessed is through a web or desktop application running on a powerful laptop. This opens the door for busy employees to access and analyze their company information away from their desk, but it has the more powerful effect of bringing first-class support to companies operating in mobile-first economies.

Business Intelligence 100

Business Intelligence Metadata BI MongoDB 100

Towards Data Science

MAY 7, 2024

meetings with sales reps from all of the SaaS data integration tooling companies and are granted 14 day access to try their wares. This will help create an overall picture of logging, metadata, latency, CDC, and all the other things that come with a pipeline. Gather the team together and determine what these things are.

ETL Tools 84

ETL Tools Metadata Data Integration Data Science 84

Cloudera

APRIL 2, 2024

We understand the importance of protecting sensitive government data from unauthorized access, and we have implemented robust security measures to safeguard this information. Purpose Built for Security Here at Cloudera, we focus on ensuring that our platform meets the highest standards of security and regulatory compliance.

Designing 96

Designing Process Government Metadata 96

François Nguyen

MARCH 22, 2021

Just an illustration – not the truth and you certainly can do it with other technologies. TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. TL;DR After setting up and organizing the teams, we are describing 4 topics to make data mesh a reality. It looks nice but it is really not !

Architecture 130

Architecture Technology Google Cloud Metadata 130

Towards Data Science

FEBRUARY 6, 2024

Accessing Operational Data I used to connect to views in transactional databases or APIs offered by operational systems to request the raw data. Accessibility : I could easily request access to these data products. Metadata update Data products need to be understandable. Does it sound familiar?

Systems 83

Systems Raw Data Metadata Data Cleanse 83

Netflix Tech

NOVEMBER 14, 2023

Input : List of source tables and required processing mode Output : Psyberg identifies new events that have occurred since the last high watermark (HWM) and records them in the session metadata table. The session metadata table can then be read to determine the pipeline input. Audit Run various quality checks on the staged data.

Metadata 97

Metadata Data Pipeline Scala Data Workflow 97

Monte Carlo

JANUARY 5, 2024

In this article, we’ll peel back the 5 layers that make up data lakehouse architecture: data ingestion, data storage, metadata, API, and data consumption, understand the expanded opportunities a data lakehouse opens up for generative AI, and how to maintain data quality throughout the pipeline with data observability. Metadata layer 4.

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

Monte Carlo

JANUARY 5, 2024

In this article, we’ll peel back the 5 layers that make up data lakehouse architecture: data ingestion, data storage, metadata, API, and data consumption, understand the expanded opportunities a data lakehouse opens up for generative AI, and how to maintain data quality throughout the pipeline with data observability. Metadata layer 4.

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

The Modern Data Company

APRIL 15, 2024

Manages information about data access and monitors data quality. It’s a combination of the data itself, the metadata (information about the data), code (the set of instructions on how to process or analyze the data), and infrastructure (the technological framework that supports it all). What Exactly Is a Data Product?

Government 52

Government Metadata Data Data Lake 52

Snowflake

APRIL 29, 2024

This strategy must incorporate business impact analyses as well as backup and recovery plans in the event of a security incident or loss of access to data. Access Control: Snowflake allows customers to define granular permissions for user roles, minimizing the risk of unauthorized access to sensitive data.

Transportation 83

Transportation Data Governance Government Consulting 83

Netflix Tech

MARCH 10, 2023

Studio applications use this service to store their media assets, which then goes through an asset cycle of schema validation, versioning, access control, sharing, triggering configured workflows like inspection, proxy generation etc. This pattern grows over time when we need to access and update the existing assets metadata.

Management 94

Management Kafka Metadata Media 94

Knowledge Hut

MARCH 7, 2024

Two, it creates a commonality of data definitions, concepts, metadata and the like. As far as Big Data is concerned, data security should be high on their priorities as most modern businesses are vulnerable to fake data generation, especially if cybercriminals have access to the database of a business.

Big Data 98

Big Data Bytes Data Governance Raw Data 98

Monte Carlo

JANUARY 29, 2024

Snowflake recently announced the launch of Snowflake Horizon —a unified collection of compliance, security, privacy, interoperability, and access capabilities—that make it easy for organizations to govern and take action on their data and apps across clouds, teams, partners, and customers. Volume: Are there too many or too few rows?

Metadata 64

Metadata High Quality Data Data Pipeline Machine Learning 64

Precisely

JUNE 6, 2023

Tendü Yogurtçu offered that organizations today are viewing data as a product, and they are working to streamline access to both data and metadata for faster insights that help business users deliver tangible value. The group kicked off the session by exchanging ideas about what it means to have a modern data architecture.

Data Architecture 52

Data Architecture Architecture Metadata Data Lake 52

ThoughtSpot

MAY 31, 2023

This includes services that: Manage and monitor the tenant-specific resources—this does not include access to tenant data Maintains indexed data to serve as your application home page. This multi-tenant service isolates the tenant metadata index, authorizing and filtering the search answer requests from every tenant.

Data Security 86

Data Security Metadata Transportation Data Warehouse 86

Data Engineering Weekly

MARCH 24, 2024

The APIs support emitting unstructured log lines and typed metadata key-value pairs (per line). The extracted key-value pairs are written to the line’s metadata. Query clusters support interactive and bulk queries on one or more log streams with predicate filters on log text and metadata.

Data Engineering 62

Data Engineering Data Engineer Engineering Metadata 62

Cloudera

JUNE 7, 2023

By now, almost everyone across the tech landscape has heard of the Zero Trust (ZT) security model, which assumes that every device, application, or user attempting to access a network is not to be trusted (see NIST definitions below). The data is protected but it is also accessible by the people who need it. It’s a win-win.

Government 79

Government Metadata Data Lake Data 79

Knowledge Hut

APRIL 4, 2024

You can think of a cookbook as a collection of all the recipes, files, characteristics, and metadata you'll need to implement a specific scenario. Chef is an open-source configuration management tool developed by Opscode to solve the problem of manual and repetitive infrastructure management tasks.

Architecture 52

Architecture Amazon Web Services Metadata AWS 52

Netflix Tech

JUNE 1, 2023

It also included metadata about ads, such as ad placement and impression-tracking events. A Kafka consumer retrieved the playback manifests with ad metadata and simulated a device playing the content and triggering the impression-tracking events. As we were gearing up for launch, we wanted to ensure it would go as smoothly as possible.

Algorithm 136

Algorithm Metadata Kafka Systems 136

Data Engineering Weekly

DECEMBER 29, 2022

Data Catalog as a passive web portal to display metadata requires significant rethinking to adopt modern data workflow, not just adding “modern” in its prefix. I know that is an expensive statement to make😊 To be fair, I’m a big fan of data catalogs, or metadata management , to be precise. The modern(?)

Metadata 62

Metadata Data Warehouse ETL Tools Data Workflow 62

Ascend.io

FEBRUARY 28, 2024

Through the design and maintenance of efficient data pipelines , data engineers facilitate the seamless flow and accessibility of data for AI processing. By ensuring data is well-organized and readily accessible, data engineers empower AI systems to deliver targeted and effective solutions.

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

Monte Carlo

MAY 18, 2023

You may even already have access to one of them. No-code Editor : It’s possible to augment the lineage captured from machine metadata with a drag and drop no-code editor. One reviewer said, “I’m in the process of setting up OpenMetadata in our org, ingesting metadata from Athena, Superset and Dagster.

Metadata 64

Metadata Data Lake Government Data Governance 64

Monte Carlo

MARCH 4, 2024

To do this, data observability solutions like Monte Carlo will piece together information from query logs, data aggregates and system metadata. All of this is to say that how table formats are structured, and specifically the metadata, impacts how they can be monitored and observed. And what about wildcard tables?

Metadata 52

Metadata Data Lake Data Warehouse Data Engineering 52

Data Engineering Weekly

JUNE 29, 2023

The quest to simplify data access is there forever, but with the advancement in LLM, I think it will become a reality. Databricks and Snowflake are better places to index the data and its metadata to enable natural language query capabilities. On top of it, it does support access control for queries and maintains the permission model.

Data Engineering 76

Data Engineering Data Engineer Google Cloud Engineering 76

Edureka

MAY 6, 2024

MLOps is an emerging discipline that aims to unify and streamline the machine learning system development (Dev) and operations (Ops) lifecycle. It involves collaboration between data scientists, ML engineers, and IT professionals to automate and optimize the end-to-end process of building, deploying, and maintaining machine learning applications.

Machine Learning 52

Machine Learning Metadata Programming Language Healthcare 52

Cloudera

AUGUST 10, 2021

Impala Row Filtering to set access policies for rows when reading from a table. Atlas / Kafka integration provides metadata collection for Kafa producers/consumers so that consumers can manage, govern, and monitor Kafka metadata and metadata lineage in the Atlas UI. Figure 1: sales group SELECT access.

Cloud 96

Cloud Kafka Metadata SQL 96