Snowflake

APRIL 29, 2024

This strategy must incorporate business impact analyses as well as backup and recovery plans in the event of a security incident or loss of access to data. Access Control: Snowflake allows customers to define granular permissions for user roles, minimizing the risk of unauthorized access to sensitive data.

Transportation 80

Transportation Data Governance Government Consulting 80

Edureka

MAY 6, 2024

Whether you are a newbie or an experienced individual, if you want to explore more about the concepts of MLOPS, then you just click on the right blog. But before we begin, Let’s have a look at what we will be covering in this blog: What is MLOPS? Proper metadata management helps in understanding, grouping, and sorting data.

Machine Learning 52

Machine Learning Metadata Programming Language Healthcare 52

Cloudera

AUGUST 10, 2021

Impala Row Filtering to set access policies for rows when reading from a table. We expand on this feature later in this blog. Atlas / Kafka integration provides metadata collection for Kafa producers/consumers so that consumers can manage, govern, and monitor Kafka metadata and metadata lineage in the Atlas UI.

Cloud 96

Cloud Kafka Metadata SQL 96

Netflix Tech

MARCH 10, 2023

Studio applications use this service to store their media assets, which then goes through an asset cycle of schema validation, versioning, access control, sharing, triggering configured workflows like inspection, proxy generation etc. This pattern grows over time when we need to access and update the existing assets metadata.

Management 94

Management Kafka Metadata Media 94

Data Engineering Podcast

SEPTEMBER 4, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!

IT 100

IT Metadata MongoDB MySQL 100

Cloudera

JUNE 7, 2023

By now, almost everyone across the tech landscape has heard of the Zero Trust (ZT) security model, which assumes that every device, application, or user attempting to access a network is not to be trusted (see NIST definitions below). The data is protected but it is also accessible by the people who need it.

Government 78

Government Metadata Data Lake Data 78

Netflix Tech

JUNE 1, 2023

In this blog post, we’ll discuss the methods we used to ensure a successful launch, including: How we tested the system Netflix technologies involved Best practices we developed Realistic Test Traffic Netflix traffic ebbs and flows throughout the day in a sinusoidal pattern. Basic with ads was launched worldwide on November 3rd.

Algorithm 136

Algorithm Metadata Kafka Systems 136

Cloudera

MARCH 2, 2023

It allows multiple data processing engines, such as Flink, NiFi, Spark, Hive, and Impala to access and analyze data in simple, familiar SQL tables. The post Streaming Ingestion for Apache Iceberg With Cloudera Stream Processing appeared first on Cloudera Blog.

Process 112

Process SQL Kafka Database 112

Databand.ai

JULY 19, 2023

Metadata Management Metadata, or ‘data about data’, is a crucial component of data management. Data lineage tools provide robust metadata management capabilities, allowing businesses to capture, store, and manage metadata associated with their data. This fosters a data-driven culture within the organization.

Pipeline-centric 52

Pipeline-centric Data Governance Metadata Government 52

Cloudera

NOVEMBER 29, 2023

It simplifies recording model parameters, metadata, and metrics ensuring transparent bookkeeping. You can track each iteration, compare changes, and ensure that you always have access to the version that suits your needs. The post Announcing General Availability of Model Registry appeared first on Cloudera Blog.

Machine Learning 82

Machine Learning Metadata Cloud Project 82

Cloudera

AUGUST 26, 2021

In this blog post, we will ingest a real world dataset into Ozone, create a Hive table on top of it and analyze the data to study the correlation between new vaccinations and new cases per country using a Spark ML Jupyter notebook in CML. Learn more about the impacts of global data sharing in this blog, The Ethics of Data Exchange.

Data Science 107

Data Science Cloud Hadoop Metadata 107

Cloudera

JANUARY 19, 2024

As described in our recent blog post , an SQL AI Assistant has been integrated into Hue with the capability to leverage the power of large language models (LLMs) for a number of SQL tasks. This blog post aims to help you understand what you can do to get started with generative AI assisted SQL using Hue image version ​​2023.0.16.0

SQL 71

SQL Data Warehouse AWS Architecture 71

phData: Data Engineering

SEPTEMBER 19, 2023

This blog will guide you through the best data modeling methodologies and processes for your data lake, helping you make informed decisions and optimize your data management practices. Data Governance and Security By defining data models, organizations can establish policies, access controls, and security measures to protect sensitive data.

Data Lake 52

Data Lake Process Metadata Data Warehouse 52

Cloudera

MARCH 3, 2022

Data analytics and machine learning can become a business and a compliance risk if data security, governance, lineage, metadata management, and automation are not holistically applied across the entire data lifecycle and all environments. Find out more about unlocking the potential of your data in this whitepaper.

Data Governance 95

Data Governance Government Metadata Medical 95

Cloudyard

DECEMBER 7, 2023

Read Time: 3 Minute, 0 Second In this blog post, we will delve into Snowflake’s Listing feature, which operates on the data sharing provider and consumer model. Listing in Consumer Click on Get and it will start replicating the data from provider Account. Data Replication Once done we will get the mail.

Metadata 52

Metadata SQL Data Cloud 52

Data Engineering Weekly

MARCH 10, 2024

Google: Croissant- a metadata format for ML-ready datasets Google Research introduced Croissant, a new metadata format designed to make datasets ML-ready by standardizing the format, facilitating easier use in machine learning projects. Pradheep Arjunan - Shared insights on AZ's journey from on-prem to the cloud data warehouses.

Data Engineering 52

Data Engineering Data Engineer Engineering Datasets 52

Databand.ai

AUGUST 30, 2023

This includes defining roles and responsibilities related to managing datasets and setting guidelines for metadata management. This may include tasks such as data profiling, data cleansing, and metadata management. Data profiling: Regularly analyze dataset content to identify inconsistencies or errors.

Building 52

Building Data Cleanse Data Governance Datasets 52

Cloudyard

APRIL 4, 2024

This blog post explores how Snowpark, a powerful tool for data processing within Snowflake, can be used to perform RFM segmentation and unlock actionable customer insights. However, the volume of daily transaction data poses challenges in effectively segmenting customers and optimizing engagement.

Retail 40

Retail Data Ingestion Metadata Datasets 40

Workfall

AUGUST 23, 2023

Inside this directory, we’ll create a file named ‘manifest.json’ This file will play a crucial role in defining the metadata, permissions, and settings for our extension. Once we access the browser console, we’ll see the weather data containing all the necessary information provided by the OpenWeather API.

Metadata 83

Metadata Coding Project Utilities 83

Netflix Tech

NOVEMBER 13, 2023

In this blog post, we will introduce speech and music detection as an enabling technology for a variety of audio applications in Film & TV, as well as introduce our speech and music activity detection (SMAD) system which we recently published as a journal article in EURASIP Journal on Audio, Speech, and Music Processing.

Datasets 89

Datasets Metadata Algorithm Architecture 89

Cloudera

JUNE 25, 2021

This blog post outlines detailed step by step instructions to perform Hive Replication from an on-prem CDH cluster to a CDP Public Cloud Data Lake. In order to copy or migrate data from CDH cluster to CDP Data Lake cluster, the on-prem CDH cluster should be able to access the CDP cloud storage. Hadoop SQL Policies overview.

Cloud 68

Cloud Data Lake Cloud Storage Metadata 68

Cloudera

MARCH 17, 2023

Hopefully this blog will give ChatGPT an opportunity to learn and correct itself while counting towards my 2023 contribution to social good. The one key component that is missing is a common, shared table format, that can be used by all analytic services accessing the lakehouse data. I thought this was a fairly comprehensive list.

Education 60

Education Unstructured Data Data Lake Data Warehouse 60

Cloudera

MARCH 14, 2023

In our previous DataFlow Designer blog post , we introduced you to the new user interface and highlighted its key capabilities. In this blog post we will put these capabilities in context and dive deeper into how the built-in, end-to-end data flow life cycle enables self-service data pipeline development.

Data Pipeline 80

Data Pipeline Designing Kafka Metadata 80

Data Engineering Podcast

MARCH 22, 2021

Your host is Tobias Macey and today I’m interviewing Raghu Murthy about his recent work of making change data capture more accessible and maintainable Interview Introduction How did you get involved in the area of data management? e.g. APIs and third party data sources How can we integrage CDC into metadata/lineage tooling?

Data Warehouse 100

Data Warehouse Metadata Data Lake Hadoop 100

Cloudera

JULY 15, 2021

This blog post provides an overview of best practice for the design and deployment of clusters incorporating hardware and operating system configuration, along with guidance for networking and security as well as integration with existing enterprise infrastructure. Introduction and Rationale. Networking .

Architecture 96

Architecture Cloud Kafka Hadoop 96

Data Engineering Podcast

JANUARY 23, 2022

You can observe your pipelines with built in metadata search and column level lineage. With complete API access, a user-friendly interface, and automated yet flexible alerting, you’ve got everything you need to establish and maintain trust in your data. What have been some of the recurring themes that are discussed in those events?

Building 100

Building Management Data Pipeline Metadata 100

Cloudera

JULY 14, 2023

In this blog, I will describe a few strategies one could undertake for various use cases. They simply read the underlying data (not even full read, they just read the parquet headers) and create corresponding Iceberg metadata files. This will be discussed in a later blog. The underlying data file locations will not change.

Metadata 69

Metadata Data Warehouse Big Data Ecosystem Java 69

Rockset

NOVEMBER 14, 2022

In this blog, we will describe the SSB configuration, results and performance enhancements. RocksDB provides an in-memory cache, called the block cache, that keeps frequently accessed data blocks in memory. The metadata lookup operation takes a significant proportion of the working set memory, impacting query latency.

Metadata 52

Metadata Datasets Cloud Accessible 52

Cloudera

SEPTEMBER 23, 2020

Comprehensive auditing is provided to enable enterprises to effectively and efficiently meet their compliance requirements by auditing access and other types of operations across OpDB (through HBase). User, business classification of asset accessed. Policy outcome (access or deny). Policy outcome (access or deny).

Database 80

Database Data Lake Metadata Java 80

Cloudera

DECEMBER 10, 2020

In a multi-tenant environment, many users access the same data sources. The very high degree of compression built into these file formats reduces storage space and time to access storage. These techniques range from distributing concurrent query for overall throughput to metadata caching, data caching, and results caching.

Data Warehouse 98

Data Warehouse Cloud Storage Metadata Data 98

LinkedIn Engineering

DECEMBER 19, 2023

Figure 3: Generalized rolling upgrade deployment flow Namenode deployment overview The namenode is the central component of HDFS and is responsible for storing the metadata information about files and directories in the HDFS cluster. This metadata includes the namespace, file permissions, and the mapping of data blocks to datanodes.

Big Data 92

Big Data Hadoop Metadata Data 92

Cloudera

AUGUST 3, 2021

Access to innovation: The open source community drives change, innovation, and feature functionality at a higher rate than any one organization, even with heavy R&D investment. . Flexibility and choice: Organizations always have access to their data, and don’t get locked-in by any vendor. Keep data lineage secure and governed.

Manufacturing 107

Manufacturing Retail Entertainment Government 107

phData: Data Engineering

OCTOBER 12, 2022

In this blog post, we’ll dive into six of the most powerful dbt Cloud features and many others that you probably don’t know about. Data Health with the Metadata API As mentioned earlier, dbt Cloud provides an easy-to-query metadata API that contains all the metadata about a dbt project and the various runs.

Cloud 52

Cloud Metadata SQL Data Warehouse 52

AltexSoft

FEBRUARY 11, 2023

This suggests that today, there are many companies that face the need to make their data easily accessible, cleaned up, and regularly updated. If you are not familiar with the above-mentioned concepts, we suggest you to follow the links above to learn more about each of them in our blog posts.

Data Architect 87

Data Architect Certification Generalist Big Data 87

Cloudera

JUNE 7, 2022

Common security, governance, metadata, replication, and automation enable CDP to operate as an integrated system. As big data has evolved to cloud data and now to hybrid data, it has gotten more complex for businesses to access, use and create value from it. We believe that data can make what is impossible today, possible tomorrow.

IT 106

IT Unstructured Data Data Architecture Government 106

Cloudera

SEPTEMBER 29, 2022

The data product is not just the data itself, but a bunch of metadata that surrounds it — the simple stuff like schema is a given. There are tons of blogs/videos etc about data mesh. The post The Top Three Entangled Trends in Data Architectures: Data Mesh, Data Fabric, and Hybrid Architectures appeared first on Cloudera Blog.

Architecture 73

Architecture Data Architecture Metadata Data Warehouse 73

DataKitchen

AUGUST 9, 2021

The domain requires a team that creates/updates/runs the domain, and we can’t forget metadata: catalogs, lineage, test results, processing history, etc., …. The domain is accessed through its external interfaces, which can be easily remembered as the four “W’s” and an “H:”. How does one get access to a domain?

Pharmaceutical 98

Pharmaceutical Raw Data Data Lake Data 98