Rockset

MARCH 18, 2024

In this blog, we’ll discuss how RAG fits into the paradigm of real-time data processing and show an example product recommendation application using both Kafka and Flink on Confluent Cloud together with Rockset. These additional inputs are referred to as metadata filtering.

Cloud 64

Cloud Building Metadata Kafka 64

Zalando Engineering

JUNE 30, 2020

Our Engineering Blog was launched in June 2020 after a long break of the previous tech blog. What customizations we applied to design the blog and the publishing process. Static Site Generator Our previous tech blog used a CMS which only a limited number of people had access to. It's actively developed.

Engineering 40

Engineering Bytes AWS Python 40

Cloudera

JUNE 7, 2023

By now, almost everyone across the tech landscape has heard of the Zero Trust (ZT) security model, which assumes that every device, application, or user attempting to access a network is not to be trusted (see NIST definitions below). The data is protected but it is also accessible by the people who need it.

Government 79

Government Metadata Data Lake Data 79

Data Engineering Podcast

NOVEMBER 20, 2022

Summary The majority of blog posts and presentations about data engineering and analytics assume that the consumers of those efforts are internal business users accessing an environment controlled by the business. Atlan is the metadata hub for your data ecosystem. Atlan is the metadata hub for your data ecosystem.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Netflix Tech

NOVEMBER 14, 2023

By Abhinaya Shetty , Bharath Mummadisetty This blog post will cover how Psyberg helps automate the end-to-end catchup of different pipelines, including dimension tables. The session metadata table can then be read to determine the pipeline input. Metadata Recording : Metadata is persisted for traceability.

Metadata 97

Metadata Data Pipeline Scala Data Workflow 97

Tweag

OCTOBER 25, 2023

Maintainers also add all sorts of metadata such as the name, the version, the license, the homepage, the description, and so on, which could be used to make an even richer graph. nixtract ’s goal is to make it accessible in a way that’s easier to consume. nixtract ’s goal is to make it accessible in a way that’s easier to consume.

Metadata 107

Metadata Accessible Accessibility Python 107

Netflix Tech

MARCH 10, 2023

Studio applications use this service to store their media assets, which then goes through an asset cycle of schema validation, versioning, access control, sharing, triggering configured workflows like inspection, proxy generation etc. This pattern grows over time when we need to access and update the existing assets metadata.

Management 94

Management Kafka Metadata Media 94

LinkedIn Engineering

JULY 19, 2023

Open source data lakehouse deployments are built on the foundations of compute engines (like Apache Spark, Trino, Apache Flink), distributed storage (HDFS, cloud blob stores), and metadata catalogs / table formats (like Apache Iceberg, Delta, Hudi, Apache Hive Metastore). Tables (not files/blobs) are the only API abstraction for end-users.

Big Data 78

Big Data Data Management Management Metadata 78

Snowflake

NOVEMBER 2, 2023

To give customers flexibility for how they fit Snowflake into their architecture, Iceberg Tables can be configured to use either Snowflake or an external service like AWS Glue as the tables’s catalog to track metadata, with an easy one-line SQL command to convert to Snowflake in a metadata-only operation.

Data Lake 89

Data Lake Data Warehouse Cloud Unstructured Data 89

Cloudera

AUGUST 10, 2021

Impala Row Filtering to set access policies for rows when reading from a table. We expand on this feature later in this blog. Atlas / Kafka integration provides metadata collection for Kafa producers/consumers so that consumers can manage, govern, and monitor Kafka metadata and metadata lineage in the Atlas UI.

Cloud 96

Cloud Kafka Metadata SQL 96

Cloudera

JANUARY 19, 2024

As described in our recent blog post , an SQL AI Assistant has been integrated into Hue with the capability to leverage the power of large language models (LLMs) for a number of SQL tasks. This blog post aims to help you understand what you can do to get started with generative AI assisted SQL using Hue image version ​​2023.0.16.0

SQL 71

SQL Data Warehouse AWS Architecture 71

DoorDash Engineering

DECEMBER 5, 2023

Orchestrator service triggers an API call against the GitHub repo for Kafka Topics, creating a pull request, or PR, for a new topic and corresponding access control list (ACL) entry. All Topics, even those without any messages, include some metadata that is backed up to disk.

Kafka 91

Kafka Programming Language Metadata Architecture 91

Cloudera

MARCH 2, 2023

It allows multiple data processing engines, such as Flink, NiFi, Spark, Hive, and Impala to access and analyze data in simple, familiar SQL tables. The post Streaming Ingestion for Apache Iceberg With Cloudera Stream Processing appeared first on Cloudera Blog.

Process 112

Process SQL Kafka Database 112

Snowflake

AUGUST 22, 2023

In an earlier blog post, I discussed how the notion of a product emerges from a business need. If the data has specific access, usage and security requirements, these must be built into the data product. Finally, not to be overlooked are the metadata and documentation required to ensure the product can easily be used.

Metadata 71

Metadata Data AWS Business Analyst 71

Netflix Tech

JUNE 1, 2023

In this blog post, we’ll discuss the methods we used to ensure a successful launch, including: How we tested the system Netflix technologies involved Best practices we developed Realistic Test Traffic Netflix traffic ebbs and flows throughout the day in a sinusoidal pattern. Basic with ads was launched worldwide on November 3rd.

Algorithm 136

Algorithm Metadata Kafka Systems 136

Data Engineering Podcast

SEPTEMBER 4, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!

IT 100

IT Metadata MongoDB MySQL 100

Cloudera

MARCH 3, 2022

Data analytics and machine learning can become a business and a compliance risk if data security, governance, lineage, metadata management, and automation are not holistically applied across the entire data lifecycle and all environments. Find out more about unlocking the potential of your data in this whitepaper.

Data Governance 96

Data Governance Government Metadata Medical 96

Tweag

MAY 16, 2023

Since the previous stable version ( 0.3.1 ), efforts have been made on three principal fronts: tooling (in particular the language server), the core language semantics (contracts, metadata, and merging), and the surface language (the syntax and the stdlib). The | symbol attaches metadata to fields.

MySQL 131

MySQL Metadata Data Validation Coding 131

Databand.ai

AUGUST 30, 2023

Learn more in our detailed guide to data reliability testing Metadata Management Metadata management is the process of organizing, maintaining, and using metadata to improve the quality, consistency, and usability of your data.

Data Cleanse 52

Data Cleanse Metadata High Quality Data Datasets 52

Zalando Engineering

OCTOBER 18, 2023

The query directives are generally useful for clients to express certain types of metadata for the query. We can access the request parameters (like pathname) by running GraphQL over HTTP. They also have an excellent point: query directives are good for annotating the query with metadata and not for resolver logic.

Metadata 86

Metadata Coding Banking Designing 86

ProjectPro

FEBRUARY 8, 2023

Read this blog to understand everything about AWS Glue that makes it one of the most popular data integration solutions in the industry. Then, Glue writes the job's metadata into the embedded AWS Glue Data Catalog. By using AWS Glue Data Catalog, multiple systems can store and access metadata to manage data in data silos.

AWS 98

AWS Scala Metadata Data Lake 98

Rockset

AUGUST 27, 2023

This blog attempts to arm you with some knowledge of your future, the problems you will face, and questions you may not know yet that you need to ask. A proper survey of these approaches would fill many blog posts of this size. The same applies to the metadata of your system. Be very wary of homerolled “vector-search infra.”

Metadata 52

Metadata Database Algorithm SQL 52

Snowflake

AUGUST 31, 2023

Catalog Integration: Our newly developed Catalog Integration feature allows you to seamlessly plug Snowflake into other Iceberg catalogs tracking table metadata. In this blog post, we’ll dive into the details of these features and the benefits for customers. In addition to Iceberg External Tables, we introduced Native Iceberg Tables.

Metadata 83

Metadata AWS Data Lake Datasets 83

Cloudera

NOVEMBER 29, 2023

It simplifies recording model parameters, metadata, and metrics ensuring transparent bookkeeping. You can track each iteration, compare changes, and ensure that you always have access to the version that suits your needs. The post Announcing General Availability of Model Registry appeared first on Cloudera Blog.

Machine Learning 84

Machine Learning Metadata Cloud Project 84

Edureka

MAY 6, 2024

Whether you are a newbie or an experienced individual, if you want to explore more about the concepts of MLOPS, then you just click on the right blog. But before we begin, Let’s have a look at what we will be covering in this blog: What is MLOPS? Proper metadata management helps in understanding, grouping, and sorting data.

Machine Learning 52

Machine Learning Metadata Programming Language Healthcare 52

Cloudyard

DECEMBER 7, 2023

Read Time: 3 Minute, 0 Second In this blog post, we will delve into Snowflake’s Listing feature, which operates on the data sharing provider and consumer model. Listing in Consumer Click on Get and it will start replicating the data from provider Account. Data Replication Once done we will get the mail.

Metadata 52

Metadata SQL Data Cloud 52

Cloudera

AUGUST 26, 2021

In this blog post, we will ingest a real world dataset into Ozone, create a Hive table on top of it and analyze the data to study the correlation between new vaccinations and new cases per country using a Spark ML Jupyter notebook in CML. Learn more about the impacts of global data sharing in this blog, The Ethics of Data Exchange.

Data Science 109

Data Science Cloud Hadoop Metadata 109

Netflix Tech

MAY 5, 2021

To support such use cases, access control at the user workspace and project workspace granularity is extremely important for presenting a globally consistent view of pertinent data to these artists. 2 , are the file system interface, the API interface, and the metadata and data stores. The major pieces, as shown in Fig.

Metadata 69

Metadata Bytes Media Cloud Storage 69

Engineering at Meta

MAY 3, 2023

Ducktail is known to target a number of platforms across the internet, including: LinkedIn to socially engineer people into downloading malware; Browsers like Google Chrome, Microsoft Edge, Brave, and Firefox to gain access to people’s information on desktop; and File-hosting services such as Dropbox and Mega, to host malware.

Media 113

Media Metadata Coding Database 113

Cloudera

JUNE 25, 2021

This blog post outlines detailed step by step instructions to perform Hive Replication from an on-prem CDH cluster to a CDP Public Cloud Data Lake. In order to copy or migrate data from CDH cluster to CDP Data Lake cluster, the on-prem CDH cluster should be able to access the CDP cloud storage. Hadoop SQL Policies overview.

Cloud 69

Cloud Data Lake Cloud Storage Metadata 69

Cloudera

MARCH 17, 2023

Hopefully this blog will give ChatGPT an opportunity to learn and correct itself while counting towards my 2023 contribution to social good. The one key component that is missing is a common, shared table format, that can be used by all analytic services accessing the lakehouse data. I thought this was a fairly comprehensive list.

Education 61

Education Unstructured Data Data Lake Data Warehouse 61

Cloudera

MARCH 14, 2023

In our previous DataFlow Designer blog post , we introduced you to the new user interface and highlighted its key capabilities. In this blog post we will put these capabilities in context and dive deeper into how the built-in, end-to-end data flow life cycle enables self-service data pipeline development.

Data Pipeline 81

Data Pipeline Designing Kafka Metadata 81

phData: Data Engineering

SEPTEMBER 19, 2023

This blog will guide you through the best data modeling methodologies and processes for your data lake, helping you make informed decisions and optimize your data management practices. Data Governance and Security By defining data models, organizations can establish policies, access controls, and security measures to protect sensitive data.

Data Lake 52

Data Lake Process Metadata Data Warehouse 52

RandomTrees

FEBRUARY 6, 2024

Generative AI with Data Lineage: By automating the process of collecting lineage metadata, generating visualizations of data lineage, and identifying and troubleshooting data lineage problems. This can save time and effort for data engineers, and it can also help to ensure that ETL pipelines are more accurate and reliable.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

Cloudera

SEPTEMBER 23, 2020

Comprehensive auditing is provided to enable enterprises to effectively and efficiently meet their compliance requirements by auditing access and other types of operations across OpDB (through HBase). User, business classification of asset accessed. Policy outcome (access or deny). Policy outcome (access or deny).

Database 81

Database Data Lake Metadata Java 81

Databand.ai

JULY 19, 2023

Metadata Management Metadata, or ‘data about data’, is a crucial component of data management. Data lineage tools provide robust metadata management capabilities, allowing businesses to capture, store, and manage metadata associated with their data. This fosters a data-driven culture within the organization.

Pipeline-centric 52

Pipeline-centric Data Governance Metadata Government 52

Engineering at Meta

OCTOBER 31, 2023

SCARF coordinates several kinds of tasks for each data system: metadata collection (e.g., Measuring usage SCARF tracks two primary metrics to determine if deletion is safe: It measures static usage by identifying code that appears to query a type of data; and it measures runtime usage by observing access patterns in production.

Data 93

Data Metadata Coding Relational Database 93