Zalando Engineering

JUNE 30, 2020

Our Engineering Blog was launched in June 2020 after a long break of the previous tech blog. What customizations we applied to design the blog and the publishing process. Static Site Generator Our previous tech blog used a CMS which only a limited number of people had access to. So which static site generator to choose?

Engineering 40

Engineering Bytes AWS Python 40

Cloudera

APRIL 18, 2024

In those discussions, it was clear that everyone understood the need to treat data estates more cohesively as a whole—that means bringing more attention to security, data governance, and metadata management, the latter of which has become increasingly popular. Learn more about how Cloudera is accelerating enterprise AI.

Metadata 92

Metadata Government Data Governance Data 92

Rockset

MARCH 18, 2024

In this blog, we’ll discuss how RAG fits into the paradigm of real-time data processing and show an example product recommendation application using both Kafka and Flink on Confluent Cloud together with Rockset. These additional inputs are referred to as metadata filtering.

Cloud 64

Cloud Building Metadata Kafka 64

Knowledge Hut

APRIL 25, 2024

apiVersion: v1 kind: Pod metadata: name: Postgres spec: containers: - name: Postgres image: Postgres: 3.1 apiVersion: v1 kind: Pod metadata: name: Tomcat spec: containers: - name: Tomcat image: tomcat: 8.0 apiVersion : v1 kind: Pod metadata: name : nginx spec: containers : - name : nginx image : nginx:1.14.2

Database-centric 52

Database-centric Metadata MongoDB Pipeline-centric 52

Netflix Tech

NOVEMBER 14, 2023

By Abhinaya Shetty , Bharath Mummadisetty In the inaugural blog post of this series, we introduced you to the state of our pipelines before Psyberg and the challenges with incremental processing that led us to create the Psyberg framework within Netflix’s Membership and Finance data engineering team.

Data Process 90

Data Process Process Metadata Finance 90

Snowflake

DECEMBER 4, 2023

With this public preview, those external catalog options are either “GLUE”, where Snowflake can retrieve table metadata snapshots from AWS Glue Data Catalog, or “OBJECT_STORE”, where Snowflake retrieves metadata snapshots directly from the specified cloud storage location. With these three options, which one should you use?

Building 89

Building Metadata Cloud Storage AWS 89

Netflix Tech

NOVEMBER 14, 2023

In this three-part blog post series, we introduce you to Psyberg , our incremental data processing framework designed to tackle such challenges! It leverages Iceberg metadata to facilitate processing incremental and batch-based data pipelines. Given our role on this critical path, accuracy is paramount. Psyberg: The Game Changer!

Data Engineering 105

Data Engineering Data Engineer Engineering Metadata 105

Cloudera

APRIL 2, 2024

Our use of metadata also ensures persistent context across all analytics and cloud environments, making it easier to maintain consistent security policies and access controls. The post FedRAMP In Process Designation, A Milestone in Cybersecurity Commitment appeared first on Cloudera Blog.

Designing 96

Designing Process Government Metadata 96

Data Engineering Podcast

NOVEMBER 20, 2022

Summary The majority of blog posts and presentations about data engineering and analytics assume that the consumers of those efforts are internal business users accessing an environment controlled by the business. Atlan is the metadata hub for your data ecosystem. Atlan is the metadata hub for your data ecosystem.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Netflix Tech

NOVEMBER 14, 2023

By Abhinaya Shetty , Bharath Mummadisetty This blog post will cover how Psyberg helps automate the end-to-end catchup of different pipelines, including dimension tables. The session metadata table can then be read to determine the pipeline input. Metadata Recording : Metadata is persisted for traceability.

Metadata 97

Metadata Data Pipeline Scala Data Workflow 97

Tweag

MAY 16, 2023

Since the previous stable version ( 0.3.1 ), efforts have been made on three principal fronts: tooling (in particular the language server), the core language semantics (contracts, metadata, and merging), and the surface language (the syntax and the stdlib). The | symbol attaches metadata to fields.

MySQL 132

MySQL Metadata Data Validation Coding 132

Data Council

MARCH 25, 2021

How to Build a Community (Fishtown Analytics) Claire Carroll's first personal blog post on community-building is a must-read. Here's our March 2021 roundup of links from across the web that we selected for you: 1. As Fishtown Analytics' community manager for the last 2.5

Metadata 52

Metadata Management Building IT 52

Databand.ai

AUGUST 30, 2023

Learn more in our detailed guide to data reliability testing Metadata Management Metadata management is the process of organizing, maintaining, and using metadata to improve the quality, consistency, and usability of your data.

Data Cleanse 52

Data Cleanse Metadata High Quality Data Datasets 52

Rockset

AUGUST 27, 2023

This blog attempts to arm you with some knowledge of your future, the problems you will face, and questions you may not know yet that you need to ask. A proper survey of these approaches would fill many blog posts of this size. The same applies to the metadata of your system.

Metadata 52

Metadata Database Algorithm SQL 52

Data Engineering Weekly

DECEMBER 10, 2023

The blog is an excellent comparison study of Ray vs. Dask’s performance. The author discusses the OneTable sync mechanism among all three major LakeHouse formats in this blog. The blog discusses Psyberg’s two operational models, stateless & stateful data processing. Stores metadata to utilize later.

Data Engineering 76

Data Engineering Data Engineer Engineering Metadata 76

Data Engineering Podcast

SEPTEMBER 4, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!

IT 100

IT Metadata MongoDB MySQL 100

LinkedIn Engineering

JULY 19, 2023

Open source data lakehouse deployments are built on the foundations of compute engines (like Apache Spark, Trino, Apache Flink), distributed storage (HDFS, cloud blob stores), and metadata catalogs / table formats (like Apache Iceberg, Delta, Hudi, Apache Hive Metastore). Tables are governed as per agreed upon company standards.

Big Data 78

Big Data Data Management Management Metadata 78

Data Engineering Podcast

FEBRUARY 26, 2023

Contact Info LinkedIn Blog Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? TimeXtender Logo]([link] TimeXtender is a holistic, metadata-driven solution for data integration, optimized for agility. When is a data mesh the wrong choice?

Building 147

Building Metadata Machine Learning Data Integration 147

Snowflake

AUGUST 31, 2023

Catalog Integration: Our newly developed Catalog Integration feature allows you to seamlessly plug Snowflake into other Iceberg catalogs tracking table metadata. In this blog post, we’ll dive into the details of these features and the benefits for customers. In addition to Iceberg External Tables, we introduced Native Iceberg Tables.

Metadata 82

Metadata AWS Data Lake Datasets 82

Zalando Engineering

OCTOBER 18, 2023

The query directives are generally useful for clients to express certain types of metadata for the query. We have a separate blog post explaining the details of how Zalando uses persisted queries and how we think about schema stability and granular control. There are numerous other cases that we'll cover in future blog posts.

Metadata 87

Metadata Coding Banking Designing 87

Data Council

DECEMBER 21, 2020

The Modern Data Stack (Fishtown Analytics) This long-form post on the dbt blog is a must-read. Here's our December 2020 roundup of links from across the web that could be relevant to you: 1.

Metadata 52

Metadata Architecture Data Technology 52

RandomTrees

APRIL 18, 2024

By leveraging Python’s capabilities, we can create scripts that analyze Informatica workflows, extract relevant metadata, and generate equivalent Talend jobs. Analysing Informatica Workflows : The first step in the conversion process is to analyze Informatica workflows to extract essential metadata and transformation logic.

Coding 52

Coding Retail Metadata Python 52

Workfall

JUNE 12, 2023

In this blog, we will cover: What Is CDC and Its Benefits? These additional columns store metadata like timestamps, user IDs, and change types, ensuring granular change tracking and auditability. We will come up with more such use cases in our upcoming blogs. Where Is CDC Used and Who Uses It?

Telecommunication 80

Telecommunication Metadata Healthcare Finance 80

Cloudera

AUGUST 10, 2021

We expand on this feature later in this blog. Atlas / Kafka integration provides metadata collection for Kafa producers/consumers so that consumers can manage, govern, and monitor Kafka metadata and metadata lineage in the Atlas UI. Figure 8: Data lineage based on Kafka Atlas Hook metadata. x, and 6.3.x,

Cloud 97

Cloud Kafka Metadata SQL 97

Netflix Tech

JUNE 1, 2023

In this blog post, we’ll discuss the methods we used to ensure a successful launch, including: How we tested the system Netflix technologies involved Best practices we developed Realistic Test Traffic Netflix traffic ebbs and flows throughout the day in a sinusoidal pattern. Basic with ads was launched worldwide on November 3rd.

Algorithm 136

Algorithm Metadata Kafka Systems 136

Databand.ai

JULY 19, 2023

Metadata Management Metadata, or ‘data about data’, is a crucial component of data management. Data lineage tools provide robust metadata management capabilities, allowing businesses to capture, store, and manage metadata associated with their data.

Pipeline-centric 52

Pipeline-centric Data Governance Metadata Government 52

Netflix Tech

NOVEMBER 29, 2023

As noted in an earlier blog post , we began developing an HDR variant of VMAF; let’s call it HDR-VMAF. and based on content characteristics and/or metadata signaled in the bitstream. A vital aspect of such development is subjective testing with HDR encodes in order to generate training data. Krasula, A. Choudhury, S. Malfait, A.

Metadata 100

Metadata Electronics Algorithm Software Engineer 100

Cloudera

FEBRUARY 14, 2024

This blog discusses a few problems that you might encounter with Iceberg tables and offers strategies on how to optimize them in each of those scenarios. A bloated metadata.json file could increase both read/write times because a large metadata file needs to be read/written every time. Iceberg doesn’t delete the old data files.

Bytes 56

Bytes Metadata Data Lake SQL 56

Rockset

NOVEMBER 7, 2023

As a result, indexing of newly ingested vectors and metadata does not negatively impact search performance. In this blog, we’ll dig into how Rockset has fully integrated vector search into its search and analytics database. Rockset supports streaming updates to metadata and vectors in an efficient way.

Cloud 64

Cloud Metadata Database SQL 64

Netflix Tech

MARCH 10, 2023

This platform has evolved from supporting studio applications to data science applications, machine-learning applications to discover the assets metadata, and build various data facts. During this evolution, quite often we receive requests to update the existing assets metadata or add new metadata for the new features added.

Management 94

Management Kafka Metadata Media 94

Tweag

NOVEMBER 1, 2023

This blog post is part of the series exploring the foundations of the Nickel configuration language. Metadata can be attached to record fields, giving them more expressive power and the ability to better describe an interface for a partial configuration. The last important remaining piece to explore is the merge system. s t r i n g.

Metadata 107

Metadata Programming Programming Language Coding 107

LinkedIn Engineering

APRIL 25, 2023

In our previous blog post on GraphQL, we explained how LinkedIn uses GraphQL to expedite the process of onboarding new use-cases for external API partners. In this blog post, we will cover how the GraphQL layer is architected for use by our internal engineers to build member and customer facing applications.

Architecture 109

Architecture Metadata Java Transportation 109

Airbnb Tech

NOVEMBER 28, 2023

In this blog post, we share our innovative approach to scoring data quality, Airbnb’s Data Quality Score (“DQ Score”). Luckily, we had two existing tools that would make this much easier: Dataportal (Airbnb’s data catalog and exploration UI), and the Unified Metadata Service (UMS).

Data Warehouse 98

Data Warehouse Metadata Data Certification 98

Scott Logic

APRIL 22, 2024

In my previous blog post I looked at various technologies for sending data asynchronously between services including RabbitMQ, Kafka, AWS EventBridge. The data bus isn’t shown in the diagram because the discussion in this blog is relatively independent of which you pick.

AWS 52

AWS Metadata Systems Kafka 52

phData: Data Engineering

SEPTEMBER 19, 2023

This blog will guide you through the best data modeling methodologies and processes for your data lake, helping you make informed decisions and optimize your data management practices. Check out our Data Governance on Snowflake blog! Metadata Management Data modeling methodologies help in managing metadata within the data lake.

Data Lake 52

Data Lake Process Metadata Data Warehouse 52

Cloudera

AUGUST 26, 2021

In this blog post, we will ingest a real world dataset into Ozone, create a Hive table on top of it and analyze the data to study the correlation between new vaccinations and new cases per country using a Spark ML Jupyter notebook in CML. Learn more about the impacts of global data sharing in this blog, The Ethics of Data Exchange.

Data Science 108

Data Science Cloud Hadoop Metadata 108

Pinterest Engineering

NOVEMBER 7, 2023

For these reasons, and others detailed in our original PubSub Client blog post , our team has decided to invest in building, productionalizing, and most recently open-sourcing PubSub Client (PSC). years since our previous blog post, PSC has been battle-tested at large scale in Pinterest with notably positive feedback and results.

Kafka 98

Kafka Java Software Engineer Software Engineering 98