Zalando Engineering

JUNE 30, 2020

Our Engineering Blog was launched in June 2020 after a long break of the previous tech blog. What customizations we applied to design the blog and the publishing process. Static Site Generator Our previous tech blog used a CMS which only a limited number of people had access to. It's actively developed.

Engineering 40

Engineering Bytes AWS Python 40

Netflix Tech

NOVEMBER 14, 2023

In this three-part blog post series, we introduce you to Psyberg , our incremental data processing framework designed to tackle such challenges! We’ll discuss batch data processing, the limitations we faced, and how Psyberg emerged as a solution. Let’s dive in! What is late-arriving data? How does late-arriving data impact us?

Data Engineering 105

Data Engineering Data Engineer Engineering Metadata 105

Netflix Tech

NOVEMBER 14, 2023

By Abhinaya Shetty , Bharath Mummadisetty This blog post will cover how Psyberg helps automate the end-to-end catchup of different pipelines, including dimension tables. In the previous installments of this series, we introduced Psyberg and delved into its core operational modes: Stateless and Stateful Data Processing.

Metadata 97

Metadata Data Pipeline Scala Data Workflow 97

Snowflake

DECEMBER 4, 2023

With this public preview, those external catalog options are either “GLUE”, where Snowflake can retrieve table metadata snapshots from AWS Glue Data Catalog, or “OBJECT_STORE”, where Snowflake retrieves metadata snapshots directly from the specified cloud storage location. With these three options, which one should you use?

Building 94

Building Metadata Cloud Storage AWS 94

Edureka

MAY 6, 2024

It involves collaboration between data scientists, ML engineers, and IT professionals to automate and optimize the end-to-end process of building, deploying, and maintaining machine learning applications. But before we begin, Let’s have a look at what we will be covering in this blog: What is MLOPS? Why do we need MLOPS?

Machine Learning 52

Machine Learning Metadata Programming Language Healthcare 52

Netflix Tech

MARCH 10, 2023

This platform has evolved from supporting studio applications to data science applications, machine-learning applications to discover the assets metadata, and build various data facts. During this evolution, quite often we receive requests to update the existing assets metadata or add new metadata for the new features added.

Management 94

Management Kafka Metadata Media 94

Data Engineering Podcast

FEBRUARY 26, 2023

In this episode he shares that journey and the combination of technical and organizational challenges that he encountered in the process. Contact Info LinkedIn Blog Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? When is a data mesh the wrong choice?

Building 147

Building Metadata Machine Learning Data Integration 147

Workfall

JUNE 12, 2023

In this blog, we will cover: What Is CDC and Its Benefits? CDC also plays a crucial role in data integration and ETL processes. It enables automation in business processes, event-driven applications, and microservices, reacting to data changes in near real-time. Where Is CDC Used and Who Uses It?

Telecommunication 80

Telecommunication Metadata Healthcare Finance 80

Netflix Tech

JUNE 1, 2023

In this blog post, we’ll discuss the methods we used to ensure a successful launch, including: How we tested the system Netflix technologies involved Best practices we developed Realistic Test Traffic Netflix traffic ebbs and flows throughout the day in a sinusoidal pattern. Basic with ads was launched worldwide on November 3rd.

Algorithm 136

Algorithm Metadata Kafka Systems 136

Databand.ai

JULY 19, 2023

This allows businesses to identify any transformations or processes that may be compromising the data’s integrity. Root cause analysis and debugging: Data lineage tools facilitate this process by providing a clear view of the data’s journey, allowing businesses to pinpoint exactly where the issue occurred.

Pipeline-centric 52

Pipeline-centric Data Governance Metadata Government 52

Snowflake

APRIL 29, 2024

Recover: establish processes for rapid recovery after incidents to ensure business continuity. This translates to several key requirements and their associated benefits: More stringent technical and process-oriented security measures, enhancing protection for both FEs and their ICT third-party providers. Five Core Pillars of DORA 1.

Transportation 82

Transportation Data Governance Government Consulting 82

Netflix Tech

NOVEMBER 29, 2023

As noted in an earlier blog post , we began developing an HDR variant of VMAF; let’s call it HDR-VMAF. and based on content characteristics and/or metadata signaled in the bitstream. For a given rate-quality operating point, the DO process helps allocate bits among the various shots while maximizing an overall objective function.

Metadata 100

Metadata Electronics Algorithm Software Engineer 100

Cloudera

AUGUST 10, 2021

We understand that migrating your data platform to the latest version can be an intricate task, and at Cloudera we’ve worked hard to simplify this process for all our customers. . We expand on this feature later in this blog. Figure 8: Data lineage based on Kafka Atlas Hook metadata. x, and 6.3.x,

Cloud 96

Cloud Kafka Metadata SQL 96

Christophe Blefari

JANUARY 20, 2024

The Rise of the Data Engineer The Downfall of the Data Engineer Functional Data Engineering — a modern paradigm for batch data processing There is a global consensus stating that you need to master a programming language (Python or Java based) and SQL in order to be self-sufficient. Here a small benchmark between some popular formats.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Pinterest Engineering

NOVEMBER 7, 2023

For these reasons, and others detailed in our original PubSub Client blog post , our team has decided to invest in building, productionalizing, and most recently open-sourcing PubSub Client (PSC). years since our previous blog post, PSC has been battle-tested at large scale in Pinterest with notably positive feedback and results.

Kafka 98

Kafka Java Software Engineer Software Engineering 98

Cloudera

NOVEMBER 29, 2023

The Model Registry is designed to streamline these processes, offering a variety of tools and features. It simplifies recording model parameters, metadata, and metrics ensuring transparent bookkeeping. The post Announcing General Availability of Model Registry appeared first on Cloudera Blog. What does the Model Registry offer?

Machine Learning 82

Machine Learning Metadata Cloud Project 82

Netflix Tech

NOVEMBER 13, 2023

In this blog post, we will introduce speech and music detection as an enabling technology for a variety of audio applications in Film & TV, as well as introduce our speech and music activity detection (SMAD) system which we recently published as a journal article in EURASIP Journal on Audio, Speech, and Music Processing.

Datasets 89

Datasets Metadata Algorithm Architecture 89

Cloudyard

APRIL 4, 2024

This blog post explores how Snowpark, a powerful tool for data processing within Snowflake, can be used to perform RFM segmentation and unlock actionable customer insights. Implementation: Implementation: We developed a Snowpark script to process customer transaction data stored in the Snowflake database.

Retail 40

Retail Data Ingestion Metadata Datasets 40

Lyft Engineering

SEPTEMBER 6, 2023

Routing overrides metadata: embed metadata in API request headers defining which offloaded deployment the request will get routed to. Context propagation : routing overrides metadata is propagated throughout the service mesh. Routing overrides metadata with ProxyApp Staging overrides was well-received by our product engineers.

Building 80

Building Metadata Electronics Engineering 80

Cloudera

MARCH 9, 2021

In this blog, we’ll highlight the key CDP aspects that provide data governance and lineage and show how they can be extended to incorporate metadata for non-CDP systems from across the enterprise. Extending Atlas’ metadata model. Processes: File transfer process. ETL/DB Load process. HIVE Table.

Data Governance 93

Data Governance Government Metadata Datasets 93

Databand.ai

AUGUST 30, 2023

A data quality strategy details the processes, tools, and techniques employed to ensure your company’s data is accurate, consistent, complete, and up-to-date. This includes defining roles and responsibilities related to managing datasets and setting guidelines for metadata management.

Building 52

Building Data Cleanse Data Governance Datasets 52

LinkedIn Engineering

DECEMBER 19, 2023

These clusters are the backbone for storing and processing extensive data volumes, empowering us to deliver essential features and services to members, such as personalized recommendations, enhanced search functionality, and valuable insights. This metadata includes the namespace, file permissions, and the mapping of data blocks to datanodes.

Big Data 92

Big Data Hadoop Metadata Data 92

Lyft Engineering

OCTOBER 3, 2023

Finally, as the subject of this blog post, we can assess data quality via batch compute analytics on our data warehouse, providing a comprehensive albeit slower evaluation compared to the previously mentioned methods. These flow through Kafka , our event streaming platform, before being processed by Flink , our streaming compute framework.

Big Data 84

Big Data Metadata Data Warehouse Data 84

Cloudera

JULY 14, 2023

In this blog, I will describe a few strategies one could undertake for various use cases. It could be much easier to simply stop all those jobs rather than allowing them to continue during the migration process. This will be discussed in a later blog. Hive creates Iceberg’s metadata files for the same exact table.

Metadata 69

Metadata Data Warehouse Big Data Ecosystem Java 69

Cloudera

AUGUST 26, 2021

In this blog post, we will ingest a real world dataset into Ozone, create a Hive table on top of it and analyze the data to study the correlation between new vaccinations and new cases per country using a Spark ML Jupyter notebook in CML. Learn more about the impacts of global data sharing in this blog, The Ethics of Data Exchange.

Data Science 107

Data Science Cloud Hadoop Metadata 107

Data Engineering Weekly

MARCH 10, 2024

Google: Croissant- a metadata format for ML-ready datasets Google Research introduced Croissant, a new metadata format designed to make datasets ML-ready by standardizing the format, facilitating easier use in machine learning projects. Data engineers build the systems that store and process sensitive information.

Data Engineering 52

Data Engineering Data Engineer Engineering Datasets 52

Databand.ai

AUGUST 30, 2023

It aims to streamline data ingestion, processing, and analytics by automating and integrating various data workflows. It encompasses the systems, tools, and processes that enable businesses to manage their data more efficiently and effectively. As a result, they can be slow, inefficient, and prone to errors.

Architecture 52

Architecture Data Ingestion Data Governance Data Cleanse 52

Workfall

AUGUST 23, 2023

Inside this directory, we’ll create a file named ‘manifest.json’ This file will play a crucial role in defining the metadata, permissions, and settings for our extension. This file played a crucial role in defining the metadata, permissions, and settings for our extension. appeared first on The Workfall Blog.

Metadata 83

Metadata Coding Project Utilities 83

Data Engineering Weekly

MAY 16, 2023

I won’t bore you with the importance of data quality in the blog. The bias toward correctness will increase the processing time, which may not be feasible when speed is a priority. Let’s talk about the data processing types. Two-Phase WAP The Two-Phase WAP, as the name suggests, follows two copy processes.

Engineering 110

Engineering Kafka Data Pipeline Data Warehouse 110

Cloudera

MARCH 3, 2022

Data analytics and machine learning can become a business and a compliance risk if data security, governance, lineage, metadata management, and automation are not holistically applied across the entire data lifecycle and all environments. From Bad to Worse. Afterall, retrofitting good governance is a momentous task.

Data Governance 95

Data Governance Government Metadata Medical 95

Confluent

AUGUST 29, 2019

Stream processing engines like KSQL furthermore give you the ability to manipulate all of this fluently. All of the code and setup discussed in this blog post can be found in this GitHub repository , so you can try it yourself! Meanwhile, in Kafka, the data shows up as we expect but with some extra metadata published by the producer.

Kafka 55

Kafka Process Algorithm Cloud 55

phData: Data Engineering

DECEMBER 16, 2021

If you missed our first blog on what a data catalog is , be sure to check it out! In this blog, we’ll explore what the ideal team to support your data catalog looks like. The Business Steward will be most familiar with the business drivers mentioned in the first blog in this series. How will Human-Driven Metadata be Captured?

Metadata 52

Metadata Datasets Data Education 52

Knowledge Hut

MAY 9, 2024

In this blog post, we will look at some of the world's highest paying data science jobs, what they entail, and what skills and experience you need to land them. Responsibilities Responsibilities of data modelers include validating data models, evaluating existing systems, ensuring data consistency, and optimizing metadata.

52

52

Cloudera

MARCH 14, 2023

In our previous DataFlow Designer blog post , we introduced you to the new user interface and highlighted its key capabilities. In this blog post we will put these capabilities in context and dive deeper into how the built-in, end-to-end data flow life cycle enables self-service data pipeline development.

Data Pipeline 80

Data Pipeline Designing Kafka Metadata 80

Cloudera

SEPTEMBER 29, 2022

It is a set of rubrics around people, process, and technology choices that allow for companies to scale their data systems. . The data product is not just the data itself, but a bunch of metadata that surrounds it — the simple stuff like schema is a given. Data mesh is about people, process, and technology. Data mesh defined.

Architecture 73

Architecture Data Architecture Metadata Data Warehouse 73

Netflix Tech

MAY 4, 2023

This blog series will examine the tools, techniques, and strategies we have utilized to achieve this goal. This blog post will provide a detailed analysis of replay traffic testing, a versatile technique we have applied in the preliminary validation phase for multiple migration initiatives.

Utilities 135

Utilities Systems Architecture Coding 135

LinkedIn Engineering

DECEMBER 12, 2023

PEDAL also consists of a metadata store that holds various algorithmic parameters, including the scale of noise that we introduce and whether we should use one-shot or continual observation algorithms. The application may then choose to further post-process the result, but PEDAL ensures the result is private.

Algorithm 77

Algorithm Metadata SQL Datasets 77