Zalando Engineering

JUNE 30, 2020

Our Engineering Blog was launched in June 2020 after a long break of the previous tech blog. What customizations we applied to design the blog and the publishing process. Static Site Generator Our previous tech blog used a CMS which only a limited number of people had access to. It's actively developed.

Engineering 40

Engineering Bytes AWS Python 40

Netflix Tech

MARCH 10, 2023

Studio applications use this service to store their media assets, which then goes through an asset cycle of schema validation, versioning, access control, sharing, triggering configured workflows like inspection, proxy generation etc. This pattern grows over time when we need to access and update the existing assets metadata.

Management 94

Management Kafka Metadata Media 94

Snowflake

APRIL 29, 2024

Recover: establish processes for rapid recovery after incidents to ensure business continuity. This translates to several key requirements and their associated benefits: More stringent technical and process-oriented security measures, enhancing protection for both FEs and their ICT third-party providers. Five Core Pillars of DORA 1.

Transportation 83

Transportation Data Governance Government Consulting 83

Databand.ai

AUGUST 30, 2023

It utilizes various techniques to identify and resolve data quality issues, ensuring that high-quality data is used for business processes and decision-making. A shorter time-to-value indicates that your organization is efficient at processing and analyzing data for decision-making purposes.

Data Cleanse 52

Data Cleanse Metadata High Quality Data Datasets 52

Tweag

OCTOBER 25, 2023

Maintainers also add all sorts of metadata such as the name, the version, the license, the homepage, the description, and so on, which could be used to make an even richer graph. nixtract ’s goal is to make it accessible in a way that’s easier to consume. nixtract ’s goal is to make it accessible in a way that’s easier to consume.

Metadata 107

Metadata Accessible Accessibility Python 107

Edureka

MAY 6, 2024

It involves collaboration between data scientists, ML engineers, and IT professionals to automate and optimize the end-to-end process of building, deploying, and maintaining machine learning applications. But before we begin, Let’s have a look at what we will be covering in this blog: What is MLOPS? Why do we need MLOPS?

Machine Learning 52

Machine Learning Metadata Programming Language Healthcare 52

Tweag

APRIL 19, 2023

The MedTech industry is buzzing thanks to a continuous stream of innovation, promising to be more precise, efficient and accessible than ever. To allow innovation in medical imaging with AI, we need efficient and affordable ways to store and process these WSIs at scale. But as it turns out, we can’t use it. _slides_specs. width , spec.

Medical 60

Medical Process Cloud Bytes 60

LinkedIn Engineering

JULY 19, 2023

Open source data lakehouse deployments are built on the foundations of compute engines (like Apache Spark, Trino, Apache Flink), distributed storage (HDFS, cloud blob stores), and metadata catalogs / table formats (like Apache Iceberg, Delta, Hudi, Apache Hive Metastore). Tables (not files/blobs) are the only API abstraction for end-users.

Big Data 78

Big Data Data Management Management Metadata 78

Netflix Tech

JUNE 1, 2023

In this blog post, we’ll discuss the methods we used to ensure a successful launch, including: How we tested the system Netflix technologies involved Best practices we developed Realistic Test Traffic Netflix traffic ebbs and flows throughout the day in a sinusoidal pattern. Basic with ads was launched worldwide on November 3rd.

Algorithm 136

Algorithm Metadata Kafka Systems 136

Snowflake

AUGUST 22, 2023

The data product lifecycle includes the following stages: Discovery Design Development Deployment Let’s take a look at what they entail and the roles that lead and support each of them: Discovery starts the process. In an earlier blog post, I discussed how the notion of a product emerges from a business need.

Metadata 71

Metadata Data AWS Business Analyst 71

Snowflake

NOVEMBER 2, 2023

To give customers flexibility for how they fit Snowflake into their architecture, Iceberg Tables can be configured to use either Snowflake or an external service like AWS Glue as the tables’s catalog to track metadata, with an easy one-line SQL command to convert to Snowflake in a metadata-only operation.

Data Lake 89

Data Lake Data Warehouse Cloud Unstructured Data 89

DoorDash Engineering

DECEMBER 5, 2023

Almost six billion messages are processed each day at an average rate of four million messages per minute, which sometimes peaks at double that rate. The slowest step in that process was provisioning Kafka Topics, which involved on-call engineer approval and was prone to failure, further increasing the on-call load.

Kafka 91

Kafka Programming Language Metadata Architecture 91

Cloudera

AUGUST 10, 2021

We understand that migrating your data platform to the latest version can be an intricate task, and at Cloudera we’ve worked hard to simplify this process for all our customers. . Impala Row Filtering to set access policies for rows when reading from a table. We expand on this feature later in this blog. x, and 6.3.x,

Cloud 96

Cloud Kafka Metadata SQL 96

Databand.ai

JULY 19, 2023

This allows businesses to identify any transformations or processes that may be compromising the data’s integrity. Root cause analysis and debugging: Data lineage tools facilitate this process by providing a clear view of the data’s journey, allowing businesses to pinpoint exactly where the issue occurred.

Pipeline-centric 52

Pipeline-centric Data Governance Metadata Government 52

Cloudera

NOVEMBER 29, 2023

The Model Registry is designed to streamline these processes, offering a variety of tools and features. It simplifies recording model parameters, metadata, and metrics ensuring transparent bookkeeping. You can track each iteration, compare changes, and ensure that you always have access to the version that suits your needs.

Machine Learning 84

Machine Learning Metadata Cloud Project 84

ProjectPro

FEBRUARY 8, 2023

Read this blog to understand everything about AWS Glue that makes it one of the most popular data integration solutions in the industry. For e.g., Finaccel, a leading tech company in Indonesia, leverages AWS Glue to easily load, process, and transform their enterprise data for further processing. billion by 2026?

AWS 98

AWS Scala Metadata Data Lake 98

LinkedIn Engineering

APRIL 25, 2023

In our previous blog post on GraphQL, we explained how LinkedIn uses GraphQL to expedite the process of onboarding new use-cases for external API partners. In this blog post, we will cover how the GraphQL layer is architected for use by our internal engineers to build member and customer facing applications.

Architecture 109

Architecture Metadata Java Transportation 109

Netflix Tech

DECEMBER 3, 2022

All the above commands are very likely to be described in separate future blog posts, but right now let’s focus on the dataflow sample command. This logic consists of the following parts: DDL code, table metadata information, data transformation and a few audit steps. test_sparksql_write.py

Data Pipeline 143

Data Pipeline Scala Metadata Food 143

Zalando Engineering

OCTOBER 18, 2023

The query directives are generally useful for clients to express certain types of metadata for the query. We can access the request parameters (like pathname) by running GraphQL over HTTP. This process is cumbersome, but it is on purpose. See our blog post GraphQL persisted queries and schema stability for more details.

Metadata 86

Metadata Coding Banking Designing 86

Cloudera

JANUARY 19, 2024

As described in our recent blog post , an SQL AI Assistant has been integrated into Hue with the capability to leverage the power of large language models (LLMs) for a number of SQL tasks. This blog post aims to help you understand what you can do to get started with generative AI assisted SQL using Hue image version ​​2023.0.16.0

SQL 71

SQL Data Warehouse AWS Architecture 71

Cloudera

MARCH 3, 2022

Data analytics and machine learning can become a business and a compliance risk if data security, governance, lineage, metadata management, and automation are not holistically applied across the entire data lifecycle and all environments. Afterall, retrofitting good governance is a momentous task.

Data Governance 96

Data Governance Government Metadata Medical 96

Cloudyard

APRIL 4, 2024

This blog post explores how Snowpark, a powerful tool for data processing within Snowflake, can be used to perform RFM segmentation and unlock actionable customer insights. Implementation: Implementation: We developed a Snowpark script to process customer transaction data stored in the Snowflake database.

Retail 40

Retail Data Ingestion Metadata Datasets 40

RandomTrees

FEBRUARY 6, 2024

Generative AI corrects data imbalances, ensuring fair sentiment analysis on e-commerce platforms, enriches training data for natural language processing (NLP) tasks. Data’s exponential growth : Gen AI holds the potential to address this challenge by automating data processing tasks, extracting valuable insights from the vast amounts of data.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

Netflix Tech

NOVEMBER 13, 2023

In this blog post, we will introduce speech and music detection as an enabling technology for a variety of audio applications in Film & TV, as well as introduce our speech and music activity detection (SMAD) system which we recently published as a journal article in EURASIP Journal on Audio, Speech, and Music Processing.

Datasets 89

Datasets Metadata Algorithm Architecture 89

Databand.ai

AUGUST 30, 2023

A data quality strategy details the processes, tools, and techniques employed to ensure your company’s data is accurate, consistent, complete, and up-to-date. This includes defining roles and responsibilities related to managing datasets and setting guidelines for metadata management.

Building 52

Building Data Cleanse Data Governance Datasets 52

LinkedIn Engineering

DECEMBER 19, 2023

These clusters are the backbone for storing and processing extensive data volumes, empowering us to deliver essential features and services to members, such as personalized recommendations, enhanced search functionality, and valuable insights. This metadata includes the namespace, file permissions, and the mapping of data blocks to datanodes.

Big Data 92

Big Data Hadoop Metadata Data 92

Engineering at Meta

OCTOBER 31, 2023

In our first post on automating product deprecation , we discussed the complexities of product deprecations and introduced SCARF’s workflow management tools that guide engineers through a coordinated process to safely deprecate products. If any of the assets are not in use, SCARF initiates a process to safely remove them.

Data 93

Data Metadata Coding Relational Database 93

Cloudera

MARCH 14, 2023

In our previous DataFlow Designer blog post , we introduced you to the new user interface and highlighted its key capabilities. In this blog post we will put these capabilities in context and dive deeper into how the built-in, end-to-end data flow life cycle enables self-service data pipeline development.

Data Pipeline 81

Data Pipeline Designing Kafka Metadata 81

Cloudera

AUGUST 26, 2021

In this blog post, we will ingest a real world dataset into Ozone, create a Hive table on top of it and analyze the data to study the correlation between new vaccinations and new cases per country using a Spark ML Jupyter notebook in CML. Learn more about the impacts of global data sharing in this blog, The Ethics of Data Exchange.

Data Science 109

Data Science Cloud Hadoop Metadata 109

DoorDash Engineering

APRIL 12, 2023

A centralized metrics layer ensures accurate and reliable measurement of experiment results and streamlines the analysis process by minimizing the need for ad-hoc analysis. Additionally, having a centralized repository of metrics facilitates easy access to metrics, empowering all members of an organization to analyze experiment results.

SQL 82

SQL Metadata Raw Data Government 82

Data Engineering Weekly

MARCH 10, 2024

Google: Croissant- a metadata format for ML-ready datasets Google Research introduced Croissant, a new metadata format designed to make datasets ML-ready by standardizing the format, facilitating easier use in machine learning projects. Data engineers build the systems that store and process sensitive information.

Data Engineering 52

Data Engineering Data Engineer Engineering Datasets 52

Data Engineering Weekly

JULY 23, 2023

This blog post will delve into these questions, tackle common misconceptions, and give you an intuitive understanding of how to think about GPUs. Atlan AI leverages metadata that Atlan captures across the data stack to make AI part of your data stack. The blog overviews potential information leakage and how to minimize it.

Data Engineering 52

Data Engineering Data Engineer Engineering Deep Learning 52

LinkedIn Engineering

JANUARY 16, 2024

In a previous blog post, we talked about how we built our anti-abuse platform using CASAL. In this blog post, we'll go deeper into how we manage account restrictions. When we detected that a member’s intent veered into abusive territory, we set the process of imposing restrictions in motion.

Kafka 84

Kafka Relational Database Java Architecture 84

Netflix Tech

SEPTEMBER 24, 2021

By Xiaomei Liu , Rosanna Lee , Cyril Concolato Introduction Behind the scenes of the beloved Netflix streaming service and content, there are many technology innovations in media processing. Packaging has always been an important step in media processing. Uploading and downloading data always come with a penalty, namely latency.

Cloud 95

Cloud Bytes Cloud Storage Media 95

dbt Developer Hub

JANUARY 8, 2024

Each node is assigned a unique_id (like model.my_project.my_model ) and the manifest stores all the metadata about that model in a dictionary associated to that id. Different runs of your project in different environments result in different metadata written to the manifest. That’s exactly what defer does!

Coding 59

Coding Metadata Cloud Healthcare 59

Cloudera

JULY 14, 2023

In this blog, I will describe a few strategies one could undertake for various use cases. It could be much easier to simply stop all those jobs rather than allowing them to continue during the migration process. This will be discussed in a later blog. Hive creates Iceberg’s metadata files for the same exact table.

Metadata 69

Metadata Data Warehouse Big Data Ecosystem Java 69

Cloudera

AUGUST 3, 2021

The last eighteen months is causing supply chain forecasters to rethink the definition and incorporate risk into the planning process. . Point solutions promote themselves as being specialists in specific business process areas and the preferred solution – data warehousing, machine learning as examples.

Manufacturing 108

Manufacturing Retail Entertainment Government 108