Booking.com Engineering

DECEMBER 2, 2022

We take advantage of this feature in our ad bidding systems, maintaining consistent data views from our Account Specialists’ spreadsheets, to our Data Scientists’ notebooks, to our bidding system’s in-memory data. Managing Workflows Data, delivered with quality at the required moment, is at the heart of our system performance.

Systems 52

Systems Cloud MySQL Relational Database 52

Knowledge Hut

APRIL 25, 2024

Kubernetes (sometimes shortened to K8s with the 8 standing for the number of letters between the “K” and the “s”) is an open-source system to deploy, scale, and manage containerized applications anywhere. apiVersion: v1 kind: Pod metadata: name: Postgres spec: containers: - name: Postgres image: Postgres: 3.1 What are Kubernetes Pods?

Database-centric 81

Database-centric Metadata MongoDB Pipeline-centric 81

Netflix Tech

JUNE 1, 2023

In this blog post, we’ll discuss the methods we used to ensure a successful launch, including: How we tested the system Netflix technologies involved Best practices we developed Realistic Test Traffic Netflix traffic ebbs and flows throughout the day in a sinusoidal pattern. Basic with ads was launched worldwide on November 3rd.

Algorithm 136

Algorithm Metadata Kafka Systems 136

Engineering at Meta

FEBRUARY 6, 2024

DotSlash handles transparently fetching, decompressing, and verifying the appropriate remote artifact for the current operating system and CPU. Our continuous integration (CI) system supports special configuration for DotSlash jobs where a user must specify: A set of builds to run (these can span multiple platforms). Because of how #!

Metadata 115

Metadata Coding Building Project 115

Snowflake

APRIL 23, 2024

With just a few clicks, you can get ServiceNow data directly in your Snowflake account and combine it with other data sources, including ERP, HR and CRM systems. To monitor the performance of mainframes, servers and cloud systems, the company needs to analyze relevant metadata and promptly address any concerns or issues.

Data Warehouse 89

Data Warehouse Data Integration Consulting Metadata 89

Tweag

NOVEMBER 1, 2023

The last important remaining piece to explore is the merge system. In this post, we’ll see how to use the Nickel merge system to write reusable configuration modules, and why the merge approach seems to be more adapted for modular configurations than plain functions, despite Nickel being a functional language. s t r i n g.

Metadata 105

Metadata Programming Programming Language Coding 105

Tweag

MAY 16, 2023

That’s a dangerous mistake: with the advent of IaC for the cloud, configuration has become an important aspect of modern software systems, and a critical point of failure. A REPL nickel repl , a markdown documentation generator nickel doc and a nickel query command to retrieve metadata, types and contracts from code.

MySQL 131

MySQL Metadata Data Validation Coding 131

ProjectPro

MARCH 5, 2021

Welcome to the World of Recommender Systems!!! Table of Contents What is a Recommender System? The invisible pieces of code that form the gears and cogs of the modern machine age, algorithms have given the world everything from social media feeds to search engines and satellite navigation to music recommendation systems.

Algorithm 40

Algorithm Systems Python Datasets 40

Ascend.io

FEBRUARY 28, 2024

The foundation of any AI system is high-quality data. Engineers ensure the availability of clean, structured data, a necessity for AI systems to learn from patterns, make accurate predictions, and automate decision-making processes. And crucially, what does the future hold for data engineering in an AI-driven world?

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

Monte Carlo

FEBRUARY 26, 2024

It uses metadata to create a picture of the data, as well as the relationships between data assets of diverse sources, and the processing that takes place as data moves through systems. As data structure changes in connected systems, the changes are automatically captured and imported to the data catalog.

Metadata 52

Metadata Government Data Data Governance 52

Netflix Tech

DECEMBER 3, 2022

Thanks to the Netflix internal lineage system (built by Girish Lingappa ) Dataflow migration can then help you identify downstream usage of the table in question. For more information on this and other examples please visit the Dataflow documentation page." " ) COMMENT "Example dataset brought to you by Dataflow.

Data Pipeline 143

Data Pipeline Scala Metadata Food 143

Rockset

APRIL 18, 2023

We’re excited to introduce vector search on Rockset to power fast and efficient search experiences, personalization engines, fraud detection systems and more. In the demo, you’ll see how Rockset delivers search results in 15 milliseconds over thousands of documents. Why use vector search?

Unstructured Data 105

Unstructured Data Metadata Machine Learning SQL 105

Lyft Engineering

JUNE 28, 2023

In early 2022, Lyft already had a comprehensive Machine Learning Platform called LyftLearn composed of model serving , training , CI/CD, feature serving , and model monitoring systems. However, streaming data was not supported as a first-class citizen across many of the platform’s systems — such as training, complex monitoring, and others.

Machine Learning 115

Machine Learning Building Metadata Kafka 115

ProjectPro

FEBRUARY 8, 2023

Then, Glue writes the job's metadata into the embedded AWS Glue Data Catalog. AWS Glue then creates data profiles in the catalog, a repository for all data assets' metadata, including table definitions, locations, and other features. Why Use AWS Glue? Establish a crawler schedule. AWS Glue object list searches and filtering.

AWS 98

AWS Scala Metadata Data Lake 98

Christophe Blefari

JUNE 16, 2023

Yann's vision goes toward AI systems learning and reasoning like animals and humans. Data Documentation 101: Why? — Marie wrote best practices for establishing complete and reliable data documentation. The first advice is about the documentation readers: data team, business users or other stakeholders.

Programming Language 130

Programming Language SQL PostgreSQL Data 130

Data Engineering Podcast

NOVEMBER 27, 2022

This has led to inefficiencies in how data is stored, accessed, and shared across process and system boundaries. In this episode Wes McKinney shares the ways that Arrow and its related projects are improving the efficiency of data systems and driving their next stage of evolution. Atlan is the metadata hub for your data ecosystem.

Data Process 100

Data Process Process Metadata Business Intelligence 100

Snowflake

NOVEMBER 2, 2023

For example, customers who need a centralized store of data in large volume and variety – including JSON, text files, documents, images, and video – have built their data lake with Snowflake. Systems that generate data add new columns to accommodate additional information, which requires downstream tables to evolve accordingly.

Data Lake 85

Data Lake Data Warehouse Cloud Unstructured Data 85

Snowflake

APRIL 29, 2024

DORA: Building a More Secure Financial System DORA, enacted in January 2023, moves beyond reactive measures, requiring FEs and their service providers to proactively identify vulnerabilities, prevent disruptions and plan for swift recovery from incidents. Five Core Pillars of DORA 1.

Transportation 79

Transportation Data Governance Government Consulting 79

dbt Developer Hub

DECEMBER 19, 2023

The documentation will help you familiarize yourself with how our users interact with dbt. Integration points ​ Discovery API (formerly referred to as Metadata API) Overview — This GraphQL API allows you to query the metadata that dbt Cloud generates every time you run a dbt project. New to dbt Cloud? ​

Metadata 52

Metadata Cloud Accessible Accessibility 52

Rock the JVM

OCTOBER 1, 2023

Introduction RPC stands for Remote Procedure Call, it’s a client-server communication protocol where one program can request a service on a different address that may be on the same or different system connected by a network. Metadata provides access to read and write metadata values to be exchanged during a call. import fs2.Stream

Scala 63

Scala Metadata Transportation Java 63

Knowledge Hut

JANUARY 2, 2024

You will learn how to build up Kube-state-metrics system, pull and collect metrics, deploy a Prometheus server and metrics exporters, configure alerts with Alertmanager, and create Grafana dashboards. Metric Endpoint: The systems you want Prometheus to monitor should disclose their metrics on an endpoint called /metrics.

Architecture 98

Architecture Metadata Utilities Data Collection 98

LinkedIn Engineering

OCTOBER 5, 2023

LinkedIn is on the forefront of leveraging EBR technology to revolutionize the way we approach search and recommendation systems. Figure 1 - Example of an Embeddings Graph Embedding based retrieval (EBR) is a method that is used at the early stages of a recommendation or search system.

IT 133

IT Metadata Designing Algorithm 133

Rockset

NOVEMBER 7, 2023

As a result, indexing of newly ingested vectors and metadata does not negatively impact search performance. FAISS-IVF at Rockset While Rockset is algorithm agnostic in its implementation of similarity indexing, for the initial implementation we leveraged FAISS-IVF as it’s widely used, well documented and supports updates.

Cloud 64

Cloud Metadata Database SQL 64

Knowledge Hut

JANUARY 23, 2024

Mongo DB is a popular NoSQL and open-source document-oriented database which allows a highly scalable and flexible document structure. MongoDB Projection is a special feature allowing you to select only the necessary data rather than selecting the whole set of data from the document. What is MongoDB Projection?

MongoDB 52

MongoDB Project Database-centric NoSQL 52

Cloudera

AUGUST 30, 2022

It outlines a scenario in which “recently married people might want to change their names on their driver’s licenses or other documentation. As such, managers at different agencies need to sort through multiple systems to make sure these documents are delivered correctly—even though they all apply to the same individuals.”.

Data Architecture 77

Data Architecture Architecture Data Lake NoSQL 77

Zalando Engineering

OCTOBER 18, 2023

The query directives are generally useful for clients to express certain types of metadata for the query. The type system directives are available for the locations listed below. validate ( schema , document , [ ( context ) => ({ Variable ( node ) { const isSensitive = context. getArgument () ? directives ? push ( node.

Metadata 86

Metadata Coding Banking Designing 86

Netflix Tech

DECEMBER 14, 2018

In the previous blog posts in this series, we introduced the N etflix M edia D ata B ase ( NMDB ) and its salient “Media Document” data model. In this post we will provide details of the NMDB system architecture beginning with the system requirements?—?these key value stores generally allow storing any data under a key).

Media 94

Media Database Metadata Data Schemas 94

Knowledge Hut

APRIL 4, 2024

Chef Solo) Users write system configuration files that are called ‘Recipes’, which are then organized into ‘Cookbooks’. Nodes, or the systems needing the configuration, are the final part. Keeping the Chef repository in sync with the source code revision control system. Validating Chef's source code.

Architecture 52

Architecture Amazon Web Services Metadata AWS 52

Towards Data Science

MAY 26, 2023

Usually an external system is needed (e.g. The good news is, if you use Hamilton, there isn’t the need for another system to get lineage; making use of lineage is straightforward too! Taking this code and connecting with e.g. a version control system (e.g. Adding metadata to make lineage more useful!

Metadata 86

Metadata Data Science Coding Machine Learning 86

phData: Data Engineering

SEPTEMBER 19, 2023

A data lake is a centralized repository containing extensive storage for raw, unfiltered data coming into a company’s data storage system. Data modeling methodologies are systematic approaches used to design and define the structure and relationships of data within a system. What is a Data Lake?

Data Lake 52

Data Lake Process Metadata Data Warehouse 52

Rockset

DECEMBER 14, 2023

As part of the steps that are run in the final aggregator when a query has completed, a record containing metadata associated with your query is created. They contain some basic metadata associated with your query but cannot be used with the query profiler. We will be introducing more system source types as time goes on.

Metadata 64

Metadata Systems Project Designing 64

Monte Carlo

MAY 18, 2023

Data lineage tools help you uncover and map the complex relationships between data elements across various sources, systems, and processes. There are a few reasons for this, but it’s mainly because data lineage involves so many layers–system, code, users– that are constantly evolving. You may even already have access to one of them.

Metadata 64

Metadata Data Lake Government Data Governance 64

Zalando Engineering

JANUARY 30, 2023

Our Incident Playbooks cover emergency procedures to initiate in case a certain set of conditions is met, for example when one of our systems is overloaded and the existing resiliency measures (e.g. The procedures document the conditions (e.g. Number of applications per playbook count What are incident playbooks?

Management 57

Management Metadata Software Engineer Software Engineering 57

Knowledge Hut

MAY 3, 2024

Whether it is an ebook, digitally signed agreements, password-protected documents, or scanned documents such as passports, the most preferred file format is PDF or Portable Document Format. It was originally developed by Adobe and is a file format used to present and transfer documents easily and reliably.

Python 52

Python Digital Media Coding Programming 52

Data Engineering Podcast

DECEMBER 16, 2019

The solution to discoverability and tracking of data lineage is to incorporate a metadata repository into your data platform. The metadata repository serves as a data catalog and a means of reporting on the health and status of your datasets when it is properly integrated into the rest of your tools.

Metadata 100

Metadata PostgreSQL Datasets Data Warehouse 100

ThoughtSpot

JUNE 28, 2023

Fine-grained personal access token that allows Read access to metadata and Read and Write access to code and commit statuses. For more information, explore our developer documentation on the new version control REST APIs for Git. master) Deploy commits from the latest version of a Git branch to a destination ThoughtSpot instance.

Metadata 72

Metadata Accessible Accessibility Utilities 72

Precisely

SEPTEMBER 11, 2023

By 2025, 50% of data and analytics leaders will be using augmented MDM and active metadata to enhance their capabilities – demonstrating that beyond data quality, automation is also in demand for data governance, data catalog, and security solutions. Problem: “We’re uncertain about compliance with privacy regulations!

Metadata 52

Metadata Data Integration Datasets Data Analysis Tools 52