Precisely

APRIL 11, 2023

Do I have access to the data? What do I need to do to access the data now and make it easy to find and access in the future? Data that’s scattered throughout your organization – within various systems and with multiple owners – poses a big challenge. What existing data does the organization have to support my initiative?

Accessible 52

Accessible Accessibility Metadata Data Integration 52

Data Engineering Podcast

FEBRUARY 20, 2022

You can observe your pipelines with built in metadata search and column level lineage. Your host is Tobias Macey and today I’m interviewing Guy Yachdav, Director of Software Engineering at Immunai, about his work at Immunai to wrangle biological data for advancing research into the human immune system. regulatory, security, etc.)

Systems 100

Systems Software Engineer Software Engineering Data Warehouse 100

LinkedIn Engineering

DECEMBER 6, 2022

With a reasonably sizable footprint of servers in data centers, LinkedIn is responsible for ensuring that these hosts are always on an operating system (OS) version deemed the ���latest and greatest��� for all intents and purposes. An OS snapshot is a collection of bootfiles (initrd, vmlinuz), RPMs, and a few extra metadata.

Systems 55

Systems Metadata Datasets Designing 55

Netflix Tech

SEPTEMBER 29, 2022

Timestone: Netflix’s High-Throughput, Low-Latency Priority Queueing System with Built-in Support for Non-Parallelizable Workloads by Kostas Christidis Introduction Timestone is a high-throughput, low-latency priority queueing system we built in-house to support the needs of Cosmos , our media encoding platform. Over the past 2.5

Systems 85

Systems Metadata Media Kafka 85

Netflix Tech

MARCH 10, 2021

ConsoleMe: A Central Control Plane for AWS Permissions and Access By Curtis Castrapel , Patrick Sanders , and Hee Won Kim At AWS re:Invent 2020, we open sourced two new tools for managing multi-account AWS permissions and access. Groups beyond software engineering teams are standing up their own systems and automation.

AWS 100

AWS Accessible Accessibility Cloud 100

Engineering at Meta

MARCH 18, 2024

Systems and application logs play a key role in operations, observability, and debugging workflows at Meta. We designed the system to support service-level guarantees on log freshness, completeness, durability, query latency, and query result completeness. Each log line can have zero or more metadata key-value pairs attached to it.

Engineering 89

Engineering Metadata Architecture Designing 89

Snowflake

NOVEMBER 29, 2023

In medicine, lower sequencing costs and improved clinical access to NGS technology has been shown to increase diagnostic yield for a range of diseases, from relatively well-understood Mendelian disorders, including muscular dystrophy and epilepsy , to rare diseases such as Alagille syndrome.

Metadata 81

Metadata Healthcare Medical Data Storage 81

Knowledge Hut

APRIL 25, 2024

Kubernetes (sometimes shortened to K8s with the 8 standing for the number of letters between the “K” and the “s”) is an open-source system to deploy, scale, and manage containerized applications anywhere. apiVersion: v1 kind: Pod metadata: name: Postgres spec: containers: - name: Postgres image: Postgres: 3.1

Database-centric 81

Database-centric Metadata MongoDB Pipeline-centric 81

Knowledge Hut

JANUARY 2, 2024

You will learn how to build up Kube-state-metrics system, pull and collect metrics, deploy a Prometheus server and metrics exporters, configure alerts with Alertmanager, and create Grafana dashboards. Metric Endpoint: The systems you want Prometheus to monitor should disclose their metrics on an endpoint called /metrics.

Architecture 98

Architecture Metadata Utilities Data Collection 98

Data Engineering Podcast

DECEMBER 4, 2022

The typical conception of how it is accessed is through a web or desktop application running on a powerful laptop. This opens the door for busy employees to access and analyze their company information away from their desk, but it has the more powerful effect of bringing first-class support to companies operating in mobile-first economies.

Business Intelligence 100

Business Intelligence Metadata BI MongoDB 100

Snowflake

APRIL 23, 2024

The connector provides immediate access to up-to-date ServiceNow data without the need to manually integrate against API endpoints. With just a few clicks, you can get ServiceNow data directly in your Snowflake account and combine it with other data sources, including ERP, HR and CRM systems.

Data Warehouse 91

Data Warehouse Data Integration Consulting Metadata 91

Netflix Tech

JUNE 1, 2023

In this blog post, we’ll discuss the methods we used to ensure a successful launch, including: How we tested the system Netflix technologies involved Best practices we developed Realistic Test Traffic Netflix traffic ebbs and flows throughout the day in a sinusoidal pattern. Basic with ads was launched worldwide on November 3rd.

Algorithm 136

Algorithm Metadata Kafka Systems 136

The Modern Data Company

JANUARY 22, 2024

The central piece to this narrative is DataOS®, Modern’s state-of-the-art data operating system designed to unify data across silos on a scalable level. By humanizing data and streamlining access, we’re redefining how organizations harness their data’s power to drive innovation and achieve transformative business outcomes.

Data Integration 98

Data Integration Metadata Government Unstructured Data 98

Snowflake

NOVEMBER 2, 2023

Snowflake Horizon is Snowflake’s built-in governance solution with a unified set of compliance, security, privacy, interoperability, and access capabilities. Snowflake continues to advance Snowflake Horizon with additional capabilities for compliance, security, privacy, interoperability, and access.

Metadata 101

Metadata Government AWS Medical 101

Snowflake

APRIL 29, 2024

DORA: Building a More Secure Financial System DORA, enacted in January 2023, moves beyond reactive measures, requiring FEs and their service providers to proactively identify vulnerabilities, prevent disruptions and plan for swift recovery from incidents. These features further enhance the protection of customers’ sensitive data.

Transportation 81

Transportation Data Governance Government Consulting 81

Data Engineering Podcast

JULY 31, 2022

Because of its centrality to your data systems it is valuable for debugging, governance, understanding context, and myriad other purposes. Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities.

IT 100

IT Metadata MongoDB MySQL 100

Data Engineering Podcast

OCTOBER 30, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Many conversations around data and analytics are focused on self-service access. Atlan is the metadata hub for your data ecosystem.

Business Intelligence 130

Business Intelligence Metadata MongoDB MySQL 130

LinkedIn Engineering

JULY 19, 2023

Open source data lakehouse deployments are built on the foundations of compute engines (like Apache Spark, Trino, Apache Flink), distributed storage (HDFS, cloud blob stores), and metadata catalogs / table formats (like Apache Iceberg, Delta, Hudi, Apache Hive Metastore). Tables (not files/blobs) are the only API abstraction for end-users.

Big Data 78

Big Data Data Management Management Metadata 78

Data Engineering Weekly

MARCH 24, 2024

The author goes beyond comparing the tools to various offerings from streaming vendors in stream processing and Kafka protocol-supported systems. The logging engine to debug AI workflow logs is an excellent system design study if you’re interested in it. The extracted key-value pairs are written to the line’s metadata.

Data Engineering 62

Data Engineering Data Engineer Engineering Metadata 62

Knowledge Hut

MARCH 7, 2024

Syncing Across Data Sources Once you import data into Big Data platforms you may also realize that data copies migrated from a wide range of sources on different rates and schedules can rapidly get out of the synchronization with the originating system. Two, it creates a commonality of data definitions, concepts, metadata and the like.

Big Data 98

Big Data Bytes Data Governance Raw Data 98

Towards Data Science

MAY 26, 2023

Usually an external system is needed (e.g. The good news is, if you use Hamilton, there isn’t the need for another system to get lineage; making use of lineage is straightforward too! Taking this code and connecting with e.g. a version control system (e.g. Adding metadata to make lineage more useful!

Metadata 85

Metadata Data Science Coding Machine Learning 85

Netflix Tech

NOVEMBER 14, 2023

Input : List of source tables and required processing mode Output : Psyberg identifies new events that have occurred since the last high watermark (HWM) and records them in the session metadata table. The session metadata table can then be read to determine the pipeline input. Audit Run various quality checks on the staged data.

Metadata 97

Metadata Data Pipeline Scala Data Workflow 97

Edureka

MAY 6, 2024

MLOps is an emerging discipline that aims to unify and streamline the machine learning system development (Dev) and operations (Ops) lifecycle. MLOps refers to the practice and discipline within machine learning that aims to unify and streamline machine learning system development (Dev) and operations (Ops).

Machine Learning 52

Machine Learning Metadata Programming Language Healthcare 52

Rockset

MARCH 18, 2024

There are several challenges that emerge when building real-time RAG pipelines: Real-time delivery of embeddings & updates Real-time metadata filtering Scale and efficiency for real-time data In the following sections, we’ll discuss these challenges broadly and delve into how they apply more specifically to vector search and vector databases.

Cloud 64

Cloud Building Metadata Kafka 64

Data Engineering Podcast

AUGUST 6, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Can you describe the technical elements of that system? How does that activity map to data systems/products?

Metadata 100

Metadata MongoDB MySQL Business Intelligence 100

The Modern Data Company

JANUARY 22, 2024

The central piece to this narrative is DataOS®, Modern’s state-of-the-art data operating system designed to unify data across silos on a scalable level. By humanizing data and streamlining access, we’re redefining how organizations harness their data’s power to drive innovation and achieve transformative business outcomes.

Data Integration 87

Data Integration Metadata Government Unstructured Data 87

Monte Carlo

FEBRUARY 26, 2024

It uses metadata to create a picture of the data, as well as the relationships between data assets of diverse sources, and the processing that takes place as data moves through systems. Additionally, data catalogs can centrally enforce rules of access, security, and governance, all in a way that is both automatic and visible.

Metadata 52

Metadata Government Data Data Governance 52

Striim

FEBRUARY 5, 2024

It is not a standalone product but comprises key elements, including data integration, ensuring the smooth merging of data; data quality, maintaining high data standards; metadata management, organizing and understanding data context; and security, safeguarding data integrity.

Pharmaceutical 52

Pharmaceutical Data Cleanse Metadata Medical 52

ProjectPro

MARCH 5, 2021

Welcome to the World of Recommender Systems!!! Table of Contents What is a Recommender System? The invisible pieces of code that form the gears and cogs of the modern machine age, algorithms have given the world everything from social media feeds to search engines and satellite navigation to music recommendation systems.

Algorithm 40

Algorithm Systems Python Datasets 40

dbt Developer Hub

DECEMBER 19, 2023

Integration points ​ Discovery API (formerly referred to as Metadata API) Overview — This GraphQL API allows you to query the metadata that dbt Cloud generates every time you run a dbt project. The job level will only provide you the metadata of one job, giving you only a small snapshot of part of the project.

Metadata 52

Metadata Cloud Accessible Accessibility 52

Ascend.io

FEBRUARY 28, 2024

The foundation of any AI system is high-quality data. Engineers ensure the availability of clean, structured data, a necessity for AI systems to learn from patterns, make accurate predictions, and automate decision-making processes. And crucially, what does the future hold for data engineering in an AI-driven world?

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

Knowledge Hut

APRIL 4, 2024

Chef Solo) Users write system configuration files that are called ‘Recipes’, which are then organized into ‘Cookbooks’. Nodes, or the systems needing the configuration, are the final part. Keeping the Chef repository in sync with the source code revision control system. Validating Chef's source code. Nodes can be of any type, i.e:

Architecture 52

Architecture Amazon Web Services Metadata AWS 52

Data Engineering Podcast

MAY 18, 2021

Summary Data lineage is the common thread that ties together all of your data pipelines, workflows, and systems. In order to eliminate the wasted effort of building custom integrations every time you want to combine lineage information across systems Julien Le Dem introduced the OpenLineage specification.

Metadata 100

Metadata Kafka Data Warehouse Hadoop 100

Data Engineering Podcast

MARCH 27, 2022

You have full control over your data and their plugin system lets you integrate with all of your other data tools, including data warehouses and SaaS platforms. How have the design and goals of the system changed or evolved since you started working on it? Can you describe what Privacera is and the story behind it?

Data Governance 130

Data Governance Government Cloud Building 130

Tweag

MAY 16, 2023

That’s a dangerous mistake: with the advent of IaC for the cloud, configuration has become an important aspect of modern software systems, and a critical point of failure. A REPL nickel repl , a markdown documentation generator nickel doc and a nickel query command to retrieve metadata, types and contracts from code.

MySQL 131

MySQL Metadata Data Validation Coding 131

Data Engineering Podcast

AUGUST 28, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Atlan is the metadata hub for your data ecosystem. And don’t forget to thank them for their continued support of this show!

Data Engineering 100

Data Engineering Data Engineer MongoDB Metadata 100

Monte Carlo

MARCH 4, 2024

Table of Contents How Data Observability Works Made For The Lakehouse: Iceberg and Delta Transform in place: Dynamic Tables and Delta Live Tables Why This Matters How Data Observability Works Data observability provides full visibility into your data, systems, and code so you can find and fix bad data quickly.

Metadata 52

Metadata Data Lake Data Warehouse Data Engineering 52