LinkedIn Engineering

JUNE 15, 2023

In this blog post, we will discuss the AvroTensorDataset API, techniques we used to improve data processing speeds by up to 162x over existing solutions (thereby decreasing overall training time by up to 66%), and performance results from benchmarks and production. an array within a map, within a union, etc…). Default is 128 * 1024 (128KB).

Datasets 102

Datasets Bytes Process Data Ingestion 102

Pinterest Engineering

NOVEMBER 22, 2023

In the first blog, we will share a short summary on the GokuS and GokuL architecture, data format for Goku Long Term, and how we improved the bootstrap time for our storage and serving components. More information about the architecture can be found in the GokuL blog and the cost reduction blog.

Database 93

Database Bytes Kafka Architecture 93

Netflix Tech

SEPTEMBER 24, 2021

Our previous tech blog Packaging award-winning shows with award-winning technology detailed our packaging technology deployed on the streaming side. The inspection stage examines the input media for compliance with Netflix’s delivery specifications and generates rich metadata.

Cloud 95

Cloud Bytes Cloud Storage Media 95

Monte Carlo

FEBRUARY 9, 2023

In this blog post we’ll dive into data vault architecture; challenges and best practices for maintaining data quality; and how data observability can help. architecture (with some minor deviations) to achieve their data integration objectives around scalability and use of metadata. “A What is a Data Vault model?

Architecture 52

Architecture Raw Data Metadata Data Warehouse 52

Netflix Tech

MAY 5, 2021

Netflix Drive relies on a data store that will be the persistent storage layer for assets, and a metadata store which will provide a relevant mapping from the file system hierarchy to the data store entities. 2 , are the file system interface, the API interface, and the metadata and data stores. The major pieces, as shown in Fig.

Metadata 69

Metadata Bytes Media Cloud Storage 69

Rock the JVM

AUGUST 9, 2022

For a more concrete example, we are going to write a program that will parse markdown files, extract words identified as tags, and then regenerate those files with tag-related metadata injected back into them. FileInputStream In our example later, we are going to process blog posts to parse tag meta-data. collectAll [ String ].

Scala 40

Scala Bytes Kafka Programming 40

Cloudera

AUGUST 14, 2020

One key part of the fault injection service is a very lightweight passthrough fuse file system that is used by Ozone for storing all its persistent data and metadata. The APIs are generic enough that we could target both Ozone data and metadata for failure/corruption/delays. NetFilter Extension. Fault Injection Framework: Github.

Hadoop 96

Hadoop Bytes Metadata Programming Language 96

Confluent

JULY 1, 2019

When a client (producer/consumer) starts, it will request metadata about which broker is the leader for a partition—and it can do this from any broker. The key thing is that when you run a client, the broker you pass to it is just where it’s going to go and get the metadata about brokers in the cluster from. The default is 0.0.0.0,

Kafka 99

Kafka Metadata AWS Bytes 99

Scribd Technology

MAY 18, 2021

Looking around the internet, there are few approaches people will blog about but many would either cost too much, be really complicated to setup/maintain, or both. The user requirements are likely relatable to a lot of folks: My application emits data into Kafka that I want to analyze later.

Kafka 52

Kafka Data Warehouse Bytes Metadata 52

Tweag

APRIL 19, 2023

In this blog post, I will explain the underlying technical challenges and share the solution that we helped implement at kaiko.ai , a MedTech startup in Amsterdam that is building a Data Platform to support AI research in hospitals. A solution is to read the bytes that we need when we need them directly from Blob Storage. width , spec.

Medical 60

Medical Process Cloud Bytes 60

Confluent

MAY 29, 2019

If you want to follow along and execute all the commands included in this blog post (and the next), you can check out this GitHub repository , which also includes the necessary Docker Compose functionality for running a compatible KSQL and Confluent Platform environment using the recently released Confluent 5.2.1. Sample repository.

Kafka 94

Kafka Management Bytes SQL 94

Datakin

OCTOBER 14, 2021

Run models & capture lineage metadata When working with Datakin (or any other OpenLineage backend) it’s important to generate the dbt docs first. Our schema has changed, and we want Datakin to have the latest metadata about tables and columns. % . % dbt debug Running with dbt=0.21.0 dbt version: 0.21.0 python version: 3.9.7

Google Cloud 52

Google Cloud Datasets Bytes Metadata 52

DoorDash Engineering

JANUARY 23, 2024

DoorDash’s internal platform team already has built many features which come in handy, like an Asgard-based microservice, which comes with a good set of built-in features like request-metadata, logging, and dynamic-value framework integration. New input formats: Currently, the platform is supporting byte-based input.

Architecture 74

Architecture Metadata Bytes Systems 74

ProjectPro

JANUARY 31, 2022

This blog walks you through what does Snowflake do , the various features it offers, the Snowflake architecture, and so much more. This layer stores the metadata needed to optimize a query or filter data. For instance, only a small number of operations, such as deleting all of the records from a table, are metadata-only.

Architecture 52

Architecture IT Data Warehouse Amazon Web Services 52

ProjectPro

NOVEMBER 22, 2021

StructType is a collection of StructField objects that determines column name, column data type, field nullability, and metadata. To define the columns, PySpark offers the pyspark.sql.types import StructField class, which has the column name (String), column type (DataType), nullable column (Boolean), and metadata (MetaData).

Hadoop 52

Hadoop Python Datasets Metadata 52

Confluent

APRIL 30, 2019

We will use his tool to generate graphical illustrations of all topologies in this blog post. Of course, this would require you to have deep knowledge of Streams DSL topology generation internals (or to have been a reader of this blog post :)) in order to make the appropriate code changes. What’s next? release.

Kafka 90

Kafka Coding Process Bytes 90

Dataquest

NOVEMBER 4, 2019

In this blog post, we’ll use data from web server logs to answer questions about our visitors. If you’re unfamiliar, every time you visit a web page, such as the Dataquest Blog , your browser is sent data from a web server. To host this blog, we use a high-performance web server called Nginx. PingdomPageSpeed/1.0

Data Pipeline 52

Data Pipeline Python Building Raw Data 52

ProjectPro

JANUARY 31, 2023

In this blog, we'll dive into some of the most commonly asked big data interview questions and provide concise and informative answers to help you ace your next big data job interview. NameNode is often given a large space to contain metadata for large-scale files. And storing these metadata in RAM will become problematic.

Big Data 52

Big Data Hadoop AWS Relational Database 52

Rockset

FEBRUARY 14, 2020

tesla-integration" You’ll notice in the results that not only will you see the lat and long you sent to the Kafka topic but some metadata that Rockset has added too including an ID, a timestamp and some Kafka metadata, this can be seen in Fig 2. select * from commons."tesla-integration" According to Postman that returned in 0.2

Kafka 40

Kafka SQL Metadata Bytes 40

ProjectPro

JULY 6, 2016

This is just a hypothetical case that we are talking about and if you prepare well, you will be able to answer any HBase Interview Question, during your next Hadoop job interview, having read ProjectPro Hadoop Interview Questions blogs. To iterate through these values in reverse order-the bytes of the actual value should be written twice.

Hadoop 40

Hadoop Bytes Metadata MongoDB 40

ProjectPro

JUNE 29, 2021

This blog brings you the most popular Kafka interview questions and answers divided into various categories such as Apache Kafka interview questions for beginners, Advanced Kafka interview questions/Apache Kafka interview questions for experienced, Apache Kafka Zookeeper interview questions, etc. What do you understand about quotas in Kafka?

Kafka 40

Kafka Bytes Big Data Java 40

nodeSWAT

JUNE 21, 2016

The following blog post is a long one, but hang in there, it will be worth it. Did you know that by default, NPM keeps all the packages and metadata it ever downloads in its cache folder indefinitely? link] So what happens is that when you install things, NPM will store the tarballs and metadata into the packages folder.

Metadata 52

Metadata Google Cloud Coding Bytes 52