Confluent

MAY 29, 2019

Now in part 2, we’ll discuss the challenges we faced developing, building, and deploying the KSQL portion of our application and how we used Gradle to address them. In part 3, we’ll explore using Gradle to build and deploy KSQL user-defined functions (UDFs) and Kafka Streams microservices. Introduction. Sample repository. KSQL primer.

Kafka 95

Kafka Management Bytes SQL 95

Dataquest

NOVEMBER 4, 2019

In this tutorial, we’re going to walk through building a data pipeline using Python and SQL. In this blog post, we’ll use data from web server logs to answer questions about our visitors. In this blog post, we’ll use data from web server logs to answer questions about our visitors. PingdomPageSpeed/1.0

Data Pipeline 52

Data Pipeline Python Building Raw Data 52

DoorDash Engineering

JANUARY 23, 2024

With all of that in mind, we set out to build a new logistics configuration platform that could not only meet today’s demands but could continue to scale well into DoorDash’s future growth. Instead, we focused on a basic concept: Build rock-solid small components and then iterate on top of them.

Architecture 74

Architecture Metadata Bytes Systems 74

Tweag

APRIL 19, 2023

In this blog post, I will explain the underlying technical challenges and share the solution that we helped implement at kaiko.ai , a MedTech startup in Amsterdam that is building a Data Platform to support AI research in hospitals. A solution is to read the bytes that we need when we need them directly from Blob Storage.

Medical 60

Medical Process Cloud Bytes 60

Datakin

OCTOBER 14, 2021

Run models & capture lineage metadata When working with Datakin (or any other OpenLineage backend) it’s important to generate the dbt docs first. 00:27:46 | Building catalog 00:27:54 | Catalog written to /Users/rturk/projects/demo/dbt/stacko/target/catalog.json Ok! . % dbt debug Running with dbt=0.21.0 dbt version: 0.21.0

Google Cloud 52

Google Cloud Datasets Bytes Metadata 52

Confluent

APRIL 30, 2019

When building a topology with the Processor API, you explicitly name each processing node in the topology, and also provide the name(s) of all of its parent nodes (the only exception are source nodes, which do not have any parents). .< build(properties); final KafkaStreams streams = new KafkaStreams(topology, properties); streams.

Kafka 90

Kafka Coding Process Bytes 90

ProjectPro

JANUARY 31, 2022

This blog walks you through what does Snowflake do , the various features it offers, the Snowflake architecture, and so much more. Snowflake Features that Make Data Science Easier Building Data Applications with Snowflake Data Warehouse Snowflake Data Warehouse Architecture How Does Snowflake Store Data Internally?

Architecture 52

Architecture IT Data Warehouse Amazon Web Services 52

ProjectPro

NOVEMBER 22, 2021

StructType is a collection of StructField objects that determines column name, column data type, field nullability, and metadata. To define the columns, PySpark offers the pyspark.sql.types import StructField class, which has the column name (String), column type (DataType), nullable column (Boolean), and metadata (MetaData).

Hadoop 52

Hadoop Python Datasets Metadata 52

ProjectPro

JANUARY 31, 2023

In this blog, we'll dive into some of the most commonly asked big data interview questions and provide concise and informative answers to help you ace your next big data job interview. NameNode is often given a large space to contain metadata for large-scale files. But the concern is - how do you become a big data professional?

Big Data 52

Big Data Hadoop AWS Relational Database 52

ProjectPro

JULY 6, 2016

This is just a hypothetical case that we are talking about and if you prepare well, you will be able to answer any HBase Interview Question, during your next Hadoop job interview, having read ProjectPro Hadoop Interview Questions blogs. To iterate through these values in reverse order-the bytes of the actual value should be written twice.

Hadoop 40

Hadoop Bytes Metadata MongoDB 40

Rockset

FEBRUARY 14, 2020

tesla-integration" You’ll notice in the results that not only will you see the lat and long you sent to the Kafka topic but some metadata that Rockset has added too including an ID, a timestamp and some Kafka metadata, this can be seen in Fig 2. I used a pre-existing example to build the HTML file. select * from commons."tesla-integration"

Kafka 40

Kafka SQL Metadata Bytes 40

ProjectPro

JUNE 29, 2021

This blog brings you the most popular Kafka interview questions and answers divided into various categories such as Apache Kafka interview questions for beginners, Advanced Kafka interview questions/Apache Kafka interview questions for experienced, Apache Kafka Zookeeper interview questions, etc. What do you understand about quotas in Kafka?

Kafka 40

Kafka Bytes Big Data Java 40

nodeSWAT

JUNE 21, 2016

The following blog post is a long one, but hang in there, it will be worth it. Did you know that by default, NPM keeps all the packages and metadata it ever downloads in its cache folder indefinitely? link] So what happens is that when you install things, NPM will store the tarballs and metadata into the packages folder.

Metadata 52

Metadata Google Cloud Coding Bytes 52