Data Engineering Weekly

MARCH 31, 2024

link] DoorDash: Setting Up Kafka Multi-Tenancy DoorDash writes an insightful article about setting Multi-Tenancy in the Kafka cluster and the role of a consumer proxy in setting this up. Polars written in Rust offer 5-10X more performance than Pandas!! The problem statement is certainly pretty interesting.

Data Engineering 62

Data Engineering Data Engineer Engineering Scala 62

Data Engineering Weekly

DECEMBER 2, 2023

S3 Express One Zone can improve data access speeds by 10x and reduce request costs by 50% compared to S3 Standard and scales to process millions of requests per minute. The LakeHouse tools, such as Apache Hudi, support incremental querying to reduce the latency. DuckDB is the recent attempt to build in-process OLAP engines.

IT 85

IT BI AWS Kafka 85

LinkedIn Engineering

APRIL 18, 2023

Figure 1: Salt master minion flow via event bus In this post, we will share how we scaled Salt by adding our layers and integrating it with LinkedIn infrastructure to achieve 10x more remote execution for automating any kind of onDemand operational jobs on host, with more scalabilty than before.

MySQL 103

MySQL Python Bytes Kafka 103

Rockset

JANUARY 28, 2022

Due to remote learning, most of our key metrics grew by 10X,” Sjogreen said in a video interview with SiliconANGLE’s theCUBE as part of the AWS Startup Showcase in September 2021. How E-Learning Platform, Seesaw, Scaled 10x During Shutdown with Rockset & Hightouch. That’s observability. Watch the webinar below.

NoSQL 52

NoSQL PostgreSQL MongoDB ETL Tools 52

Rockset

AUGUST 5, 2022

However, we have plans to grow usage 5-10x in the next year. Using our previous serving infrastructure, the data would have to be sent through Confluent-hosted instances of Apache Kafka and ksqlDB and then denormalized and/or rolled up. The bigger pain point was the high operational overhead of Elasticsearch for our small team.

Kafka 52

Kafka Machine Learning SQL Data Pipeline 52

ProjectPro

NOVEMBER 15, 2021

Apache Beam Source: Google Cloud Platform Apache Beam is an advanced unified programming open-source model launched in 2016. To execute pipelines, beam supports numerous distributed processing back-ends, including Apache Flink, Apache Spark , Apache Samza, Hazelcast Jet, Google Cloud Dataflow, etc.

Big Data 52

Big Data Project Metadata Programming Language 52

Monte Carlo

JUNE 14, 2022

Apache Iceberg The Snowflake presenters had a lot of energy around external table support on Apace Iceberg tables. Native Apache Iceberg tables are a high-performance format for huge analytic tables. This goes to private preview very soon, too. Christian then announced a new way to ingest data in Snowflake: Snowpipe streaming.

Cloud 52

Cloud Data Ingestion Government Python 52

Rockset

JUNE 8, 2023

Learn more about using DynamoDB with Rockset eBook: Real-time Analytics on DynamoDB: The Ultimate Guide DynamoDB Filtering and Aggregation Queries Using SQL on Rockset Case Study: Real-Time Insights Help Propel 10X Growth at E-Learning Provider Seesaw This can significantly speed up the development of real-time applications.

NoSQL 52

NoSQL AWS SQL Database 52

Cloudera

MAY 22, 2018

In the realm of machine learning, for example, data scientists can now accelerate deep learning by 5x-10x by utilizing specialized resources like GPUs. We’re seriously upgrading our search capabilities with Apache Solr 7 to provided integrated search alongside our Analytic Database and your business analysts’ preferred choice of BI tools.

Building 40

Building Data Warehouse Government Business Analyst 40