Data Engineering Weekly

DECEMBER 2, 2023

[link] Amazon S3 Express One Zone is a high-performance, single-availability Zone storage class purpose-built to deliver consistent single-digit millisecond data access for your most frequently accessed data and latency-sensitive applications. The combination of stream processing + OLAP storage like Pinot. Presto tried with RaptorX.

IT 85

IT BI AWS Kafka 85

Rockset

AUGUST 5, 2022

However, we have plans to grow usage 5-10x in the next year. Using our previous serving infrastructure, the data would have to be sent through Confluent-hosted instances of Apache Kafka and ksqlDB and then denormalized and/or rolled up. Take how Rockset decouples storage from compute. Only then could we query the data.

Kafka 52

Kafka Machine Learning SQL Data Pipeline 52

ProjectPro

NOVEMBER 15, 2021

Apache Beam Source: Google Cloud Platform Apache Beam is an advanced unified programming open-source model launched in 2016. To execute pipelines, beam supports numerous distributed processing back-ends, including Apache Flink, Apache Spark , Apache Samza, Hazelcast Jet, Google Cloud Dataflow, etc.

Big Data 52

Big Data Project Metadata Programming Language 52

Monte Carlo

JUNE 14, 2022

When you have object storage in Snowflake and there is a failover, Snowflake will do the heavy lifting so there are no missing records or duplicates records during that failover. Apache Iceberg The Snowflake presenters had a lot of energy around external table support on Apace Iceberg tables. This is coming soon to preview.

Cloud 52

Cloud Data Ingestion Government Python 52

Rockset

JUNE 8, 2023

Elasticsearch has a tightly coupled architecture that does not separate compute and storage. Even managed Elasticsearch requires dealing with replication, resharding, index growth, and performance tuning of the underlying instances. This means resources are often overprovisioned because they cannot be independently scaled.

NoSQL 52

NoSQL AWS SQL Database 52