Big Data Tools

MAY 19, 2022

Based on the “map-reduce” paradigm, they allow you to compute the next DAGs from the current state – a very useful feature, which incidentally has been available in Luigi for a while. Kyuubi 1.5.1 – Kyuubi is a JDBC server built over Apache Spark, but as of version 1.5.0, RocketMQ Streams 1.0.1

Data Engineering 52

Data Engineering Data Engineer Engineering Big Data Tools 52

Big Data Tools

AUGUST 3, 2021

Kotlin API for Apache Spark – A year after showing you the first preview, we released version 1.0. Apache Spark already has two official APIs for JVM – Scala and Java – but we’re hoping the Kotlin API will be useful as well, as we’ve introduced several unique features. They’ve added some interesting features in this version.

Data Engineering 52

Data Engineering Data Engineer Engineering Big Data Tools 52

Big Data Tools

AUGUST 3, 2021

Kotlin API for Apache Spark – A year after showing you the first preview, we released version 1.0. Apache Spark already has two official APIs for JVM – Scala and Java – but we’re hoping the Kotlin API will be useful as well, as we’ve introduced several unique features. They’ve added some interesting features in this version.

Data Engineering 52

Data Engineering Data Engineer Engineering Big Data Tools 52

Cloudera

FEBRUARY 9, 2021

Cloudera, an innovator in providing data products to address these types of challenges, has introduced some new products, such as Kudu low-latency storage and Druid high-performance real-time analytics database that have been successfully implemented in the customer on-premises data centres for these types of real-time data warehousing use cases.

Data Warehouse 67

Data Warehouse Cloud Kafka Cloud Storage 67

Rockset

JANUARY 23, 2020

For databases such as CockroachDB, having a reliable, high-performance way to store and retrieve data from stable storage is essential. The advantages of RocksDB over other store engines are: Technical design. A Brief History of Distributed Databases The era of Web 2.0 brought with it a renewed interest in database design.

Database 40

Database MySQL Kafka NoSQL 40

Zalando Engineering

NOVEMBER 29, 2017

Second in our series about the use of Apache Kafka’s Streams API by Zalando This is the second in a series about the use of Apache Kafka’s Streams API by Zalando, Europe’s leading online fashion platform. See Ranking Websites in Real-time with Apache Kafka’s Streams API for the first post in the series.

Kafka 40

Kafka AWS Amazon Web Services Utilities 40

Monte Carlo

JULY 7, 2022

You can read more about Convoy’s approach from our blog with their Head of Product, Data Platform, Chad Sanderson, “ The modern data warehouse is broken.” You can read more about Convoy’s approach from our blog with their Head of Product, Data Platform, Chad Sanderson, “ The modern data warehouse is broken.”

Data Warehouse 52

Data Warehouse Software Engineer Software Engineering Data 52

Rockset

MARCH 27, 2023

This is part two in Rockset’s Making Sense of Real-Time Analytics on Streaming Data series. In part 1 , we covered the technology landscape for real-time analytics on streaming data. In this post, we’ll explore the differences between real-time analytics databases and stream processing frameworks. With that, let’s dive in.

Database 52

Database Process Scala SQL 52

Confluent

APRIL 30, 2019

With the release of Apache Kafka ® 2.1.0, Kafka Streams introduced the processor topology optimization framework at the Kafka Streams DSL layer. This framework opens the door for various optimization techniques from the existing data stream management system (DSMS) and data stream processing literature.

Kafka 90

Kafka Coding Process Bytes 90

Snowflake

MARCH 30, 2023

In this blog post we will use what we have learned in this Data Vault blog series to support the data preparation requirements for ML on Snowflake, using Data Vault patterns for modeling and automation. For this reason, a new data management for ML framework has emerged to help manage this complexity: the “feature store.”

Engineering 81

Engineering Raw Data Data Science Scala 81

LinkedIn Engineering

MAY 9, 2023

Instead, we chose to introduce Couchbase as a centralized storage tier cache for read scaling. In this blog post we’ll discuss our decision to leverage Couchbase, the challenges that arose, and how we addressed each challenge in our final solution. Member profiles are stored in Espresso in Avro binary format.

Database 133

Database Project Datasets Designing 133

dbt Developer Hub

NOVEMBER 3, 2022

Here are a few things we’ll address: Why you may want to use an architecture like this How to structure your event messages How to use dbt macros to make it easy to ingest new event streams Event Streams at Merit ​ At Merit, we’re building the leading verified identity platform. One key focus of our platform is data quality.

Kafka 52

Kafka ETL Tools BI Database 52

Cloudera

OCTOBER 19, 2022

In this post I will demonstrate how Kafka Connect is integrated in the Cloudera Data Platform (CDP), allowing users to manage and monitor their connectors in Streams Messaging Manager while also touching on security features such as role-based access control and sensitive information handling. Kafka Connect.

Kafka 71

Kafka MySQL Bytes Data 71

Confluent

MAY 9, 2019

Storing events in a stream and connecting streams via stream processors provide a generic, data-centric, distributed application runtime that you can use to build ETL, event streaming applications, applications for recording metrics and anything else that has a real-time data requirement.

Kafka 93

Kafka Pipeline-centric Architecture Database-centric 93

Big Data Tools

FEBRUARY 9, 2022

The one remaining free tool I’m aware of is Arenadata Cluster Manager , but the free version doesn’t allow the user to do certain things, like deploy HA name nodes. We all know Apache NiFi, a stream processing tool with its own processing engine. Here’s what’s happening in the world of data engineering right now.

Data Engineering 52

Data Engineering Data Engineer Engineering Big Data Tools 52

Big Data Tools

FEBRUARY 9, 2022

The one remaining free tool I’m aware of is Arenadata Cluster Manager , but the free version doesn’t allow the user to do certain things, like deploy HA name nodes. We all know Apache NiFi, a stream processing tool with its own processing engine. Here’s what’s happening in the world of data engineering right now.

Data Engineering 52

Data Engineering Data Engineer Engineering Big Data Tools 52

Confluent

FEBRUARY 13, 2019

Once settled on the event streaming approach, I’ll provide a high-level dataflow of how we design systems for payment processing at scale using this approach. The paradigm shift to the event streaming architecture often leads to a lightbulb moment of inspiration and clarity. Event-driven, streaming architecture.

Architecture 90

Architecture Programming Kafka Database-centric 90

ProjectPro

NOVEMBER 15, 2021

This blog will walk through the most popular and fascinating open source big data projects. It derives its name “Beam” which is from “Batch” + “Stream” from its functionalities for both batch and streaming the parallel processing pipelines for data.

Big Data 52

Big Data Project Metadata Programming Language 52

phData: Data Engineering

JANUARY 3, 2022

A data strategy is an evolving set of tools, processes, rules, and regulations that define how a company collects, stores, transforms, manages, shares, and utilizes data. Your business likely has competitors that are trying to beat you to market, technology is constantly evolving, and so are your customers. How do we minimize costs?

IT 52

IT AWS Software Engineer Software Engineering 52

Confluent

JULY 31, 2019

Delivers the new Confluent Operator for cloud-native automation on Kubernetes, a redesigned Confluent Control Center user interface to simplify how you manage event streams, and a preview of Role-Based Access Control for enterprise-grade security. Automate Kafka operations for production environments. Secure the platform end to end.

Kafka 19

Kafka Cloud Accessible Accessibility 19

ProjectPro

NOVEMBER 7, 2016

This blog will give you an indepth insight into the architecture of hadoop and its major components- HDFS, YARN, and MapReduce. This blog post explains the Hadoop architecture and the factors to consider when designing and building a Hadoop cluster for production success. Understanding the Hadoop architecture now gets easier!

Hadoop 40

Hadoop Architecture IT Big Data 40

Zalando Engineering

JANUARY 22, 2018

While this can already be interesting with “traditional” designs, the move to event-driven architectures can magnify these difficulties, especially with simplistic event streams. It consists of many smaller components focused on the ultimate goal of making articles sellable in Zalando’s online stores.

Media 40

Media Kafka Architecture Designing 40

ProjectPro

JULY 27, 2021

This blog is your one-stop solution for the top 100+ Data Engineer Interview Questions and Answers. In this blog, we have collated the frequently asked data engineer interview questions based on tools and technologies that are highly useful for a data engineer in the Big Data industry. that leverage big data analytics and tools.

Data Engineering 40

Data Engineering Data Engineer Engineering Hadoop 40