Data Engineering Podcast

APRIL 1, 2018

I understand that your original architecture used RabbitMQ as your ingest mechanism, which you then migrated to Kafka. I understand that your original architecture used RabbitMQ as your ingest mechanism, which you then migrated to Kafka. What was your initial motivation for that change? What was your initial motivation for that change?

Amazon Web Services 100

Amazon Web Services Cloud PostgreSQL Kafka 100

Picnic Engineering

FEBRUARY 8, 2023

This setup also has a drawback as it does not offer “read isolation¹” — for example, we could see the RabbitMQ queues getting created but after a short while you could see them deleted due to a compensation transaction. 1]: read isolation: This isolation level guarantees that any data read is committed at the moment it is read.

Data Pipeline 105

Data Pipeline Kafka Data Architecture 105

Data Engineering Podcast

JULY 27, 2021

Links Pulsar Podcast Episode Streamnative Episode Datastax Astra Streaming Datastax Astra DB Luna Streaming Distribution Datastax Cassandra Kesque (formerly Kafkaesque) Kafka RabbitMQ Prometheus Grafana Pulsar Heartbeat Pulsar Summit Pulsar Summit Presentation on Kafka Connectors Replicated Chaos Engineering Fallout chaos engineering tools Jepsen Podcast (..)

Building 100

Building Management Kafka Data Warehouse 100

Data Engineering Podcast

NOVEMBER 26, 2019

To help other people find the show please leave a review on iTunes and tell your friends and co-workers Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat Links Sentry Podcast.__init__

Data Warehouse 100

Data Warehouse Building PostgreSQL Kafka 100

Zalando Engineering

JANUARY 31, 2018

How we deployed RabbitMQ on AWS In an effort to move away from our legacy monolithic service, we decided take on the challenge of building a new communication platform based on a micro service architecture, which would be more focused and more easily manageable. This allows for many interesting and useful scenarios.

Cloud 40

Cloud AWS Programming Language Utilities 40

Data Engineering Podcast

FEBRUARY 3, 2018

Contact Info Matteo merlimat on GitHub @merlimat on Twitter Rajan @dhabaliaraj on Twitter rhabalia on GitHub Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?

Kafka 100

Kafka Data Pipeline Data Science Data Engineering 100

Data Engineering Podcast

FEBRUARY 3, 2018

Contact Info Matteo merlimat on GitHub @merlimat on Twitter Rajan @dhabaliaraj on Twitter rhabalia on GitHub Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?

Kafka 100

Kafka Data Pipeline Data Science Data Engineering 100

Data Engineering Podcast

MARCH 16, 2020

Links Apache CouchDB FoundationDB Podcast Episode IBM Cloudant Experimental Particle Physics FPGA == Field Programmable Gate Array Apache Software Foundation CRDT == Conflict-free Replicated Data Type Podcast Episode Erlang Riak RabbitMQ Heisenbug Kubernetes Property Based Testing The intro and outro music is from The Hug by The Freak Fandango Orchestra (..)

Building 100

Building Data Warehouse NoSQL Data Lake 100

Data Engineering Podcast

MARCH 25, 2019

Links CluedIn Copenhagen, Denmark A/B Testing Data Fabric Dataiku RapidMiner Azure Machine Learning Studio CRM (Customer Relationship Management) Graph Database Data Lake GraphQL DGraph Podcast Episode RabbitMQ GDPR (General Data Protection Regulation) Master Data Management Podcast Interview OAuth Docker Kubernetes Helm DevOps DataOps DevOps vs DataOps (..)

Building 100

Building Data Lake Machine Learning Kafka 100

Scott Logic

NOVEMBER 14, 2023

It’s a slighty grey area because: Arguably a broker+queues product like RabbitMQ is a pub/sub system but it is slightly different to my definition above in that the consumer is listening to specific queues and also the consumer does the initial connection Some services offer both push and pull so this isn’t a perfect classification.

Technology 52

Technology Kafka AWS Data 52

Towards Data Science

FEBRUARY 18, 2024

AMQP-based message brokers , like RabbitMQ , are commonly employed for asynchronous message passing between services and task queues. A message broker is a widely adopted system for messaging, in which producers write messages to the broker, and consumers are notified by the broker and receive these messages.

Kafka 80

Kafka Python Process Google Cloud 80

ProjectPro

JUNE 29, 2021

In cases where a simple task queue is needed, there are better alternatives like the RabbitMQ. RabbitMQ vs. Kafka Interview Questions Messaging systems like Apache Kafka and RabbitMQ allow you to manage big data streams in distributed computing, including tasks such as data consumption, reading, writing, processing, etc.

Kafka 40

Kafka Bytes Big Data Java 40

Knowledge Hut

MARCH 20, 2024

Application queuing is handled by RabbitMQ. Pinterest A creative web-based application that has come a long way to explore the lifestyle sections like recipes, home ideas, style inspiration, and many others is Pinterest. Originally it was made by Python and Django. The engineers used Redis and Membase to handle catching.

MongoDB 52

MongoDB Programming Language Java Python 52

Scott Logic

APRIL 22, 2024

In my previous blog post I looked at various technologies for sending data asynchronously between services including RabbitMQ, Kafka, AWS EventBridge. This time round I’ll look at the messages themselves which over the last few years I’ve found to be a more complex and nuanced topic than expected.

AWS 52

AWS Metadata Systems Kafka 52

The Pragmatic Engineer

JUNE 13, 2023

Here’s a breakdown of employee numbers from Idan: Tens of people (between 10 and 30) maintaining hardware 25-30 people maintaining data infrastructure like Kafka or RabbitMQ.

Cloud 189

Cloud Utilities Database BI 189

Scott Logic

JUNE 6, 2023

Open source components such as Open Messaging benchmarking tools for comparing middleware are starting to appear (currently Apache RocketMQ, Kafka, Pulsar, RabbitMQ, NATS streaming, REDIS and Pravega).

Architecture 69

Architecture Coding Designing Systems 69

Knowledge Hut

MAY 31, 2023

Building this type of platform requires integration of front-end frameworks like ReactJS and AngularJS together with back-end tools such as NodeJS, ExpressJS, MongoDB, Redis, and RabbitMQ alongside exposing RESTful API interfaces for mobile application synchronization and data exchange with mainframes maintaining historical transaction details and (..)

Database 52

Database Coding MongoDB Project 52

Scott Logic

NOVEMBER 6, 2023

We start with a table summaring the various categories and providing some examples: Tech Use Examples Message buses Distribute data from services to other users/systems AWS Eventbridge, Kafka, RabbitMQ, Azure Service Bus Workflows and orchestrating ETL Where there's dependencies between data you want to pull into your warehouse and to orchestrate the (..)

Data Lake 52

Data Lake Kafka BI Unstructured Data 52

ProjectPro

JULY 21, 2015

Related Posts Apache Kafka Architecture and Its Components-The A-Z Guide Kafka vs RabbitMQ - A Head-to-Head Comparison for 2021 HBase vs Cassandra-The Battle of the Best NoSQL Databases PREVIOUS NEXT < If they need real time processing of ad-hoc queries on subset of data then Impala is a better choice.

Hadoop 40

Hadoop SQL NoSQL Kafka 40

Rockset

SEPTEMBER 25, 2020

Change streams don’t require the use of a pub-sub (publish-subscribe) model like Kafka and RabbitMQ do. This tool is much more durable during network outages, when it uses resume tokens that help keep track of where your change stream was last pulled from. You can still use the PyMongo library to interface with MongoDB.

MongoDB 52

MongoDB Kafka Database NoSQL 52

Picnic Engineering

MARCH 14, 2023

It communicates with the transport layer through an asynchronous, vendor-specific messaging protocol over RabbitMQ. The WCS is an in-house developed application, and the service under test. WCS is notified of crates that are scanned that are not in transport, and provides transport instructions.

Python 59

Python Transportation Systems Algorithm 59

LinkedIn Engineering

MARCH 14, 2023

The architecture relied heavily on external protocols/solutions ( RabbitMQ , to be precise) for solving an issue. There were multiple requests to revert a request for reimaging that couldn't complete, as per the expected SLA of end-users, because service at that site was busy processing hosts recently “Discovered” / newly racked.

MySQL 77

MySQL Architecture Database Relational Database 77

ProjectPro

JUNE 28, 2016

These qualities make it stand out from traditional message brokers such as RabbitMQ, JMS and APMQ. High fault tolerance is one of the key features desired in a real time messaging system and Kafka ticks that box as well. Table of Contents What makes Kafka popular? Kafka – a perfect candidate for Distributed Messaging What Kafka does?

Kafka 40

Kafka Systems Hadoop BI 40

Zalando Engineering

FEBRUARY 24, 2021

It is very easy to run Docker images from Java code.

PostgreSQL 40

PostgreSQL Java AWS Database 40

ProjectPro

MARCH 17, 2014

Recommended Reading: Apache Kafka Architecture and Its Components-The A-Z Guide Kafka vs RabbitMQ - A Head-to-Head Comparison 15 AWS Projects Ideas for Beginners to Practice Data Lake vs Data Warehouse - Working Together in the Cloud How to Become a Big Data Engineer Big Data Engineer Salary - How Much Can You Make?

Hadoop 52

Hadoop Big Data Unstructured Data Retail 52

Rock the JVM

AUGUST 9, 2022

Logs-As-A-Stream Many messaging platforms, such as Kafka, Pulsar, and/or RabbitMQ have what they advertise as Stream s. However, the most important take-away from this feature, is that memoization is not in our broad definition above - this is not a feature of all streams.

Scala 40

Scala Bytes Kafka Programming 40

ProjectPro

MARCH 23, 2015

Top 100 Hadoop Interview Questions and Answers 2016 Difference between Hive and Pig - The Two Key components of Hadoop Ecosystem Make a career change from Mainframe to Hadoop - Learn Why Apache Kafka Architecture and Its Components-The A-Z Guide Kafka vs RabbitMQ - A Head-to-Head Comparison for 2021 PREVIOUS NEXT <

Hadoop 40

Hadoop Big Data Architecture Metadata 40

ProjectPro

OCTOBER 15, 2014

Make a career change from Mainframe to Hadoop - Learn Why Apache Kafka Architecture and Its Components-The A-Z Guide Kafka vs RabbitMQ - A Head-to-Head Comparison for 2021 HBase vs Cassandra-The Battle of the Best NoSQL Databases PREVIOUS NEXT <

Hadoop 52

Hadoop Unstructured Data Java SQL 52

Confluent

SEPTEMBER 26, 2019

explains in more detail why many new integration architectures leverage Apache Kafka instead of legacy tools like RabbitMQ, ETL, and ESB. Enterprise Service Bus (ESB) – Friends, Enemies or Frenemies?

Kafka 21

Kafka BI SQL Datasets 21

phData: Data Engineering

JANUARY 3, 2022

Common examples include: Apache Kafka AWS Kinesis and AWS Kinesis Firehose RabbitMQ Once data has been stored, it will generally need to be processed to reach the desired state. This is often a pub/sub architecture that allows multiple consumers of data to receive updates from a publisher of data.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Governance 52

Cloudera

AUGUST 14, 2020

The Kraken fault agent installed on SUT nodes registers itself to the Kraken’s nodes service and listens for incoming messages on a RabbitMQ queue. The system/application under test nodes need not open SSH port for communication from test automation or any fault injection triggers.

Hadoop 96

Hadoop Bytes Metadata Programming Language 96

ProjectPro

NOVEMBER 7, 2016

Recommended Reading: Apache Kafka Architecture and Its Components-The A-Z Guide Kafka vs. RabbitMQ - A Head-to-Head Comparison for 2021 Big Data Engineer Salary - How Much Can You Make in 2021? For the architecture to be performance efficient, HDFS must satisfy specific pre-requisites – All the hard drives should have a high throughput.

Hadoop 40

Hadoop Architecture IT Big Data 40

AltexSoft

OCTOBER 21, 2022

popular SQL and NoSQL database management systems including Oracle, SQL Server, Postgres, MySQL, MongoDB, Cassandra, and more; cloud storage services — Amazon S3, Azure Blob, and Google Cloud Storage; message brokers such as ActiveMQ, IBM MQ, and RabbitMQ; Big Data processing systems like Hadoop ; and. You can find off-the-shelf links for.

Kafka 93

Kafka Hadoop ETL Tools Big Data 93

Confluent

FEBRUARY 26, 2019

He has been a committer on Spring Integration since 2010 and has led that project for several years, in addition to leading Spring for Apache Kafka and Spring AMQP (Spring for RabbitMQ).

Kafka 110

Kafka Project Programming Coding 110

AltexSoft

NOVEMBER 7, 2022

Celery is an asynchronous task queue for Python programs but to use it you need to set up a message broker — RabbitMQ or Redis. Two main remote options are Celery Executor and Kubernetes Executor.

PostgreSQL 52

PostgreSQL Metadata Python MySQL 52

Confluent

JUNE 26, 2019

Middleware can be anything—some custom glue code or framework, a messaging system like RabbitMQ, an ETL tool like Talend, an ESB like WSO2, or an event streaming platform like Apache Kafka. Due to this, many architects use middleware as a backbone for microservice communication to create decoupled, scalable, and highly available systems.

Kafka 109

Kafka Designing Architecture ETL Tools 109

Confluent

MAY 30, 2019

Built with RabbitMQ or the Apache Kafka Spring Cloud Stream binder. A developer can directly use or extend any of the out-of-the-box utility event streaming applications to cover common use cases or write a custom application using Spring Cloud Stream. Built with Prometheus and InfluxDB monitoring systems.

Kafka 94

Kafka Cloud Data Pipeline PostgreSQL 94