article thumbnail

Monitoring Data Replication in Multi-Datacenter Apache Kafka Deployments

Confluent

Previously in 3 Ways to Prepare for Disaster Recovery in Multi-Datacenter Apache Kafka Deployments , we provided resources for multi-datacenter designs, centralized schema management, prevention of cyclic repetition of messages, and automatic consumer offset translation to automatically resume applications.

Kafka 86
article thumbnail

How to learn data engineering

Christophe Blefari

Main technologies around stream are bus messages like Kafka and processing framework like Flink or Spark on top of the bus. Some people wrote a white paper detailing Machine Learning Operations (MLOps): Overview, Definition, and Architecture in which they write about rols and missions. workflows (Airflow, Prefect, Dagster, etc.)

Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

article thumbnail

How to configure clients to connect to Apache Kafka Clusters securely – Part 2: LDAP

Cloudera

In the previous post, we talked about Kerberos authentication and explained how to configure a Kafka client to authenticate using Kerberos credentials. In this post we will look into how to configure a Kafka client to authenticate using LDAP, instead of Kerberos. We use the Kafka-console-consumer for all the examples below.

Kafka 52
article thumbnail

Cloudera DataFlow’s key milestones and wins in 2020

Cloudera

Spin up clusters of NiFi, Kafka, or Flink very quickly onto your public cloud environments on AWS or Azure. Extending the Kafka ecosystem to CDP – Cloudera (and Hortonworks) has always prided itself on its investment and innovations around Apache Kafka. If you are interested, you can watch it on-demand as well.

Kafka 60
article thumbnail

Sysmon Security Event Processing in Real Time with KSQL and HELK

Confluent

HELK is a free threat hunting platform built on various components including the Elastic stack, Apache Kafka ® and Apache Spark. WHERE PARENT_PROCESS_PATH LIKE '%WmiPrvSE.exe%'; The results of the KSQL query can be written to a Kafka topic, which in turn can drive real-time monitoring or alerting dashboards and applications.

Process 81
article thumbnail

How to configure clients to connect to Apache Kafka Clusters securely – Part 3: PAM authentication

Cloudera

In the previous posts in this series, we have discussed Kerberos and LDAP authentication for Kafka. In this post, we will look into how to configure a Kafka cluster to use a PAM backend instead of an LDAP one. TLS is assumed to be enabled for the Apache Kafka cluster, as it should be for every secure cluster. PAM Authentication.

Kafka 75
article thumbnail

How to configure clients to connect to Apache Kafka Clusters securely – Part 4: TLS Client Authentication

Cloudera

In the previous posts in this series, we have discussed Kerberos , LDAP and PAM authentication for Kafka. In this post we will look into how to configure a Kafka cluster and client to use a TLS client authentication. TLS is assumed to be enabled for the Apache Kafka cluster, as it should be for every secure cluster.

Kafka 79