dbt Developer Hub

NOVEMBER 3, 2022

Let’s discuss how to convert events from an event-driven microservice architecture into relational tables in a warehouse like Snowflake. In this blog post we’ll dive into how we tackled one source of quality issues: directly relying on upstream database schemas. One key focus of our platform is data quality.

Kafka 52

Kafka ETL Tools BI Database 52

Picnic Engineering

SEPTEMBER 8, 2022

Almost all internal services emit events over RabbitMQ. Our pipeline captures these events and sends them to Confluent Cloud. We use the RabbitMQ Source connector for Apache Kafka Connect. Finally, the Snowflake sink connector picks the messages from topics and loads them into respective tables in DWH.

Kafka 52

Kafka Metadata AWS Java 52

Rockset

MARCH 27, 2023

This blog will clarify some conceptual differences, provide an overview of popular tools, and offer a framework for deciding which tools are best suited for specific technical requirements. These databases can ingest streaming events, index the data, and enable millisecond-latency analytical queries against that data.

Database 52

Database Process Scala SQL 52

Cloudera

APRIL 20, 2021

Streaming Analytics is a type of data analysis that processes data streams for real-time analytics. It continuously processes data from multiple streams and performs simple calculations to complex event processing for delivering sophisticated use cases. What are the business challenges with today’s data?

Hospitality 93

Hospitality Kafka Retail Data Ingestion 93

Rockset

OCTOBER 11, 2022

In this blog, we’ll compare and contrast how Elasticsearch and Rockset handle data ingestion as well as provide practical techniques for using these systems for real-time analytics. Logstash is an event processing pipeline that ingests and transforms data before sending it to Elasticsearch.

Data Ingestion 52

Data Ingestion Kafka Relational Database PostgreSQL 52

Netflix Tech

OCTOBER 15, 2019

Part I: Overview Andreas Andreakis , Falguni Jhaveri , Ioannis Papapanagiotou , Mark Cho , Poorna Reddy , Tongliang Liu Overview It is a commonly observed pattern for applications to utilize multiple datastores where each is used to serve a specific need such as storing the canonical form of data (MySQL etc.), caching (Memcached etc.),

Transportation 77

Transportation MySQL Kafka Data 77

Snowflake

MARCH 30, 2023

Data Vault’s “soft rules” are those auditable transformations that use raw (and other business vault) data artifacts whose outcomes are loaded into a business vault based on the same business entity and unit of work as raw vault. “The features you use influence more than everything else the result. ML workflow, ubr.to/3EJHjvm

Engineering 80

Engineering Raw Data Data Science Scala 80

ProjectPro

JUNE 29, 2021

Your search for Apache Kafka interview questions ends right here! Let us now dive directly into the Apache Kafka interview questions and answers and help you get started with your Big Data interview preparation! How to study for Kafka interview? What is Kafka used for? What are main APIs of Kafka?

Kafka 40

Kafka Bytes Big Data Java 40

ProjectPro

JANUARY 25, 2022

Here’s What You Need to Know About PySpark This blog will take you through the basics of PySpark, the PySpark architecture, and a few popular PySpark libraries , among other things. Did you know that, according to Linkedin, over 24,000 Big Data jobs in the US list Apache Spark as a required skill? Why use PySpark? What is PySpark?

Big Data 52

Big Data Data Process Process Kafka 52

Pinterest Engineering

AUGUST 18, 2022

The adoption of Unified Storage Service powered by TiDB is a major challenging project spanning over multiple quarters. In this blog post, we will first learn the various approaches considered for data migration with their trade offs. cannot serve the needs of our clients for the next 3–5 years. Do a validation of both data sets.

Data Ingestion 52

Data Ingestion Hadoop Database Kafka 52

Zalando Engineering

MAY 7, 2018

Real-time joins in event-driven microservices As discussed in my previous blog post , Kafka is one of the key components of our event-driven microservice architecture in Zalando’s Smart Product Platform. We use it for sequencing events and building an aggregated view of data hierarchies.

Kafka 40

Kafka Scala Java Media 40

Rockset

MARCH 19, 2020

Intro In recent years, Kafka has become synonymous with “streaming,” and with features like Kafka Streams, KSQL, joins, and integrations into sinks like Elasticsearch and Druid, there are more ways than ever to build a real-time analytics application around streaming data in Kafka.

Kafka 40

Kafka Database Process SQL 40

ProjectPro

AUGUST 24, 2021

And, out of these professions, this blog will discuss the data engineering job role. Most of us have observed that data scientist is usually labeled the hottest job of the 21st century, but is it the only most desirable job? Within no time, most of them are either data scientists already or have set a clear goal to become one.

Data Engineering 52

Data Engineering Data Engineer Coding Project 52

Rockset

OCTOBER 1, 2020

This brand has multiple stores across the country and a website allowing customers to order items for delivery or store pick up. marketing_emails (Kafka): Marketing data including items sent and any interactions. weblogs (Kafka): Website analytics such as logins, browsing, purchases and any errors.

MongoDB 40

MongoDB Kafka Building Data Warehouse 40

Confluent

AUGUST 29, 2019

We know that Apache Kafka ® is great when you’re dealing with streams, allowing you to conveniently look at streams as tables. Kafka already allows you to look at data as streams or tables; graphs are a third option, a more natural representation with a lot of grounding in theory for some use cases.

Kafka 55

Kafka Process Algorithm Cloud 55

Rockset

DECEMBER 19, 2023

This blog post distills his decade of experience into a comprehensive read, offering a detailed overview of the complexities and innovations at every stage of building a real-world recommender system. Stage 2: Ranking - We rank these candidates using some heuristic to pick the top 10 to 50 items.

Systems 52

Systems Machine Learning Deep Learning Media 52

ProjectPro

NOVEMBER 15, 2021

This blog will walk through the most popular and fascinating open source big data projects. If you are associated with the tech world, you would have unquestionably come across the term “Open-source.” When any particular project is open-sourced, it makes the source code accessible to anyone.

Big Data 52

Big Data Project Metadata Programming Language 52

Monte Carlo

JULY 7, 2022

Editor’s Note : We ran into Andrew at our London IMPACT event in early 2022. Data contracts have since became one of the most discussed topics in data engineering. You can read more about Convoy’s approach from our blog with their Head of Product, Data Platform, Chad Sanderson, “ The modern data warehouse is broken.”

Data Warehouse 52

Data Warehouse Software Engineer Software Engineering Data 52

ProjectPro

DECEMBER 10, 2021

This blog covers the top 50 most frequently asked Azure interview questions and answers. Well, this Azure interview questions and answers blog will help you land your dream cloud computing job role! It will provide you with a good sense of what areas you should focus on as you prepare for your next Azure interview.

BI 52

BI Cloud Computing SQL Database 52

Zalando Engineering

JANUARY 18, 2017

In many computer programs, the whole logic (or a vast part of it) is essentially the step-by-step processing of data. Of course, this includes the situation when we iterate over the data and just execute the processing logic on every piece of it. This was one of the rationales behind frameworks like Apache Camel or Apache Storm. via(Flow[String].map(s

Scala 52

Scala AWS Coding Data Process 52

ProjectPro

MAY 31, 2021

This blog lists over 20 big data projects you can work on to showcase your big data skills and gain hands-on experience in big data tools and technologies. Ace your big data interview by adding some unique and exciting Big Data projects to your portfolio. None of this would have been possible without the application of big data.

Big Data 52

Big Data Coding Project Hadoop 52

Confluent

AUGUST 27, 2019

We are excited to announce the release of Confluent Cloud Schema Registry in general availability (GA), available in Confluent Cloud , our fully managed event streaming service based on Apache Kafka ®. In contrast, Confluent Cloud Schema Registry is backed by Confluent-managed Kafka clusters, where schemas are stored.

Cloud 18

Cloud Kafka Electronics Metadata 18

phData: Data Engineering

JANUARY 3, 2022

We’ll work through the different facets of taking your data and extracting business value with the same rigor and process companies apply to product development. Just click this button and fill out the form to download it. You can read the full guide without giving us your email — keep scrolling !) What’s Data Strategy? Why is that?

IT 52

IT AWS Software Engineer Software Engineering 52

Confluent

SEPTEMBER 26, 2019

In the early days, many companies simply used Apache Kafka ® for data ingestion into Hadoop or another data lake. However, Apache Kafka is more than just messaging. Some Kafka and Rockset users have also built real-time e-commerce applications , for example, using Rockset’s Java, Node.js

Kafka 20

Kafka BI SQL Datasets 20

Confluent

MAY 21, 2019

This leads us to event streaming microservices patterns. In this case, a simple solution will be to publish the exact same information that it used to send in the HTTP request: { user_id: 53, timestamp: 1497842472, address: “2 Elm St. An example can be the backend architecture for an insurance product. Chattanooga, TN” }.

Kafka 110

Kafka Insurance Architecture Metadata 110

Confluent

MAY 30, 2019

Following part 1 and part 2 of the Spring for Apache Kafka Deep Dive blog series, here in part 3 we will discuss another project from the Spring team: Spring Cloud Data Flow , which focuses on enabling developers to easily develop, deploy, and orchestrate event streaming pipelines based on Apache Kafka ®.

Kafka 93

Kafka Cloud Data Pipeline PostgreSQL 93

Confluent

JUNE 11, 2019

For event streaming application developers, it is important to continuously update the streaming pipeline based on the need for changes in the individual applications in the pipeline. It is also important to understand some of the common streaming topologies that streaming developers use to build an event streaming pipeline.

Kafka 84

Kafka Cloud Java MongoDB 84

AltexSoft

OCTOBER 21, 2022

Kafka can continue the list of brand names that became generic terms for the entire type of technology. In this article, we’ll explain why businesses choose Kafka and what problems they face when using it. What is Kafka? How Apache Kafka streams relate to Franz Kafka’s books. What Kafka is used for.

Kafka 93

Kafka Hadoop ETL Tools Big Data 93

ProjectPro

JULY 30, 2021

As an aspiring machine learning professional, a portfolio is the most important asset to have in your job search. But what if you don’t have a machine learning portfolio because you are going to need diverse skills and projects under your belt to land a top machine learning gig.

Machine Learning 40

Machine Learning Project Deep Learning Banking 40

Confluent

APRIL 4, 2019

Event-driven architecture means just that: It’s all about the events. In a microservices architecture, events drive microservice actions. No event, no shoes, no service. In the most basic scenario, microservices that need to take action on a common stream of events all listen to that stream.

Kafka 108

Kafka Data Cleanse Retail Finance 108

Confluent

JULY 10, 2019

As discussed in part 2, I created a GitHub repository with Docker Compose functionality for starting a Kafka and Confluent Platform environment, as well as the code samples mentioned below. gradlew composeUp. KSQL user-defined functions. While the CASE syntax now available since KSQL 5.2.2

Kafka 86

Kafka Java Bytes SQL 86

Cloudera

JANUARY 11, 2021

I want to thank you all for joining and attending these events! I received hundreds of questions during these events, and my colleagues and I tried to answer as many as we could. Interactive demo sessions and live Q&A are what we all need these days when working remotely from home is now a norm.

Kafka 60

Kafka Data Cleanse Java Datasets 60

Confluent

JUNE 18, 2019

Distributed transactions are very hard to implement successfully, which is why we’ll introduce a log-inspired system such as Apache Kafka ®. Building an indexing pipeline at scale with Kafka Connect. Software projects of all sizes and complexities have a common challenge: building a scalable solution for search. Direct indexing.

Architecture 95

Architecture Building Kafka Database-centric 95

Cloudera

SEPTEMBER 20, 2021

There are many ways that Apache Kafka has been deployed in the field. In our Kafka Summit 2021 presentation, we took a brief overview of many different configurations that have been observed to date. In this blog series, we will discuss each of these deployments and the deployment choices made along with how they impact reliability.

Kafka 115

Kafka Systems Utilities Bytes 115

Confluent

MAY 28, 2019

Part 1 of this blog series by Gwen Shapira explained the benefits of schemas, contracts between services, and compatibility checking for schema evolution. There are three common types of operational mistakes we have observed customers make in self-managing their own Schema Registry: Bad designs. Inconsistent configurations.

Management 83

Management Kafka Java Certification 83

Confluent

FEBRUARY 27, 2019

What is more, as the world adopts the event-driven streaming architecture, how does it fit with serverless? When it comes to the emerging serverless world, It makes sense to validate how Apache Kafka ® fits in considering that it is mission critical in 90 percent of companies. Event-first FaaS? What is FaaS? Next Steps.

Kafka 109

Kafka AWS Architecture Cloud 109

Confluent

FEBRUARY 13, 2019

Part 1 of this series discussed why you need to embrace event-first thinking, while this article builds a rationale for different styles of event-driven architectures and compares and contrasts scaling, persistence and runtime models. Event-driven architecture. Event-driven, reactive architecture. Being event first.

Architecture 89

Architecture Programming Kafka Database-centric 89