Data Engineering Weekly

FEBRUARY 4, 2024

[link] [link] Grab: Rethinking Stream Processing - Data Exploration Grab writes an excellent blog about data exploration on stream processing. The solution centered around Notebook opens a Flink Session for the Kafka stream and continues the exploration. If you have a data quality problem, success like this can seem out of reach.

Data Engineering 62

Data Engineering Data Engineer Engineering PostgreSQL 62

Cloudera

JULY 21, 2022

Cloudera has partnered with Rill Data, an expert in metrics at any scale, as Cloudera’s preferred ISV partner to provide technical expertise and support services for Apache Druid customers. We want Cloudera customers that rely on Apache Druid to know that their clusters are secure and supported by the Cloudera partner ecosystem.

BI 86

BI Digital Media Data Warehouse Kafka 86

Cloudera

NOVEMBER 16, 2021

With the general availability of Cloudera DataFlow for the Public Cloud (CDF-PC) , our customers can now self-serve deployments of Apache NiFi data flows on Kubernetes clusters in a cost effective way providing auto scaling, resource isolation and monitoring with KPI-based alerting. Functions as a Service.

Google Cloud 108

Google Cloud AWS Cloud Computing Kafka 108

Cloudera

JULY 15, 2021

The release of Cloudera Data Platform (CDP) Private Cloud Base edition provides customers with a next generation hybrid cloud architecture. The recently released Cloudera Ansible playbooks provide the templates that incorporate best practices described in this blog post and can be downloaded from [link] . Edge or Gateway.

Architecture 101

Architecture Cloud Kafka Hadoop 101

Netflix Tech

JANUARY 13, 2022

This blog will explore these two systems and how they perform auto-diagnosis and remediation across our Big Data Platform and Real-time infrastructure. These steps launch jobs on clusters running Apache Spark and Presto via Genie. And we can’t discount the productivity impact it causes on data platform users.

Kafka 95

Kafka Big Data Data Machine Learning 95

Cloudera

SEPTEMBER 28, 2020

Cloudera has found that customers have spent many years investing in their big data assets and want to continue to build on that investment by moving towards a more modern architecture that helps leverage the multiple form factors. Customer A is a financial services firm running CDH 5.14.2. New Features CDH to CDP. Query Result Cache.

Cloud 131

Cloud Kafka Professional Services Metadata 131

Picnic Engineering

FEBRUARY 8, 2023

Delivering the right events at low latency and with a high volume is critical to Picnic’s system architecture. In our previous blog, Dima Kalashnikov explained how we configure our Internal services pipeline in the Analytics Platform. There are some cases where the rollback could also fail.

Data Pipeline 105

Data Pipeline Kafka Data Architecture 105

Confluent

MARCH 26, 2019

Apache-Kafka ® -based applications stand out for their ability to decouple producers and consumers using an event log as an intermediate layer. This enables choreographed service collaborations, where many components can subscribe to events stored in the event log and react to them asynchronously. Let’s imagine a “Hello, World!”

Kafka 111

Kafka Transportation Metadata Consulting 111

Data Engineering Podcast

NOVEMBER 26, 2019

As they scaled the volume of customers and data they began running into the limitations of their initial architecture. Can you describe the system architecture of Snuba and some of the ways that it differs from your initial ideas of how it would work? And for your machine learning workloads, they just announced dedicated CPU instances.

Data Warehouse 100

Data Warehouse Building PostgreSQL Kafka 100

Cloudera

APRIL 20, 2021

Transportation: Monitor truck health and performance from smartphones and tablets, prioritize needed reports, and quickly identify the nearest dealer service locations. Retail Service: Using real-time analytics to help brands, marketers, and retailers understand and learn how customers behave while they shop in the store.

Hospitality 94

Hospitality Kafka Retail Data Ingestion 94

Data Engineering Podcast

JUNE 17, 2017

To achieve that goal they embarked on a journey to refactor their monolithic architecture to be more modular and modern, and then they open sourced it! Summary Yelp needs to be able to consume and process all of the user interactions that happen in their platform in as close to real-time as possible.

Data Pipeline 100

Data Pipeline Kafka Business Intelligence Architecture 100

Knowledge Hut

SEPTEMBER 6, 2023

Big Data Frameworks : Familiarity with popular Big Data frameworks such as Hadoop, Apache Spark, Apache Flink, or Kafka are the tools used for data processing. This influx of data is handled by robust big data systems which are capable of processing, storing, and querying data at scale. What is Big Data Certification?

Big Data 52

Big Data Certification Hadoop Scala 52

LinkedIn Engineering

DECEMBER 19, 2023

Co-authors: Arjun Mohnot , Jenchang Ho , Anthony Quigley , Xing Lin , Anil Alluri , Michael Kuchenbecker LinkedIn operates one of the world’s largest Apache Hadoop big data clusters. These SSH-based processes consumed resources, negatively impacting our server and service performance.

Big Data 92

Big Data Hadoop Metadata Data 92

Cloudera

NOVEMBER 10, 2021

This blog post provides CDH users with a quick overview of Ranger as a Sentry replacement for Hadoop SQL policies in CDP. Apache Sentry is a role-based authorization module for specific components in Hadoop. Furthermore, Apache Ranger now supports Public Cloud objects stores like Amazon S3 and Azure Data Lake Store (ADLS).

Hadoop 77

Hadoop SQL Database Kafka 77

ProjectPro

FEBRUARY 21, 2023

With around 35k stars and over 26k forks on Github, Apache Spark is one of the most popular big data frameworks used by 22,760 companies worldwide. Apache Spark is the most efficient, scalable, and widely used in-memory data computation tool capable of performing batch-mode, real-time, and analytics operations.

Scala 52

Scala Programming Language Java Hadoop 52

Rockset

JULY 28, 2022

The application could submit data via a micro-service or application server that talks directly to the database, but this only moves the problem. These services would still need to talk directly to the database. If this is not already part of your architecture then it may add another layer of complexity and cost.

MongoDB 52

MongoDB Kafka NoSQL Data Lake 52

Confluent

FEBRUARY 6, 2019

The blog posts How to Build and Deploy Scalable Machine Learning in Production with Apache Kafka and Using Apache Kafka to Drive Cutting-Edge Machine Learning describe the benefits of leveraging the Apache Kafka ® ecosystem as a central, scalable and mission-critical nervous system.

Machine Learning 107

Machine Learning Python Kafka Java 107

Snowflake

AUGUST 16, 2023

Read our Summit recap blog for highlights across industries or watch Summit sessions now on-demand. Data comes in a continuous manner, and often a separate architecture is required to handle streaming data. The new Kafka connector, built with Snowpipe Streaming , now supports schema detection and evolution. Learn more here.

Scala 52

Scala Transportation Kafka Data Lake 52

Knowledge Hut

APRIL 25, 2023

Data processing: Data engineers should know data processing frameworks like Apache Spark, Hadoop, or Kafka, which help process and analyze data at scale. Data integration: Data engineers should be able to integrate data from various sources like databases, APIs, or file systems, using tools like Apache NiFi, Fivetran, or Talend.

Data Engineering 52

Data Engineering Data Engineer Engineering Google Cloud 52

Confluent

APRIL 10, 2019

Enterprises run modern data systems and services across multiple cloud providers, private clouds and on-prem multi-datacenter deployments. Instead of having many point-to-point connections between sites, the Confluent Platform provides an integrated event streaming architecture with frictionless data replication between sites.

Kafka 86

Kafka Metadata Java Cloud 86

Data Engineering Weekly

MARCH 26, 2023

Future iterations of generative AI, public services such as ChatGPT, and domain-specific versions of these underlying models will make IT, and computing to date look like the spinning jenny that was the start of the industrial revolution. Now you can win $1,000 cash by contributing a Transformation to our open-source library.

Data Engineering 52

Data Engineering Data Engineer Engineering Lambda Architecture 52

Cloudera

JANUARY 19, 2022

We have been working hard to build our cloud-native data services on Cloudera Data Platform (CDP), which include CDP Data Warehouse, CDP Operational Database, CDP Machine Learning, CDP Data Engineering and CDP Data Flow. Gartner evaluated 20 vendors in the Magic Quadrant for Cloud DBMS and the associated Critical Capabilities report.

Database 89

Database Cloud Data Warehouse Data Lake 89

ProjectPro

DECEMBER 7, 2021

This blog will give you an in-depth knowledge of what is a data pipeline and also explore other aspects such as data pipeline architecture, data pipeline tools, use cases, and so much more. Features of a Data Pipeline Data Pipeline Architecture How to Build an End-to-End Data Pipeline from Scratch? What is a Big Data Pipeline?

Data Pipeline 52

Data Pipeline Architecture Kafka AWS 52

AltexSoft

FEBRUARY 11, 2023

To get a better understanding of a data architect’s role, let’s clear up what data architecture is. Data architecture is the organization and design of how data is collected, transformed, integrated, stored, and used by a company. Data is now one of the most valuable assets for any kind of business. What is a data architect?

Data Architect 87

Data Architect Certification Generalist Big Data 87

Big Data Tools

AUGUST 3, 2021

Kotlin API for Apache Spark – A year after showing you the first preview, we released version 1.0. Apache Spark already has two official APIs for JVM – Scala and Java – but we’re hoping the Kotlin API will be useful as well, as we’ve introduced several unique features. Here’s what’s happening in data engineering right now. Cassandra 4.0

Data Engineering 52

Data Engineering Data Engineer Engineering Big Data Tools 52

Big Data Tools

AUGUST 3, 2021

Kotlin API for Apache Spark – A year after showing you the first preview, we released version 1.0. Apache Spark already has two official APIs for JVM – Scala and Java – but we’re hoping the Kotlin API will be useful as well, as we’ve introduced several unique features. Here’s what’s happening in data engineering right now. Cassandra 4.0

Data Engineering 52

Data Engineering Data Engineer Engineering Big Data Tools 52

Data Engineering Weekly

NOVEMBER 27, 2022

I have a long list of thoughts on this conversation, which might need a blog post on its own. link] Airbnb: How AI Text Generation Models Are Reshaping Customer Support at Airbnb Airbnb takes customer service from a simple customer service response template to the AI text generation model.

Data Engineering 52

Data Engineering Data Engineer Engineering SQL 52

Cloudera

AUGUST 26, 2020

To find out more about how live-streaming data might impact the financial services sector, I sat down for a chat with Dinesh Chandrasekhar, Head of Product Marketing in Cloudera’s data-in-motion Business Unit. Can you talk to us a bit about the benefits of NiFi for financial services businesses? How does that happen in near real-time?

Banking 83

Banking Data Ingestion Kafka Data Lake 83

Confluent

APRIL 18, 2019

And Confluent Cloud and Apache Kafka ® are at the center of this transformation. Kafka is business critical for us. So working with Confluent Cloud and the Kafka experts was an obvious choice,” says Stefan Frehse, Software Architect, Digitalization, for food processing machinery company BAADER.

Food 83

Food Cloud Retail Kafka 83

Big Data Tools

OCTOBER 10, 2022

This time I learned about Brooklin, a LinkedIn service for streaming data in a heterogeneous environment. One of the use cases from the product page that stood out to me in particular was the effort to mirror multiple Kafka clusters in one Brooklin cluster! It’s been a very bustling two months in Berlin. Greetings from sunny Berlin!

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Big Data Tools

OCTOBER 10, 2022

This time I learned about Brooklin, a LinkedIn service for streaming data in a heterogeneous environment. One of the use cases from the product page that stood out to me in particular was the effort to mirror multiple Kafka clusters in one Brooklin cluster! It’s been a very bustling two months in Berlin. Greetings from sunny Berlin!

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Cloudera

AUGUST 21, 2020

In a financial services context, this could be trades or transactional data. A good example of this is fraud detection, credit card fraud is a big problem in the financial services industry which can mean a significant financial loss. Hi Dinesh, thank you for joining us for today’s Q&A.

Banking 61

Banking Kafka Cloud Storage Government 61

Netflix Tech

OCTOBER 15, 2019

Beyond data synchronization, some applications also need to enrich their data by calling external services. providing advanced search capabilities (ElasticSearch etc.), caching (Memcached etc.), However, these approaches have limitations in regards to feasibility, robustness, and maintenance. To address these challenges, we developed Delta.

Transportation 77

Transportation MySQL Kafka Data 77

Databand.ai

DECEMBER 13, 2022

Currently, Charles works at PitchBook Data and he holds degrees in Algorithms, Network, Computer Architecture, and Python Programming from Bradfield School of Computer Science and Bellevue College Continuing Education. Deepak regularly shares blog content and similar advice on LinkedIn. You can also watch the video recording.

Data Engineering 52

Data Engineering Data Engineer Engineering AWS 52

Lyft Engineering

OCTOBER 3, 2023

Finally, as the subject of this blog post, we can assess data quality via batch compute analytics on our data warehouse, providing a comprehensive albeit slower evaluation compared to the previously mentioned methods. These flow through Kafka , our event streaming platform, before being processed by Flink , our streaming compute framework.

Big Data 84

Big Data Metadata Data Warehouse Data 84

Rockset

JANUARY 23, 2020

Despite their different design choices for particular protocols, these databases have adopted, for the most part, a shared-nothing, distributed computing architecture. Each node represents a unit of compute and storage that can be added to the system as needed. This is accomplished by treating every write as an append operation.

Database 40

Database MySQL Kafka NoSQL 40

Rockset

JULY 8, 2020

Recently we've seen a big increase in the use of on-demand logistics services, such as online shopping and food delivery. Rockset - The query engine powering all the models and services. For this example, we will use Kafka. You can run a background service in the app that retrieves the GPS coordinates every 10 seconds.

Kafka 40

Kafka Designing Systems Food 40