Pinterest Engineering

AUGUST 18, 2022

Ankita Girish Wagh | Senior Software Engineer, Storage and Caching Introduction and Motivation At Pinterest, HBase is one of the most critical storage backends, powering many online storage services like Zen (graph database), UMS (wide column datastore), and Ixia (near real time secondary indexing service). Stop writing to database A.

Data Ingestion 52

Data Ingestion Hadoop Database Kafka 52

Monte Carlo

JULY 7, 2022

Editor’s Note : We ran into Andrew at our London IMPACT event in early 2022. At the time, he was one of a very few people using the term “data contract.” You can read more about Convoy’s approach from our blog with their Head of Product, Data Platform, Chad Sanderson, “ The modern data warehouse is broken.”

Data Warehouse 52

Data Warehouse Software Engineer Software Engineering Data 52

Workfall

JUNE 27, 2023

Reading Time: 7 minutes The world of creating real-time applications can get complex when we have to consider latency, fault tolerance, scalability, and possible data losses. Traditionally, web sockets were the go-to option when it came to real-time applications, but think of a situation whereby there’s server downtime.

Kafka 75

Kafka Architecture AWS Transportation 75

DoorDash Engineering

DECEMBER 5, 2023

DoorDash’s Engineering teams revamped Kafka Topic creation by replacing a Terraform/Atlantis based approach with an in-house API, Infra Service. This has reduced real-time pipeline onboarding time by 95% and saved countless developer hours. Pulumi instead leverages existing programming languages to manage infrastructure.

Kafka 91

Kafka Programming Language Metadata Architecture 91

LinkedIn Engineering

JANUARY 16, 2024

In a previous blog post, we talked about how we built our anti-abuse platform using CASAL. In this blog post, we'll go deeper into how we manage account restrictions. We'll also share how we manage account restrictions at scale while maintaining our policies and improving the member experience.

Kafka 84

Kafka Relational Database Java Architecture 84

LinkedIn Engineering

DECEMBER 20, 2023

In this blog post, we discuss how we are harnessing AI to help us with abuse prevention and share an overview of our infrastructure and the role it plays in identifying and mitigating abusive behavior on our platform. At LinkedIn, trust is the cornerstone for building meaningful connections and professional relationships.

Building 91

Building Algorithm Kafka Machine Learning 91

Confluent

MAY 21, 2019

Imagine that next week I want to roll out a new risk evaluation service that depends on the user address—do I really want to ask the profile service to add some logic to call the new risk service as well? This leads us to event streaming microservices patterns. Chattanooga, TN” }. What will the profile service publish?

Kafka 110

Kafka Insurance Architecture Database 110

Cloudera

MAY 4, 2021

Event-driven and streaming architectures enable complex processing on market events as they happen, making them a natural fit for financial market applications. Flink SQL is a data processing language that enables rapid prototyping and development of event-driven and streaming applications. Streaming VWAP.

SQL 78

SQL Business Analyst Data Java 78

Cloudera

OCTOBER 26, 2020

Our continued success with Cloudera provides customers with the data they need to maximize the efficiency of their on-premise and hybrid analytic workloads,” said John Shirly, Vice President Unstructured Storage Product Management at Dell EMC. “Our Cluster capability using real-world workloads and micro-benchmarks.

Certification 86

Certification Cloud Kafka Unstructured Data 86

Scott Logic

APRIL 22, 2024

In my previous blog post I looked at various technologies for sending data asynchronously between services including RabbitMQ, Kafka, AWS EventBridge. This time round I’ll look at the messages themselves which over the last few years I’ve found to be a more complex and nuanced topic than expected.

AWS 52

AWS Metadata Systems Kafka 52

Cloudera

NOVEMBER 18, 2020

Streamed to hundreds of people around the globe, we were able to come together to celebrate some incredible successes. . In a year marked by unusual events, and disruption to our “normal” lives, it was a pleasure to recognize our customers’ most impressive achievements. Data Impact Achievement Award. Data Champions . Winner: OVO.

Medical 121

Medical Banking Telecommunication Government 121

Snowflake

MARCH 30, 2023

A 2016 data science report from data enrichment platform CrowdFlower found that data scientists spend around 80% of their time in data preparation (collecting, cleaning, and organizing of data) before they can even begin to build machine learning (ML) models to deliver business value. ML workflow, ubr.to/3EJHjvm

Engineering 83

Engineering Raw Data Data Science Scala 83

Monte Carlo

SEPTEMBER 6, 2022

In fact, we’ve written extensively about emerging data quality best practices like data SLAs , circuit breakers , immutable/semantic warehouses , schema change management , data asset certification , and moving toward a data mesh. In the wise words of our Site Reliability Engineering forebearers: “ Embrace risk.” Errant schema change?

Software Engineer 52

Software Engineer Software Engineering Retail Data 52

Rockset

DECEMBER 19, 2023

From his early days at Quora to leading projects at Facebook and his current venture at Fennel (a real-time feature store for ML), Nikhil has traversed the evolving landscape of machine learning engineering and machine learning infrastructure specifically in the context of recommendation systems.

Systems 52

Systems Machine Learning Deep Learning Media 52

Rockset

NOVEMBER 1, 2022

It’s an understandable position given the risk and horror stories associated with “Migration Projects”. This blog outlines best practices from customers I have helped migrate from Elasticsearch to Rockset , reducing risk and avoiding common pitfalls. In this blog, we distilled their migration journeys into 5 steps.

Database-centric 52

Database-centric Pipeline-centric SQL Aggregated Data 52

ProjectPro

JUNE 28, 2016

Apache Kafka is breaking barriers and eliminating the slow batch processing method that is used by Hadoop. This is just one of the reasons why Apache Kafka was developed in LinkedIn. Kafka was mainly developed to make working with Hadoop easier. Apache Kafka attempts to solve this issue.

Kafka 40

Kafka Systems Hadoop BI 40

phData: Data Engineering

JANUARY 3, 2022

Source Control Management Infrastructure as Code Build/Deploy Strategy Continuous Integration and Delivery (CI/CD) Data Quality and Validation Workflow Management Data Modeling Monitoring and Logging Business Continuity So How Do I Build a DataOps Strategy? Want to Save This eBook for Later? No problem! What’s Data Strategy?

IT 52

IT AWS Software Engineer Software Engineering 52

Confluent

SEPTEMBER 13, 2019

Static Membership is an enhancement to the current rebalance protocol that aims to reduce the downtime caused by excessive and unnecessary rebalances for general Apache Kafka ® client implementations. This applies to Kafka consumers, Kafka Connect, and Kafka Streams. Take a Kafka consumer, for example.

Kafka 21

Kafka Cloud Metadata Process 21

Confluent

OCTOBER 10, 2019

A key challenge, however, is integrating devices and machines to process the data in real time and at scale. Apache Kafka ® and its surrounding ecosystem, which includes Kafka Connect, Kafka Streams, and KSQL, have become the technology of choice for integrating and processing these kinds of datasets.

Kafka 20

Kafka Google Cloud Architecture Machine Learning 20

ProjectPro

MAY 31, 2021

This blog lists over 20 big data projects you can work on to showcase your big data skills and gain hands-on experience in big data tools and technologies. Ace your big data interview by adding some unique and exciting Big Data projects to your portfolio. Table of Contents What is a Big Data Project? How do you Create a Good Big Data Project?

Big Data 52

Big Data Coding Project Hadoop 52

Snowflake

APRIL 20, 2023

Data Pipelines Snowpipe Streaming – public preview While data generated in real time is valuable, it is more valuable when paired with historical data that helps provide context. Read our blog to learn more. Learn more about what these new functions are and how they work in our recent blog post.

Medical 57

Medical Retail Python Pharmaceutical 57

Rockset

APRIL 15, 2022

This is the second post in a series by Rockset's CTO Dhruba Borthakur on Designing the Next Generation of Data Systems for Real-Time Analytics. We'll be publishing more posts in the series in the near future, so subscribe to our blog so you don't miss them!

Analytics Application 52

Analytics Application Data Warehouse Raw Data Kafka 52

Cloudera

NOVEMBER 27, 2023

Flink Forward is a great opportunity to learn about the cutting edge of streaming and stream processing technologies. billion events/s. Many critical downstream applications consume data processed by Flink, especially telcos, financial services, and e-commerce, where real-time processing needs are pronounced.

Kafka 81

Kafka SQL ETL Tools Data Lake 81