Rockset

AUGUST 30, 2021

The latest Rockset release, SQL-based rollups, has made real-time analytics on streaming data a lot more affordable and accessible. Anyone who knows SQL, the lingua franca of analytics, can now rollup, transform, enrich and aggregate real-time data at massive scale. You can also optionally use WHERE clauses to filter out data.

SQL 52

SQL Kafka MongoDB MySQL 52

Rockset

DECEMBER 17, 2020

Together, they empower developers to build performant internal tools, such as customer 360 and logistics monitoring apps, by solely using data APIs and pre-built UI components. In this blog, we’ll be building a customer 360 app using Rockset and Retool. From there, we’ll create a data API for the SQL query we write in Rockset.

Building 40

Building Aggregated Data SQL Data Ingestion 40

Confluent

FEBRUARY 6, 2019

The blog posts How to Build and Deploy Scalable Machine Learning in Production with Apache Kafka and Using Apache Kafka to Drive Cutting-Edge Machine Learning describe the benefits of leveraging the Apache Kafka ® ecosystem as a central, scalable and mission-critical nervous system. For now, we’ll focus on Kafka.

Machine Learning 106

Machine Learning Python Kafka Java 106

Rockset

AUGUST 12, 2019

As an example, let’s say we are organizing a charity fundraiser and want a live dashboard at the event to show the progress towards our fundraising goal. Conclusion We’ve covered several approaches for building real-time analytics on DynamoDB data , each with its own pros and cons.

NoSQL 40

NoSQL AWS SQL Datasets 40

ProjectPro

DECEMBER 7, 2021

Data pipelines are a significant part of the big data domain, and every professional working or willing to work in this field must have extensive knowledge of them. Data pipelines must be scalable due to the volume of big data, which might fluctuate over time.

Data Pipeline 52

Data Pipeline Architecture Kafka AWS 52

Cloudera

JUNE 29, 2023

Incorporate data from novel sources — social media feeds, alternative credit histories (utility and rental payments), geo-spatial systems, and IoT streams — into liquidity risk models. Apply predictive-analytic and ML techniques to this data to create more accurate profiles and proactively identify high-risk customers.

Data Architecture 83

Data Architecture Architecture Management Banking 83

LinkedIn Engineering

DECEMBER 20, 2023

In this blog post, we discuss how we are harnessing AI to help us with abuse prevention and share an overview of our infrastructure and the role it plays in identifying and mitigating abusive behavior on our platform. To achieve this, we leverage Kafka messages, a robust and scalable event streaming platform.

Building 91

Building Algorithm Kafka Machine Learning 91

Knowledge Hut

OCTOBER 13, 2023

Aggregate Data: If you don't need granularity, consider aggregating data before loading it into Power BI to reduce the volume of data. Sort and Filter Early: Apply sorting and filtering in your queries as early as possible to reduce the amount of data transferred and processed.

BI 52

BI Business Analyst Datasets Raw Data 52

Lyft Engineering

SEPTEMBER 27, 2022

The very first version (see Figure 1) was designed to consume events, convert data to ML features, orchestrate model executions, and sync decision variables to their respective services. This pipeline ingests tens of millions of events per second and processes them into machine learning features.

Kafka 52

Kafka Aggregated Data Machine Learning Architecture 52

Rockset

FEBRUARY 16, 2024

In this blog, we’ll describe how Klarna implemented real-time anomaly detection at scale, halved the resolution time and saved millions of dollars using Rockset. Furthermore, Rockset’s ability to pre-aggregate data at ingestion time reduced the cost of storage and sped up queries, making the solution cost-effective at scale.

Architecture 59

Architecture SQL Data Warehouse Database 59

Netflix Tech

NOVEMBER 20, 2023

In this blog post, we talk about the landscape and the challenges in workflows at Netflix. Downstream workflows (if there is no business logic change) will be triggered by the data change due to backfill. This enables auto propagation of backfill data in multi-stage pipelines.

Process 84

Process Data Pipeline Datasets SQL 84

Databand.ai

JULY 10, 2023

Faster issue diagnosis: Aggregating data from multiple sources enables engineers to correlate events more easily when troubleshooting problems, allowing them to resolve issues more quickly and prevent future occurrences through proactive measures such as capacity planning or automated remediation actions based on observed trends.

Data Pipeline 52

Data Pipeline Algorithm Raw Data Aggregated Data 52

Scott Logic

JULY 26, 2023

My colleague Oliver Cronk has set out some of the principal risks in this blog post. In this blog post, I’ll provide an overview of how it worked. They can’t ask the bot to expose data on other customers or anything else outside of the Knowledge Graph’s purview. So, how do you mitigate the risks and harness the potential?

Banking 52

Banking Aggregated Data Retail Architecture 52

Picnic Engineering

SEPTEMBER 8, 2022

Quick re-cap: the purpose of the internal pipeline is to deliver data from dozens of Picnic back-end services such as warehousing, machine learning models, customers and order status updates. The data is loaded into Snowflake, Picnic’s single source of truth Data Warehouse (DWH). Yet, some messages are destined for the DWH only.

Kafka 52

Kafka Metadata AWS Java 52

ProjectPro

JANUARY 25, 2022

Here’s What You Need to Know About PySpark This blog will take you through the basics of PySpark, the PySpark architecture, and a few popular PySpark libraries , among other things. Finally, you'll find a list of PySpark projects to help you gain hands-on experience and land an ideal job in Data Science or Big Data.

Big Data 52

Big Data Data Process Process Kafka 52

Rockset

NOVEMBER 1, 2022

This blog outlines best practices from customers I have helped migrate from Elasticsearch to Rockset , reducing risk and avoiding common pitfalls. In this blog, we distilled their migration journeys into 5 steps. Time Series You will often have events or records with a timestamp and want to search based on a range of time.

Database-centric 52

Database-centric Pipeline-centric SQL Aggregated Data 52

AltexSoft

OCTOBER 21, 2022

This scenario involves three main characters — publishers, subscribers, and a message or event broker. A publisher (say, telematics or Internet of Medical Things system) produces data units, also called events or messages , and directs them not to consumers but to a middleware platform — a broker. Kafka cluster and brokers.

Kafka 93

Kafka Hadoop ETL Tools Big Data 93

ProjectPro

FEBRUARY 8, 2023

Do ETL and data integration activities seem complex to you? Read this blog to understand everything about AWS Glue that makes it one of the most popular data integration solutions in the industry. Did you know the global big data market will likely reach $268.4 Businesses are leveraging big data now more than ever.

AWS 98

AWS Scala Metadata Data Lake 98

Cloudera

NOVEMBER 8, 2017

Event prediction. Building a full customer 360 requires aggregating data sets into a single view. You can also read about Cloudera Data Science and Engineering here. The post Machine Learning, the DOCOMO Digital way: Two Core Use Cases appeared first on Cloudera Blog. Pattern recognition. Anomaly detection.

Machine Learning 40

Machine Learning Aggregated Data Algorithm Data Science 40

ProjectPro

AUGUST 24, 2021

Data professionals who work with raw data like data engineers, data analysts, machine learning scientists , and machine learning engineers also play a crucial role in any data science project. And, out of these professions, this blog will discuss the data engineering job role.

Data Engineering 52

Data Engineering Data Engineer Coding Project 52

U-Next

SEPTEMBER 7, 2022

The terms “ Data Warehouse ” and “ Data Lake ” may have confused you, and you have some questions. In the event that they are not the same, what are the difference s? As training data increases, deep learning requires scalability. Are these two terms used to describe the same thing?

Data Lake 52

Data Lake Data Warehouse Unstructured Data Amazon Web Services 52

Rockset

FEBRUARY 25, 2021

Joins are often used in real-time analytics applications to combine streaming data (usually representing events) with static data (like customer information). With Elasticsearch, joins are not a first class citizen and many teams end up denormalizing their data to model relationships.

SQL 40

SQL Data Pipeline Kafka Database 40

Cloudera

NOVEMBER 13, 2020

This is part of our series of blog posts on recent enhancements to Impala. Apache Impala is synonymous with high-performance processing of extremely large datasets, but what if our data isn’t huge? It turns out that Apache Impala scales down with data just as well as it scales up. The entire collection is available here.

Metadata 143

Metadata Coding SQL Database 143

ProjectPro

FEBRUARY 22, 2022

SQL Projects For Data Analysis Hoping the example above has fueled you with the zeal to enhance your programming skills in SQL , we present you with an exciting list of SQL projects for practice. You can use these SQL projects for data analysis and add them to your data analyst portfolio.

Data Analysis 52

Data Analysis SQL Project Banking 52

ProjectPro

JUNE 28, 2016

This data can be anything from clickstream data, activity/ web logs, consumer data, etc. Apache Kafka captures all this data and makes it available to enterprise users in real time. This blog post will explore why Apache Kafka was developed, what does it do and what makes Kafka so popular with Big Data analysis.

Kafka 40

Kafka Systems Hadoop BI 40

ProjectPro

NOVEMBER 15, 2021

Table of Contents 20 Open Source Big Data Projects To Contribute How to Contribute to Open Source Big Data Projects? 20 Open Source Big Data Projects To Contribute There are thousands of open-source projects in action today. This blog will walk through the most popular and fascinating open source big data projects.

Big Data 52

Big Data Project Metadata Programming Language 52

ProjectPro

JULY 27, 2021

This blog is your one-stop solution for the top 100+ Data Engineer Interview Questions and Answers. In this blog, we have collated the frequently asked data engineer interview questions based on tools and technologies that are highly useful for a data engineer in the Big Data industry.

Data Engineering 40

Data Engineering Data Engineer Engineering Hadoop 40

DoorDash Engineering

APRIL 12, 2023

Challenges of ad-hoc SQLs Our initial goal with Curie was to standardize the analysis methodologies and simplify the experiment analysis process for data scientists. Data scientists are the primary metric creators and are already familiar with SQL, so it made sense to use SQL as the language to define metrics instead of building our own DSL.

SQL 82

SQL Metadata Raw Data Government 82

Rockset

APRIL 15, 2022

This is the second post in a series by Rockset's CTO Dhruba Borthakur on Designing the Next Generation of Data Systems for Real-Time Analytics. We'll be publishing more posts in the series in the near future, so subscribe to our blog so you don't miss them! Ever since there has been streaming data, there has been out-of-order data.

Analytics Application 52

Analytics Application Data Warehouse Raw Data Kafka 52

Airbnb Tech

APRIL 30, 2021

Minerva takes fact and dimension tables as inputs, performs data denormalization, and serves the aggregated data to downstream applications. For example, data scientists have built a time series analysis tool and an email reporting framework using this API over the last two years.

Data Warehouse 52

Data Warehouse Finance Metadata Aggregated Data 52