Cloudera

JANUARY 14, 2021

This blog is the final post of a 4-part series. You can read the first blog posts, here: 1. This retailer deployed Cloudera DataFlow to tap real-time streaming data from thousands of cold storage sensors across its vast network of brick-and-mortar stores. Get to Know Your Retail Customer: 2.

Food 69

Food Retail Aggregated Data Machine Learning 69

Cloudera

FEBRUARY 9, 2023

The sudden failing of a complex data pipeline can lead to devastating consequences — especially if it goes unnoticed. This is why we build job notifications functionality into SSB, to deliver maximum reliability in your complex real-time data pipelines.

SQL 78

SQL Kafka Aggregated Data Architecture 78

DareData

JULY 5, 2023

In this blog post, we aim to share practical insights and techniques based on our real-world experience in developing data lake infrastructures for our clients - let's start! The Data Lake acts as the central repository for aggregating data from diverse sources in its raw format.

Data Lake 52

Data Lake Building Raw Data ETL Tools 52

Edureka

FEBRUARY 6, 2023

Azure Data Engineers are in high demand due to the growth of cloud-based data solutions. In this article, we will examine the duties of an Azure Data Engineer as well as the typical pay in this industry. Conclusion So this was all about the salary, job description, and skills of an Azure Data Engineer.

Data Engineering 52

Data Engineering Data Engineer Engineering Amazon Web Services 52

Rockset

AUGUST 30, 2021

The latest Rockset release, SQL-based rollups, has made real-time analytics on streaming data a lot more affordable and accessible. Anyone who knows SQL, the lingua franca of analytics, can now rollup, transform, enrich and aggregate real-time data at massive scale. You can also optionally use WHERE clauses to filter out data.

SQL 52

SQL Kafka MongoDB MySQL 52

Lyft Engineering

JUNE 28, 2023

Our goal was to develop foundations that would enable the hundreds of ML developers at Lyft to efficiently develop new models and enhance existing models with streaming data. In this blog post, we will discuss what we built in support of that goal and some of the lessons we learned along the way.

Machine Learning 115

Machine Learning Building Metadata Kafka 115

LinkedIn Engineering

DECEMBER 19, 2023

The version drift framework consolidates data from various sources, as shown in Figure 6, to create a comprehensive list of worker nodes currently running outdated versions. This framework operates on the scheduler, periodically polls relevant metrics, aggregates data, and determines which nodes have drifted.

Big Data 92

Big Data Hadoop Metadata Data 92

Lyft Engineering

NOVEMBER 29, 2023

In this particular blog post, we explain how Druid has been used at Lyft and what led us to adopt ClickHouse for our sub-second analytic system. Druid at Lyft Apache Druid is an in-memory, columnar, distributed, open-source data store designed for sub-second queries on real-time and historical data.

Kafka 104

Kafka Data Ingestion Datasets Architecture 104

Rockset

DECEMBER 17, 2020

Together, they empower developers to build performant internal tools, such as customer 360 and logistics monitoring apps, by solely using data APIs and pre-built UI components. In this blog, we’ll be building a customer 360 app using Rockset and Retool. From there, we’ll create a data API for the SQL query we write in Rockset.

Building 40

Building Aggregated Data SQL Data Ingestion 40

DoorDash Engineering

OCTOBER 17, 2023

Supported internally at DoorDash, Flink is used by many teams to run their processing jobs on streaming data. We use Flink’s built-in time-window-based aggregation functions on exposure time. We then send this aggregated data to another Kafka topic. We wanted an analytics OLAP datastore with low latency.

Education 87

Education Kafka Algorithm Data Warehouse 87

Confluent

FEBRUARY 6, 2019

The blog posts How to Build and Deploy Scalable Machine Learning in Production with Apache Kafka and Using Apache Kafka to Drive Cutting-Edge Machine Learning describe the benefits of leveraging the Apache Kafka ® ecosystem as a central, scalable and mission-critical nervous system. For now, we’ll focus on Kafka.

Machine Learning 107

Machine Learning Python Kafka Java 107

Rockset

AUGUST 12, 2019

Conclusion We’ve covered several approaches for building real-time analytics on DynamoDB data , each with its own pros and cons. Your DynamoDB table for tracking donations might look like In this scenario, it would be reasonable to track the donations per platform and the total donated so far.

NoSQL 40

NoSQL AWS SQL Datasets 40

Rockset

DECEMBER 22, 2020

There are many blog posts detailing how to build an Express API, I’ll concentrate on what is required on top of this to make calls to Elasticsearch. To experience how Rockset provides full-featured SQL queries on complex, semi-structured data, you can get started with a free Rockset account.

SQL 40

SQL Data MongoDB Aggregated Data 40

Snowflake

JUNE 22, 2023

Types of data products In a previous blog post, I discussed different forms a data product can take using the sand, glass, or lamp metaphor , depending on the consumer of the data and the use case. Some data products will be components in a more complex analysis or context-specific business application.

Building 52

Building Business Analyst Aggregated Data Data 52

ProjectPro

DECEMBER 7, 2021

Data pipelines are a significant part of the big data domain, and every professional working or willing to work in this field must have extensive knowledge of them. Before putting raw data into tables or views, DLT gives users access to the full power of SQL or Python.

Data Pipeline 52

Data Pipeline Architecture Kafka AWS 52

Cloudera

JUNE 29, 2023

Streamline KYC and AML, too While Know Your Customer (KYC) and Anti-Money-Laundering (AML) processes didn’t play a role in the recent collapses, institutions can also leverage the combination of a modern, open data architecture, advanced analytics, and machine automation to transform KYC and AML.

Data Architecture 82

Data Architecture Architecture Management Banking 82

Knowledge Hut

JANUARY 25, 2024

It doesn't matter if you're a data expert or just starting out; knowing how to clean your data is a must-have skill. The future is all about big data. This blog is here to help you understand not only the basics but also the cool new ways and tools to make your data squeaky clean.

Data Cleanse 52

Data Cleanse Datasets Data Preparation Data Science 52

LinkedIn Engineering

DECEMBER 20, 2023

In this blog post, we discuss how we are harnessing AI to help us with abuse prevention and share an overview of our infrastructure and the role it plays in identifying and mitigating abusive behavior on our platform. Standing still in the realm of abuse prevention is synonymous with regression.

Building 91

Building Algorithm Kafka Machine Learning 91

Knowledge Hut

OCTOBER 13, 2023

Aggregate Data: If you don't need granularity, consider aggregating data before loading it into Power BI to reduce the volume of data. Sort and Filter Early: Apply sorting and filtering in your queries as early as possible to reduce the amount of data transferred and processed.

BI 52

BI Business Analyst Datasets Raw Data 52

Netflix Tech

NOVEMBER 20, 2023

In this blog post, we talk about the landscape and the challenges in workflows at Netflix. Downstream workflows (if there is no business logic change) will be triggered by the data change due to backfill. This enables auto propagation of backfill data in multi-stage pipelines.

Process 84

Process Data Pipeline Datasets SQL 84

Databand.ai

JULY 10, 2023

Faster issue diagnosis: Aggregating data from multiple sources enables engineers to correlate events more easily when troubleshooting problems, allowing them to resolve issues more quickly and prevent future occurrences through proactive measures such as capacity planning or automated remediation actions based on observed trends.

Data Pipeline 52

Data Pipeline Algorithm Raw Data Aggregated Data 52

Scott Logic

JULY 26, 2023

My colleague Oliver Cronk has set out some of the principal risks in this blog post. In this blog post, I’ll provide an overview of how it worked. They can’t ask the bot to expose data on other customers or anything else outside of the Knowledge Graph’s purview. So, how do you mitigate the risks and harness the potential?

Banking 52

Banking Aggregated Data Retail Architecture 52

Snowflake

FEBRUARY 6, 2023

The latter create integrated, higher-value data products that are geared towards requirements of the data consumers on the business side; for example, a customer 360 domain aggregating data from multiple sources. Some teams are data producers but not data consumers. It’s not just the data teams.

Government 65

Government Data Data Pipeline Architecture 65

ProjectPro

JANUARY 24, 2023

One of the core features of ADF is the ability to preview your data while creating your data flows efficiently and to evaluate the outcome against a sample of data before completing and implementing your pipelines. Such features make Azure data flow a highly popular tool among data engineers.

Retail 52

Retail Big Data Data Pipeline Media 52

Rockset

FEBRUARY 16, 2024

In this blog, we’ll describe how Klarna implemented real-time anomaly detection at scale, halved the resolution time and saved millions of dollars using Rockset. Furthermore, Rockset’s ability to pre-aggregate data at ingestion time reduced the cost of storage and sped up queries, making the solution cost-effective at scale.

Architecture 59

Architecture SQL Data Warehouse Database 59

Lyft Engineering

SEPTEMBER 27, 2022

The second type of pipeline ingests Kafka topics and aggregates data into standard ML features. Relevant posts Check out our talk at Strata Data Conference 2019 ( Magic behind your Lyft ride prices ) to understand our thought process of choosing Apache Beam. It produces high-quality signals and publishes them to Kafka topics.

Kafka 52

Kafka Aggregated Data Machine Learning Architecture 52

Rockset

NOVEMBER 1, 2022

This blog outlines best practices from customers I have helped migrate from Elasticsearch to Rockset , reducing risk and avoiding common pitfalls. In this blog, we distilled their migration journeys into 5 steps. We often see ingest queries aggregate data by time.

Database-centric 52

Database-centric Pipeline-centric SQL Aggregated Data 52

Rockset

OCTOBER 26, 2022

One was to create another data pipeline that would aggregate data as it was ingested into DynamoDB. After finding Rockset through an AWS blog on creating leaderboards , we wasted no time in starting to build a new customer-facing leaderboard based on Rockset. Both would have required a lot of work.

SQL 52

SQL NoSQL Database Aggregated Data 52

AltexSoft

MAY 12, 2021

Source: AWS Machine Learning Blog. Fully connected (FC) layers aggregate data produced by previous layers to generate the final output (classification.). Owing to this method, it outperforms many neural network architectures (for example ResNet or DenseNet) in accuracy of image classification while processing data faster.

Medical 72

Medical Healthcare Datasets Machine Learning 72

U-Next

SEPTEMBER 7, 2022

As training data increases, deep learning requires scalability. Typically, data warehouses are set to read-only for analysts, who primarily read and aggregate data. It is not necessary to insert or update data since it is already clean and archival. . Conclusion . .

Data Lake 52

Data Lake Data Warehouse Unstructured Data Amazon Web Services 52

Rockset

FEBRUARY 25, 2021

With the many data sources in today’s modern architecture, this can be difficult. With Rockset, regardless of what format your data is in, your team can query it using SQL to easily parse complex data types. From there, you can join and aggregate data without using complex code.

SQL 40

SQL Data Pipeline Kafka Database 40

Cloudera

NOVEMBER 8, 2017

Building a full customer 360 requires aggregating data sets into a single view. You can also read about Cloudera Data Science and Engineering here. The post Machine Learning, the DOCOMO Digital way: Two Core Use Cases appeared first on Cloudera Blog. Driving customer insights with machine learning.

Machine Learning 40

Machine Learning Aggregated Data Algorithm Data Science 40

AltexSoft

JANUARY 26, 2022

The credit for coining the new role goes to Michael Kaminsky , a former Director of Analytics at Harry’s Grooming and a founder of Recast, who wrote the article about analytics engineering on the Locally Optimistic blog in 2019. This way of thinking was accepted by the dbt community that contributed to the popularity of the term.

Engineering 59

Engineering Software Engineer Software Engineering Data Warehouse 59

ProjectPro

JANUARY 25, 2022

Here’s What You Need to Know About PySpark This blog will take you through the basics of PySpark, the PySpark architecture, and a few popular PySpark libraries , among other things. Finally, you'll find a list of PySpark projects to help you gain hands-on experience and land an ideal job in Data Science or Big Data.

Big Data 52

Big Data Data Process Process Kafka 52

Cloudera

NOVEMBER 13, 2020

This is part of our series of blog posts on recent enhancements to Impala. Apache Impala is synonymous with high-performance processing of extremely large datasets, but what if our data isn’t huge? It turns out that Apache Impala scales down with data just as well as it scales up. The entire collection is available here.

Metadata 142

Metadata Coding SQL Database 142

Rockset

JULY 29, 2020

The main strategy to alleviate some of the pressure on the primary database is to offload some of the work to a secondary data store, and I will share some of the common patterns of this strategy in this blog series. In future articles I will discuss offloading to other types of systems.

MongoDB 40

MongoDB Database Aggregated Data Accessible 40

Picnic Engineering

SEPTEMBER 8, 2022

Some of our services send events containing aggregated data which are quite bulky, so we increase the max request size to 8MB (max value for standard Confluent Cloud clusters) and enable events compression. To make compression fruitful, we also add lingering and increase batch size to have a higher data compression rate.

Kafka 52

Kafka Metadata AWS Java 52