Cloudera

JANUARY 14, 2021

This blog is the final post of a 4-part series. You can read the first blog posts, here: 1. This retailer deployed Cloudera DataFlow to tap real-time streaming data from thousands of cold storage sensors across its vast network of brick-and-mortar stores. Get to Know Your Retail Customer: 2.

Food 70

Food Retail Aggregated Data Machine Learning 70

Rockset

FEBRUARY 16, 2024

In this blog, we’ll describe how Klarna implemented real-time anomaly detection at scale, halved the resolution time and saved millions of dollars using Rockset. They also wanted the capability to add new metrics, data points and run ad-hoc analysis as they continued to build out real-time monitoring.

Architecture 59

Architecture SQL Data Warehouse Database 59

ProjectPro

JANUARY 24, 2023

With over 80 in-built connectors and data sources, 90 in-built transformations, and the ability to process 2GB of data per hour, Azure data factory dataflows have become the de facto choice for organizations to integrate and transform data from various sources at scale.

Retail 52

Retail Big Data Data Pipeline Media 52

Edureka

FEBRUARY 6, 2023

In today’s technological environment, where data and cloud computing are becoming more and more significant, an Azure Data Engineer is extremely important. Azure Data Engineers are in high demand due to the growth of cloud-based data solutions. Capability to communicate effectively and clearly.

Data Engineering 52

Data Engineering Data Engineer Engineering Amazon Web Services 52

Rockset

NOVEMBER 1, 2022

This blog outlines best practices from customers I have helped migrate from Elasticsearch to Rockset , reducing risk and avoiding common pitfalls. These companies migrated to Rockset in days or weeks, not months or years leveraging the power and simplicity of a cloud-native database. We often see ingest queries aggregate data by time.

Database-centric 52

Database-centric Pipeline-centric SQL Aggregated Data 52

Rockset

APRIL 15, 2022

This is the second post in a series by Rockset's CTO Dhruba Borthakur on Designing the Next Generation of Data Systems for Real-Time Analytics. We'll be publishing more posts in the series in the near future, so subscribe to our blog so you don't miss them! All updates are appended rather than written over existing data records.

Analytics Application 52

Analytics Application Data Warehouse Raw Data Kafka 52

DareData

JULY 5, 2023

In this blog post, we aim to share practical insights and techniques based on our real-world experience in developing data lake infrastructures for our clients - let's start! The Data Lake acts as the central repository for aggregating data from diverse sources in its raw format.

Data Lake 52

Data Lake Building Raw Data ETL Tools 52

Rockset

AUGUST 30, 2021

It becomes prohibitively complex and expensive to use a data warehouse to serve real-time analytics. Rockset: Real-time Analytics Built for the Cloud Rockset is doing for real-time analytics what Snowflake did for batch. You can also optionally use WHERE clauses to filter out data.

SQL 52

SQL Kafka MongoDB MySQL 52

U-Next

SEPTEMBER 7, 2022

Data lakes, however, are sometimes used as cheap storage with the expectation that they are used for analytics. For building data lakes, the following technologies provide flexible and scalable data lake storage : . Gen 2 Azure Data Lake Storage . Cloud storage provided by Google . Amazon Web Services S3 .

Data Lake 52

Data Lake Data Warehouse Unstructured Data Amazon Web Services 52

ProjectPro

FEBRUARY 8, 2023

Do ETL and data integration activities seem complex to you? Read this blog to understand everything about AWS Glue that makes it one of the most popular data integration solutions in the industry. Did you know the global big data market will likely reach $268.4 Businesses are leveraging big data now more than ever.

AWS 98

AWS Scala Metadata Data Lake 98

Confluent

FEBRUARY 6, 2019

The blog posts How to Build and Deploy Scalable Machine Learning in Production with Apache Kafka and Using Apache Kafka to Drive Cutting-Edge Machine Learning describe the benefits of leveraging the Apache Kafka ® ecosystem as a central, scalable and mission-critical nervous system. For now, we’ll focus on Kafka.

Machine Learning 106

Machine Learning Python Kafka Java 106

Picnic Engineering

SEPTEMBER 8, 2022

In the “Picnic Analytics Platform: Migration from AWS Kinesis to Confluent Cloud” we described why and how we migrated from AWS Kinesis to Confluent Cloud. The data is loaded into Snowflake, Picnic’s single source of truth Data Warehouse (DWH). Our pipeline captures these events and sends them to Confluent Cloud.

Kafka 52

Kafka Metadata AWS Java 52

Rockset

OCTOBER 26, 2022

As our database of record, DynamoDB was great at ingesting external data, which we stored within a single table in DynamoDB. One was to create another data pipeline that would aggregate data as it was ingested into DynamoDB. This would require creating a new table, which would’ve required a few extra weeks of dev time.

SQL 52

SQL NoSQL Database Aggregated Data 52

AltexSoft

OCTOBER 21, 2022

A single cluster can span across multiple data centers and cloud facilities. This allows for easy horizontal scaling — just add new servers or data centers to your existing infrastructure to handle more amount of data. This enables systems using Kafka to aggregate data from many sources and to make it consistent.

Kafka 93

Kafka Hadoop ETL Tools Big Data 93

AltexSoft

JANUARY 26, 2022

The development of new tools and the improvement of old ones has led to the transformation of the traditional data roles, simplifying some job aspects and complicating other ones. Companies start hiring that one data guy or gal that wears all the hats — knows the complete data stack from both the engineering and analytics sides.

Engineering 59

Engineering Software Engineer Software Engineering Data Warehouse 59

DoorDash Engineering

APRIL 12, 2023

Challenges of ad-hoc SQLs Our initial goal with Curie was to standardize the analysis methodologies and simplify the experiment analysis process for data scientists. Acknowledgements Special thanks to Sagar Akella, Bhawana Goel, and Ezra Berger for their invaluable help in reviewing this blog article.

SQL 82

SQL Metadata Raw Data Government 82

Cloudera

NOVEMBER 9, 2018

Since data fuels the growth of smart cities, it is crucial for governments to invest in data management and data security platforms, advanced analytics, and machine learning. Cost-effectively ingest, store and utilize data from all IoT devices. Analyze all data – be it at the edge, on cloud, on-prem or in a hybrid mode.

Machine Learning 56

Machine Learning Banking Government Media 56

ProjectPro

AUGUST 24, 2021

Data professionals who work with raw data like data engineers, data analysts, machine learning scientists , and machine learning engineers also play a crucial role in any data science project. And, out of these professions, this blog will discuss the data engineering job role.

Data Engineering 52

Data Engineering Data Engineer Coding Project 52

Netflix Tech

APRIL 26, 2022

Fig 2: Internal components of Axion Axion’s fact logging client logs facts to the keystone real-time stream processing platform , which outputs data to an Iceberg table. We use Keystone as it is easy to use, reliable, scalable, and provides aggregation of facts from different cloud regions into a single AWS region.

Metadata 75

Metadata Datasets Machine Learning AWS 75

ProjectPro

DECEMBER 7, 2021

Data pipelines are a significant part of the big data domain, and every professional working or willing to work in this field must have extensive knowledge of them. Consequently, data engineers implement checkpoints so that no event is missed or processed twice. It not only consumes more memory but also slackens data transfer.

Data Pipeline 52

Data Pipeline Architecture Kafka AWS 52

ProjectPro

NOVEMBER 15, 2021

As per the surveyors, Big data (35 percent), Cloud computing (39 percent), operating systems (33 percent), and the Internet of Things (31 percent) are all expected to be impacted by open source shortly. Following these statistics, big data is set to get bigger with the evolution of open-source projects.

Big Data 52

Big Data Project Metadata Programming Language 52

Rockset

AUGUST 12, 2019

Rockset supports SQL over REST, which means any http framework in any programming language can be used to query your data, and several client libraries are provided for convenience as well. Conclusion We’ve covered several approaches for building real-time analytics on DynamoDB data , each with its own pros and cons.

NoSQL 40

NoSQL AWS SQL Datasets 40

ProjectPro

JUNE 28, 2016

This data can be anything from clickstream data, activity/ web logs, consumer data, etc. Apache Kafka captures all this data and makes it available to enterprise users in real time. This blog post will explore why Apache Kafka was developed, what does it do and what makes Kafka so popular with Big Data analysis.

Kafka 40

Kafka Systems Hadoop BI 40

ProjectPro

FEBRUARY 22, 2022

SQL Projects For Data Analysis Hoping the example above has fueled you with the zeal to enhance your programming skills in SQL , we present you with an exciting list of SQL projects for practice. You can use these SQL projects for data analysis and add them to your data analyst portfolio.

Data Analysis 52

Data Analysis SQL Project Banking 52

ProjectPro

JULY 27, 2021

This blog is your one-stop solution for the top 100+ Data Engineer Interview Questions and Answers. In this blog, we have collated the frequently asked data engineer interview questions based on tools and technologies that are highly useful for a data engineer in the Big Data industry.

Data Engineering 40

Data Engineering Data Engineer Engineering Hadoop 40