Data Science Blog: Data Engineering

NOVEMBER 15, 2023

New big data architectures and, above all, data sharing concepts such as Data Mesh are ideal for creating a common database for many data products and applications. The Event Log Data Model for Process Mining Process Mining as an analytical system can very well be imagined as an iceberg.

Architecture 52

Architecture Database-centric Process BI 52

Snowflake

FEBRUARY 7, 2023

Event data for tracking a user’s journey has always been important to product analytics—but we’re now seeing changes in how businesses work with and manage their data, including event data. The first generation of product analytics tooling was primarily about tracking and managing front-end events.

BI 78

BI Building ETL Tools Data Warehouse 78

Rockset

MARCH 10, 2022

This is the first post in a series by Rockset's CTO Dhruba Borthakur on Designing the Next Generation of Data Systems for Real-Time Analytics. A platform such as Apache Kafka/Confluent , Spark or Amazon Kinesis for publishing that stream of event data. Event streaming/stream processing has been around for almost a decade.

Data Analytics 52

Data Analytics Data Warehouse Medical MySQL 52

Rockset

FEBRUARY 24, 2023

Typically, the “units” of data being streamed are considered events, which resemble a record in a database, with some key differences. First, event data is unstructured or semi-structured and stored in a nested format like JSON or AVRO. Second, events are usually immutable (this will be a very important feature in this series!).

Kafka 52

Kafka AWS Amazon Web Services Programming Language 52

Cloudera

SEPTEMBER 15, 2022

July brings summer vacations, holiday gatherings, and for the first time in two years, the return of the Massachusetts Institute of Technology (MIT) Chief Data Officer symposium as an in-person event. A key area of focus for the symposium this year was the design and deployment of modern data platforms.

Data Lake 80

Data Lake Analytics Application Cloud Storage Architecture 80

Snowflake

MAY 4, 2023

This year, DTCC celebrates its 50th anniversary, and over the past five decades we’ve led the financial services industry through unprecedented historical events and market disruptions. As a result, you should not rely on any forward-looking statements as predictions of future events.

Portfolio 52

Portfolio Cloud Analytics Application Data Security 52

Rockset

JULY 6, 2022

This is the fifth post in a series by Rockset's CTO and Co-founder Dhruba Borthakur on Designing the Next Generation of Data Systems for Real-Time Analytics. Most notable are real-time event streams. After debuting Project Nectar, we presented it to a new set of application developers. NoSQL Comes to the Rescue.

NoSQL 52

NoSQL SQL Systems PostgreSQL 52

Cloudera

SEPTEMBER 1, 2020

It is designed to simplify deployment, configuration, and serviceability of Solr-based analytics applications. DDE also makes it much easier for application developers or data workers to self-service and get started with building insight applications or exploration services based on text or other unstructured data (i.e.

Cloud Storage 67

Cloud Storage Unstructured Data AWS Analytics Application 67

Rockset

MAY 3, 2023

Many industries including gaming, internet and financial services are mature in their adoption of event streaming platforms and have already graduated from data streams to torrents. The data generator generates documents, each document is the size of 1.25KB and represents a single event.

Data Ingestion 59

Data Ingestion Kafka Database Architecture 59

Rockset

MAY 17, 2022

This is the fourth post in a series by Rockset's CTO Dhruba Borthakur on Designing the Next Generation of Data Systems for Real-Time Analytics. The design goal was low latency and scale. Most analytical queries need this ability to join multiple data sources at query time.

SQL 52

SQL NoSQL Hadoop MongoDB 52

Rockset

FEBRUARY 18, 2021

Using RockBench, we ascertained Rockset’s suitability for many real-time analytics applications due to its ability to keep data latency to under 1 second, while ingesting 1 billion events per day, on a standard 4XLarge Virtual Instance. The SSB has also been used for performance measurements of other modern data technologies.

Analytics Application 52

Analytics Application SQL Database Structured Data 52

Knowledge Hut

MAY 31, 2023

Custom analytics applications for IoT data, including machine learning and predictive analytics. MQTT Designed for low-bandwidth communication between devices and a central broker, MQTT (Message Queueing Telemetry Transport) is amongst widely used IoT tools, lightweight publish-subscribe messaging protocol in IoT development.

Programming Language 52

Programming Language Electronics Java Programming 52

Knowledge Hut

DECEMBER 26, 2023

Given its status as one of the complete all-in-one analytics and BI systems available currently, the platform requires some getting accustomed to. Some key features include business intelligence, enterprise planning, and analytics application. TIBCO Spitfire is included in this post due to its extensive functionality.

Business Intelligence 52

Business Intelligence BI Data Mining Data Analysis 52

Knowledge Hut

JUNE 26, 2023

Finally, a well-designed user interface is an essential part of any successful data engineering project. The pipeline in this reference design collects data from two different sources, then conducts a join operation on related records from each stream, then enriches the output, and finally produces an average.

Data Engineering 98

Data Engineering Data Engineer Coding Project 98

Cloudera

FEBRUARY 9, 2021

A typical approach that we have seen in customers’ environments is that ETL applications pull data with a frequency of minutes and land it into HDFS storage as an extra Hive table partition file. In this way, the analytic applications are able to turn the latest data into instant business insights. Design Detail.

Data Warehouse 69

Data Warehouse Cloud Kafka Cloud Storage 69

Rockset

DECEMBER 9, 2021

Disclaimer: Rockset is a real-time analytics database and one of the pieces in the modern real-time data stack So What is Real-Time Data (And Why Can’t the Modern Data Stack Handle It)? Out-of-order event streams. The batch stack is not built to handle this peculiarity of event streams. Deeply-nested JSON and dynamic schemas.

Transportation 52

Transportation BI SQL Data Warehouse 52

Rockset

AUGUST 18, 2020

Modern real-time analytics and applications So what does it look like when real-time analytics is used for serving applications and not just dashboards? This means the lines between real-time analytics and real-time analytical applications are now blurring.

Data Lake 40

Data Lake Data Schemas BI Kafka 40

Rockset

MARCH 19, 2020

Intro In recent years, Kafka has become synonymous with “streaming,” and with features like Kafka Streams, KSQL, joins, and integrations into sinks like Elasticsearch and Druid, there are more ways than ever to build a real-time analytics application around streaming data in Kafka. To accomplish this we can use Kafka streams and KSQL.

Kafka 40

Kafka Database Process SQL 40

ProjectPro

MAY 31, 2021

A big data project is a data analysis project that uses machine learning algorithms and different data analytics techniques on a large dataset for several purposes, including predictive modeling and other advanced analytics applications.

Big Data 52

Big Data Coding Project Hadoop 52

ProjectPro

MAY 23, 2015

For example, an application can be written to subscribe to the Twitter firehose of every tweet written; such an application can analyse the tweets to determine Twitter's most influential users, or identify suddenly prominent events as they occur.

Big Data 52

Big Data Data Analysis Hadoop Retail 52

AltexSoft

JULY 18, 2023

Maintained by the Apache Software Foundation, Apache Spark is an open-source, unified engine designed for large-scale data analytics. The building blocks of Apache Spark Apache Spark comprises a suite of libraries and tools designed for data analysis, machine learning , and graph processing on large-scale data sets.

Big Data 52

Big Data Data Process Process Hadoop 52

AltexSoft

OCTOBER 21, 2022

After trying all options existing on the market — from messaging systems to ETL tools — in-house data engineers decided to design a totally new solution for metrics monitoring and user activity tracking which would handle billions of messages a day. Kafka is designed to handle numerous clients from both sides.

Kafka 93

Kafka Hadoop ETL Tools Big Data 93

Precisely

MARCH 10, 2023

By zeroing in on location, you can better understand relationships between people, structures, and events in the physical world. This capability opens the door to a wide array of data analytics applications. The Rise of Cloud Analytics Data analytics has advanced rapidly over the past decade.

Datasets 59

Datasets Data Warehouse Big Data Cloud Storage 59

Cloudera

APRIL 2, 2019

Intel Optane DC persistent memory was designed with ease of adoption in mind and therefore can be configured in two different operating modes : 1. When in memory mode, the data is not saved in the event of a power loss. memory mode, and 2. app direct mode.

NoSQL 47

NoSQL Google Cloud Hadoop Machine Learning 47

U-Next

SEPTEMBER 9, 2022

In other words, increasingly, organizations are now using Business Analytics to enable proactive decision-making instead of responding to events as they happen. . . Business Analytics is a flourishing technology as it can be used in every sector where data is accessible and collected. Sales increased by 24 percent. . .

Retail 40

Retail Big Data Data Analytics Utilities 40

Cloudera

NOVEMBER 8, 2022

It enables cloud-native applications to store and process mass amounts of data in a hybrid multi-cloud environment and on premises. These could be traditional analytics applications like Spark, Impala, or Hive, or custom applications that access a cloud object store natively. Conclusion.

Metadata 82

Metadata Algorithm Hadoop Cloud 82

AltexSoft

SEPTEMBER 23, 2021

The main purpose of a DW is to enable analytics: It is designed to source raw historical data, apply transformations, and store it in a structured format. This type of storage is a standard part of any business intelligence (BI) system, an analytical interface where users can query data to make business decisions.

Architecture 52

Architecture Data Lake Unstructured Data Data Warehouse 52