Lyft Engineering

NOVEMBER 29, 2023

In this particular blog post, we explain how Druid has been used at Lyft and what led us to adopt ClickHouse for our sub-second analytic system. Druid at Lyft Apache Druid is an in-memory, columnar, distributed, open-source data store designed for sub-second queries on real-time and historical data.

Kafka 104

Kafka Data Ingestion Datasets Architecture 104

Rockset

JANUARY 30, 2024

Microbatching : An option to microbatch ingestion based on the latency requirements of the use case. In this blog, we delve into each of these features and how they are giving users more cost controls for their search and AI applications. This is not a hands-free operation and also involves the transfer of data across nodes.

Data Ingestion 64

Data Ingestion Utilities Architecture SQL 64

Cloudera

AUGUST 26, 2021

In addition to big data workloads, Ozone is also fully integrated with authorization and data governance providers namely Apache Ranger & Apache Atlas in the CDP stack. While we walk through the steps one by one from data ingestion to analysis, we will also demonstrate how Ozone can serve as an ‘S3’ compatible object store.

Data Science 109

Data Science Cloud Hadoop Metadata 109

FreshBI

SEPTEMBER 11, 2023

. ” In the continuously evolving field of data-driven insights, maintaining competitiveness relies not only on in-depth analysis but also on the rapid and precise development of reports. Power BI, Microsoft's cutting-edge business analytics solution, empowers users to visualize data and seamlessly distribute insights.

BI 59

BI Consulting Datasets Data Ingestion 59

DataKitchen

MAY 10, 2024

The First of Five Use Cases in Data Observability Data Evaluation: This involves evaluating and cleansing new datasets before being added to production. This process is critical as it ensures data quality from the onset. Examples include regular loading of CRM data and anomaly detection.

Data Cleanse 49

Data Cleanse Data Ingestion Data Datasets 49

Cloudyard

APRIL 4, 2024

However, the volume of daily transaction data poses challenges in effectively segmenting customers and optimizing engagement. This blog post explores how Snowpark, a powerful tool for data processing within Snowflake, can be used to perform RFM segmentation and unlock actionable customer insights.

Retail 40

Retail Data Ingestion Metadata Datasets 40

ProjectPro

FEBRUARY 6, 2023

With over 10 million active subscriptions, 50 million active topics, and a trillion messages processed per day, Google Cloud Pub/Sub makes it easy to build and manage complex event-driven systems. Google Cloud Pub/Sub is a global, cloud-based messaging framework that has become increasingly popular among data engineers over recent years.

Google Cloud 52

Google Cloud Cloud Cloud Storage Data Ingestion 52

Cloudera

DECEMBER 14, 2021

So to improve the speed of data analysis, the IRS worked with the combined technology integrating Cloudera Data Platform (CDP) and NVIDIA’s RAPIDS Accelerator for Apache Spark 3.0. However, the CBA is a huge institution with 15 million customers and 700M daily transactions — managing the growing influx of data was challenging. .

Transportation 107

Transportation Telecommunication Banking Data Lake 107

Databand.ai

JULY 17, 2023

DataOps , short for Data Operations, is an emerging discipline that focuses on improving the collaboration, integration, and automation of data management processes. It aims to streamline the entire data lifecycle—from ingestion and preparation to analytics and reporting.

Data Pipeline 52

Data Pipeline Machine Learning High Quality Data BI 52

DataKitchen

OCTOBER 2, 2023

Data Teams and Their Types of Data Journeys In the rapidly evolving landscape of data management and analytics, data teams face various challenges ranging from data ingestion to end-to-end observability. It explores why DataKitchen’s ‘Data Journeys’ capability can solve these challenges.

Data Ingestion 40

Data Ingestion Data Government Datasets 40

Cloudera

JANUARY 20, 2021

The missing chapter is not about point solutions or the maturity journey of use cases, the missing chapter is about the data, it’s always been about the data, and most importantly the journey data weaves from edge to artificial intelligence insight. . Data Collection Challenge.

Manufacturing 105

Manufacturing Data Warehouse Kafka Retail 105

Knowledge Hut

MARCH 19, 2024

I will explore the top 10 AWS applications and their use cases in this blog. Lambda usage includes real-time data processing, communication with IoT devices, and execution of automated tasks. ECS is mostly used in microservices architecture , batch processing, and CI/CD pipelines. What is AWS?

AWS 52

AWS Cloud Computing Amazon Web Services Relational Database 52

Cloudera

FEBRUARY 16, 2021

In the previous blog post , we looked at some of the application development concepts for the Cloudera Operational Database (COD). In this blog post, we’ll see how you can use other CDP services with COD. Integrated across the Enterprise Data Lifecycle . Read about Building a Scalable Process Using NiFi, Kafka, and HBase on CDP.

Database 81

Database Machine Learning Data Lake Kafka 81

Cloudera

NOVEMBER 17, 2020

Once you have deployed the template and all the CML artifacts that go with it, you can unpick and work it backward to map the process to your own data in your own environment. . It’s not the raw transaction data but rather an anonymized feature set based on a principle component analysis (PCA) of the original data.

Deep Learning 110

Deep Learning Machine Learning Raw Data Data Ingestion 110

LinkedIn Engineering

DECEMBER 12, 2023

We say that an algorithm is differentially private if any result of the algorithm cannot depend too much on any single data record in a dataset. Hence, differential privacy provides uncertainty for whether or not an individual data record is in the dataset. for all the results that have been provided.

Algorithm 77

Algorithm Metadata SQL Datasets 77

Cloudera

APRIL 9, 2021

This is part 4 in this blog series. This blog series follows the manufacturing and operations data lifecycle stages of an electric car manufacturer – typically experienced in large, data-driven manufacturing companies. The second blog dealt with creating and managing Data Enrichment pipelines.

Machine Learning 98

Machine Learning Manufacturing Data Collection Data Science 98

Confluent

FEBRUARY 6, 2019

The blog posts How to Build and Deploy Scalable Machine Learning in Production with Apache Kafka and Using Apache Kafka to Drive Cutting-Edge Machine Learning describe the benefits of leveraging the Apache Kafka ® ecosystem as a central, scalable and mission-critical nervous system. For now, we’ll focus on Kafka.

Machine Learning 107

Machine Learning Python Kafka Java 107

Monte Carlo

MARCH 24, 2023

We’ll explore their definitions, purposes, and methods so you can ensure both data integrity and data validity in your organization. What is Data Integrity? Data integrity is the process of maintaining the consistency, accuracy, and trustworthiness of data throughout its lifecycle, including storage, retrieval, and usage.

Data Validation 52

Data Validation Data Integration Data Cleanse Data Pipeline 52

LinkedIn Engineering

JUNE 15, 2023

To remove this bottleneck, we built AvroTensorDataset , a TensorFlow dataset for reading, parsing, and processing Avro data. AvroTensorDataset speeds up data preprocessing by multiple orders of magnitude, enabling us to keep site content as fresh as possible for our members. avro", "part-00001.avro"],

Datasets 102

Datasets Bytes Process Data Ingestion 102

DareData

JULY 5, 2023

In today's data-driven world, organizations are faced with the challenge of managing and processing large volumes of data efficiently. To overcome this challenge, many companies are turning to Data Lake solutions, which provide a centralized and scalable platform for storing, processing, and analyzing data.

Data Lake 52

Data Lake Building Raw Data ETL Tools 52

RandomTrees

SEPTEMBER 6, 2020

This stage handles all the aspects of data storage like organization, file size, structure, compression, metadata, statistics. The data objects are accessible only through SQL query operations run using Snowflake. Query Processing: Query processing in Snowflake is done using virtual warehouses.

Cloud Storage 52

Cloud Storage Data Ingestion Data Cleanse Data Warehouse 52

Netflix Tech

OCTOBER 19, 2020

In our previous blog post we introduced Edgar, our troubleshooting tool for streaming sessions. Reconstructing a streaming session was a tedious and time consuming process that involved tracing all interactions (requests) between the Netflix app, our Content Delivery Network (CDN), and backend microservices.

Building 93

Building Transportation Metadata Java 93

Pinterest Engineering

SEPTEMBER 12, 2023

Behind the scenes, hundreds of ML engineers iteratively improve a wide range of recommendation engines that power Pinterest, processing petabytes of data and training thousands of models using hundreds of GPUs. transformers) became standardized, ML engineers started to show a growing appetite to iterate on datasets.

Data Process 88

Data Process Process Datasets Scala 88

ProjectPro

FEBRUARY 21, 2023

With so many data engineering certifications available , choosing the right one can be a daunting task. There are over 133K data engineer job openings in the US, but how will you stand out in such a crowded job market? Why Are Data Engineering Skills In Demand? Don’t worry!

Certification 52

Certification Data Engineering Data Engineer Engineering 52

Rockset

MAY 17, 2022

This is the fourth post in a series by Rockset's CTO Dhruba Borthakur on Designing the Next Generation of Data Systems for Real-Time Analytics. We'll be publishing more posts in the series in the near future, so subscribe to our blog so you don't miss them! Instant scaleup of data writes or queries to handle bursts of data.

SQL 52

SQL NoSQL Hadoop MongoDB 52

ProjectPro

DECEMBER 7, 2021

Data pipelines are a significant part of the big data domain, and every professional working or willing to work in this field must have extensive knowledge of them. Data Pipeline Tools AWS Data Pipeline Azure Data Pipeline Airflow Data Pipeline Learn to Create a Data Pipeline FAQs on Data Pipeline What is a Data Pipeline?

Data Pipeline 52

Data Pipeline Architecture Kafka AWS 52

Cloudera

JULY 18, 2018

Analyzing a variety of data not limited to, clinical and biological data will be necessary to gain valuable insight. Having the proper infrastructure with the required storage and processing capacity will be expected in order to efficiently design, train, execute, and deploy machine learning and AI solutions.

Machine Learning 50

Machine Learning Healthcare Medical Electronics 50

Cloudera

OCTOBER 22, 2020

Replication ( covered in this previous blog article ) has been released for a while and is among the most used features of Apache HBase. Although it is an efficient way of keeping different HBase databases in sync within a sub-second latency, replication only operates over data ingested after the feature has been enabled.

Bytes 61

Bytes Datasets Data Data Ingestion 61

ProjectPro

FEBRUARY 16, 2023

From monitoring and searching through big data to generating alerts, reports, and visualizations, Splunk offers several such features to help businesses achieve their goals. This clearly shows how crucial it is for data engineers to be familiar with the Splunk platform if they want to succeed in the big data industry.

Big Data 52

Big Data Big Data Tools Cloud Data 52

Rockset

JULY 19, 2021

However, using Druid can be problematic when used outside its normal parameters — for example, to work with nested data. In this article, we’ll discuss ingesting and using nested data in Apache Druid. Druid doesn’t store nested data in the form often found in, say, a JSON dataset.

Programming Language 40

Programming Language Data Ingestion Data Database 40

Knowledge Hut

APRIL 25, 2023

An end-to-end Data Science pipeline starts from business discussion to delivering the product to the customers. One of the key components of this pipeline is Data ingestion. It helps in integrating data from multiple sources such as IoT, SaaS, on-premises, etc., What is Data Ingestion?

Data Ingestion 52

Data Ingestion Lambda Architecture Raw Data Kafka 52

Cloudera

SEPTEMBER 17, 2018

The volume of data is now in the petabytes, and businesses require high demand on availability and reliability. An organization with bottlenecks to acquire, prepare, process, and serve data – can lead to important decisions being made with stale data. Here are some highlights: Data Ingest.

Data Warehouse 52

Data Warehouse SQL Portfolio MySQL 52

Rockset

MAY 3, 2023

To find out, we decided to test the streaming ingestion performance of Rockset’s next generation cloud architecture and compare it to open-source search engine Elasticsearch , a popular sink for Apache Kafka. For this benchmark, we evaluated Rockset and Elasticsearch ingestion performance on throughput and data latency.

Data Ingestion 59

Data Ingestion Kafka Database Architecture 59

Rockset

MARCH 1, 2023

When you deconstruct the core database architecture, deep in the heart of it you will find a single component that is performing two distinct competing functions: real-time data ingestion and query serving. When data ingestion has a flash flood moment, your queries will slow down or time out making your application flaky.

Data Ingestion 144

Data Ingestion Database Architecture Cloud Storage 144

ProjectPro

JANUARY 25, 2022

Here’s What You Need to Know About PySpark This blog will take you through the basics of PySpark, the PySpark architecture, and a few popular PySpark libraries , among other things. Finally, you'll find a list of PySpark projects to help you gain hands-on experience and land an ideal job in Data Science or Big Data.

Big Data 52

Big Data Data Process Process Kafka 52

Snowflake

APRIL 18, 2024

This blog post explores how Snowflake can help with this challenge. Legacy SIEM cost factors to keep in mind Data ingestion: Traditional SIEMs often impose limits to data ingestion and data retention. Security teams can also reduce their costs by loading certain datasets in batches instead of continuously.

Data Lake 78

Data Lake Data Ingestion Bytes Cloud Computing 78

Cloudera

MAY 19, 2021

In the previous blog post in this series, we walked through the steps for leveraging Deep Learning in your Cloudera Machine Learning (CML) projects. To try and predict this, an extensive dataset including anonymised details on the individual loanee and their historical credit history are included. Get the Dataset. Introduction.

Machine Learning 136

Machine Learning Datasets Data Science Raw Data 136