Rockset

APRIL 29, 2019

In this blog post I compare options for real-time analytics on DynamoDB - Elasticsearch , Athena, and Spark - in terms of ease of setup, maintenance, query capability, latency. DynamoDB + AWS Lambda + Elasticsearch Another approach to building a secondary index over our data is to use DynamoDB with Elasticsearch.

NoSQL 52

NoSQL PostgreSQL AWS SQL 52

Rockset

APRIL 5, 2021

More recently, they transitioned their internal apps from Elasticsearch to Rockset. We had originally built them on Elasticsearch but migrated to Rockset a year ago. Why did you move search and analytics from Elasticsearch to Rockset? Everything was that much harder when using Elasticsearch DSL.

NoSQL 40

NoSQL SQL Data Science Building 40

Zalando Engineering

DECEMBER 13, 2016

You may have previously read about our use of Elasticsearch at Zalando Tech, and especially our utilization of Elasticsearch Express : An appliance with a toolkit enabling quick deployment and management of Elasticsearch clusters. For us, clusters are not exposed to consumers directly, but rather behind applications.

Transportation 40

Transportation Utilities Process Management 40

Hevo

MAY 29, 2023

Most enterprises collect vast volumes of data over time. This data usually contains important information regarding the business, customers, etc. Storing this data in a stable database is advisable to ensure data security and integrity. MongoDB provides a simple document-type data storing and analyzing environment for your data.

MongoDB 52

MongoDB Data Security Database Data 52

Preset

DECEMBER 15, 2019

Announcing Elasticsearch in Superset, powered by a new open-source Python library from Preset

Python 40

Python 40

Zalando Engineering

NOVEMBER 23, 2016

Elasticsearch is a technology that has been gaining popularity lately at Zalando Tech. The success of Elasticsearch as a technology is due to the various use cases it fits. Whether it is full-text search, structured search, or data analytics, Elasticsearch has no competitor in solving so many diverse data problems.

AWS 40

AWS Data Ingestion Designing Data Analytics 40

Confluent

MAY 26, 2020

Using a powerful, event-driven application can help you unlock insights contained in the event streams of your business. Before we get into the technology, let’s go over some questions you […].

Building 117

Building Technology Kafka Process 117

Netflix Tech

MARCH 10, 2023

We build the data pipeline to persist the assets data in the iceberg in parallel with cassandra and elasticsearch DB. Elasticsearch version upgrade which includes backward incompatible changes, so all the assets data is read from the primary source of truth and reindexed again in the new indices.

Management 94

Management Kafka Metadata Media 94

Yelp Engineering

OCTOBER 5, 2023

While we once used Elasticsearch at Yelp, we have since built a replacement called Nrtsearch. Motivations We traditionally used a gateway to call Elasticsearch, which provides metrics, isolation rate-limiting per client, and geo.

Engineering 69

Engineering Building 69

Confluent

DECEMBER 21, 2021

Collecting and indexing logs from servers, applications, and devices enables crucial visibility into running systems. A log analytics pipeline allows teams to debug and troubleshoot issues, track historical trends, or […].

Systems 90

Systems Programming 90

Netflix Tech

JUNE 15, 2022

With a goal of abstracting users away from the complexity of interacting with Elasticsearch directly, we landed on a custom Studio Search DSL reminiscent of SQL. This also leaves the possibility of swapping Elasticsearch with a different search engine in the future. But, we are not quite done yet!

Engineering 73

Engineering Accessible Accessibility Architecture 73

Netflix Tech

JUNE 1, 2023

We stored these responses in a Keystone stream with outputs for Kafka and Elasticsearch. We used Elasticsearch dashboards to analyze results. It also included metadata about ads, such as ad placement and impression-tracking events.

Algorithm 136

Algorithm Metadata Kafka Systems 136

Netflix Tech

JANUARY 25, 2023

For supporting different search requirements we chose ElasticSearch. Different types of search queries are supported using the Elasticsearch as a backend search engine. We use ElasticSearch to support stem search for 34 different languages. We support semantic search using Open Distro for ElasticSearch .

Algorithm 113

Algorithm Media Metadata Data Ingestion 113

Rockset

JANUARY 30, 2024

Rockset’s cloud-native architecture contrasts with the tightly coupled architecture of Elasticsearch. On benchmarks, Rockset achieved up to 4x faster streaming data ingestion than Elasticsearch. The Elastic Cloud autoscaling API can be used to define policies to monitor the resource utilization of the cluster.

Data Ingestion 64

Data Ingestion Utilities Architecture SQL 64

Data Engineering Podcast

SEPTEMBER 9, 2018

Summary Elasticsearch is a powerful tool for storing and analyzing data, but when using it for logs and other time oriented information it can become problematic to keep all of your history. What are the challenges inherent to scaling an elasticsearch infrastructure to large volumes of log or metric data?

IT 100

IT PostgreSQL Scala AWS 100

Rockset

JUNE 3, 2022

We had been using Elasticsearch since our founding, going back to Elasticsearch 1.5. Elasticsearch underpinned two key processes: The SmartBanners updated instantly when consumers opened their emails The internal dashboard used by our corporate customers to oversee the success of their email campaigns.

Hospitality 52

Hospitality Datasets Aggregated Data Retail 52

Data Engineering Podcast

MARCH 18, 2018

Elasticsearch was built to make it easy to include search functionality in projects built in any language. The Elasticsearch product has been around for a long time and is widely known, but can you give a brief overview of the other components that make up the Elastic Stack and how they work together?

MongoDB 100

MongoDB NoSQL Machine Learning Data Engineering 100

Knowledge Hut

NOVEMBER 17, 2023

Amazon Elasticsearch Service Amazon Elasticsearch Service allows users to organize, lock, activate, and gauge Elasticsearch for log analytics, full-text search, application monitoring, and more. The custom JDBC and ODBC drivers allow users to download info and access it across several SQL clients.

AWS 52

AWS Amazon Web Services PostgreSQL Relational Database 52

Rockset

FEBRUARY 2, 2022

Notable here is Elasticsearch , built on top of Apache Lucene. Additionally, Elasticsearch leverages its indexing to provide the fast analytics that modern data applications require. The main benefit of this kind of approach is that there’s no need to transform data into a relational structure.

MongoDB 52

MongoDB NoSQL Data Warehouse BI 52

Knowledge Hut

NOVEMBER 16, 2023

AWS Elasticsearch: Elasticsearch may be deployed, run, and scaled with the aid of AWS Elasticsearch, a managed service. Data pipeline: It facilitates the transfer of data between services. It serves as a tool for routine, data-driven activities. Kinesis: Working with real-time streaming data in the AWS cloud.

AWS 52

AWS Amazon Web Services Cloud Computing Cloud Storage 52

Rockset

NOVEMBER 6, 2023

Sarthank Nandi explained that this was a challenge with the Elasticsearch deployment they had at Yelp , where every Elasticsearch data node was both an indexer and a searcher, resulting in indexing pressure slowing down search.

Architecture 52

Architecture Engineering Building Systems 52

Rockset

OCTOBER 4, 2023

Elasticsearch is a popular technology for efficient and scalable data storage and retrieval. Indexing is the initial process of adding data to Elasticsearch, while reindexing is essential for maintaining data accuracy and optimizing search performance. Imagine your Elasticsearch indices as well-organized libraries.

NoSQL 59

NoSQL Datasets Data Integration Data Storage 59

Netflix Tech

SEPTEMBER 29, 2022

Secondary indexes For observability purposes, we capture information about incoming messages and their transition between states in two secondary indexes maintained on Elasticsearch. Two Flink jobs — one for each type of index we maintain — consume the events from the corresponding Kafka topics, and update the indexes in Elasticsearch.

Systems 85

Systems Metadata Media Kafka 85

Netflix Tech

JULY 30, 2019

Decoupling Elasticsearch from Persistence This inter-dependency was removed by moving the indexing layer into separate persistence modules, exposing a property ( workflow.elasticsearch.instanceType ) to choose the type of indexing engine. Bulk APIs enable the operators to have macro-level control on the workflows executing within the system.

Metadata 75

Metadata Media AWS Transportation 75

Rockset

MAY 27, 2021

Of course, this is possible today with Elasticsearch. Elasticsearch has an inverted index and it also has doc value storage built using Apache Lucene. Rockset uses RocksDB under the hood, and it's a very different representation than what is done in Elasticsearch. Elastic is used at very high scale.

MongoDB 40

MongoDB Kafka Cloud SQL 40

Netflix Tech

MARCH 10, 2023

In this blog, we will be focusing on how we utilize Elasticsearch for indexing and search the assets. The second layer is Elasticsearch , which is used to discover assets based on user queries. One of the first decisions when integrating with Elasticsearch is designing the indices, their settings and mappings.

Management 103

Management Metadata Digital Media Kafka 103

Data Engineering Podcast

AUGUST 27, 2018

Links Enveil NSA GDPR Intellectual Property Zero Trust Homomorphic Encryption Ciphertext Hadoop PII (Personally Identifiable Information) TLS (Transport Layer Security) Spark Elasticsearch Side-channel attacks Spectre and Meltdown The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA Support Data Engineering Podcast

Transportation 100

Transportation Hadoop Data Security Architecture 100

Rockset

APRIL 12, 2021

Compressing Development Time with Rockset Incorporating Rockset into the CONNEX architecture allowed Command Alkon to shave significant time off the delivery of real-time analytics features compared to the Elasticsearch alternative they had considered. Elasticsearch® is a registered trademark of Elasticsearch BV.

NoSQL 40

NoSQL Transportation Electronics Data Preparation 40

Rockset

JANUARY 23, 2024

Elasticsearch is an open-source search and analytics engine based on Apache Lucene. When building applications on change data capture (CDC) data using Elasticsearch, you’ll want to architect the system to handle frequent updates or modifications to the existing documents in an index.

Metadata 52

Metadata Coding Analytics Application Python 52

Zalando Engineering

APRIL 24, 2019

Elasticsearch Operator - an operator for running Elasticsearch in Kubernetes with focus on operational aspects, like safe draining and offering auto-scaling capabilities for Elasticsearch data nodes, rather than just abstracting manifest definitions.

Cloud 52

Cloud Python Big Data Data Science 52

Data Engineering Weekly

MARCH 3, 2024

Part 2: Rippling's security data lakehouse and modular design DoorDash: Introducing DoorDash’s In-House Search Engine DoorDash writes about its in-house search engine to address scalability and efficiency issues identified with Elasticsearch's previous search architecture.

Data Engineering 67

Data Engineering Data Engineer Pipeline-centric Engineering 67

Data Engineering Podcast

OCTOBER 2, 2022

__init__ Episode Antlr Kotlin programming language PostgreSQL Podcast Episode OpenSearch ElasticSearch Redis Kubernetes Airflow BigQuery Spark Looker Mode The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA Support Data Engineering Podcast

IT 100

IT Food PostgreSQL MongoDB 100

Rockset

JANUARY 11, 2023

One solution commonly referenced is Elasticsearch which we will be diving into today. Elasticsearch is an open-source distributed search engine built on Lucene and used for text search and log analytics use cases. Elasticsearch is part of the larger ELK stack which includes Kibana, a visualization tool for analytical dashboards.

AWS 52

AWS Media NoSQL Database 52

Data Engineering Weekly

APRIL 6, 2023

The talk by Uber on their Ubermetric system migrating from ElasticSearch to Apache Pinot - which, by the way, is an incredibly flexible and powerful system. Will the modern data stack become more mature and consolidated? These are the big questions we need to ask as we move forward in the world of data.

Big Data 52

Big Data Data Governance SQL Government 52

Netflix Tech

OCTOBER 19, 2020

We started with Elasticsearch as our data store due to its flexible data model and querying capabilities. The increased operational burden of scaling ElasticSearch clusters due to high data write rate became painful for us. Storage: don’t break the bank! The high data ingestion rate eventually degraded both read and write operations.

Building 93

Building Transportation Metadata Java 93

Rockset

AUGUST 10, 2020

Another option was to use Elasticsearch, but the expertise and effort required to build out Elasticsearch would have been prohibitive. Continuing the search for a viable solution, David tried out Rockset when performing a web search for the best way to connect DynamoDB to a real-time application without using Elasticsearch.

BI 40

BI AWS SQL Business Intelligence 40

Afterpay Tech

NOVEMBER 23, 2020

The Indexing Service reads data from the DWH and writes into the Elasticsearch index. Search and ranking: The Search Service  calls Elasticsearch to find the stores relevant to the search query. We use Elasticsearch completion suggester to auto-complete store names.

Engineering 52

Engineering Building Kafka Retail 52