Rockset

SEPTEMBER 14, 2021

We’re introducing a new Rockset Integration for Apache Kafka that offers native support for Confluent Cloud and Apache Kafka, making it simpler and faster to ingest streaming data for real-time analytics. There is no need to pre-create a schema to run real-time analytics on event streams from Kafka.

Kafka 52

Kafka SQL MongoDB Computer Science 52

Striim

NOVEMBER 30, 2022

Businesses of all scales and industries have access to increasingly large amounts of data, which need to be harnessed effectively. According to an IDG Market Pulse survey , companies collect data from 400 sources on average. However, fragmented data can slow down the delivery of great product experiences and internal operations.

AWS 52

AWS Cloud Management MySQL 52

Big Data Tools

MAY 19, 2022

I’ll offer my impressions of recent developments in the data engineering space and highlight new ideas from the wider community. News A lot of engineering is about learning new things and keeping a finger on the pulse of new technologies. Here’s what’s happening in the world of data engineering right now.

Data Engineering 52

Data Engineering Data Engineer Engineering Big Data Tools 52

Big Data Tools

MAY 19, 2022

I’ll offer my impressions of recent developments in the data engineering space and highlight new ideas from the wider community. News A lot of engineering is about learning new things and keeping a finger on the pulse of new technologies. Here’s what’s happening in the world of data engineering right now.

Data Engineering 52

Data Engineering Data Engineer Engineering Big Data Tools 52

Data Engineering Podcast

OCTOBER 7, 2019

Summary The process of exposing your data through a SQL interface has many possible pathways, each with their own complications and tradeoffs. One of the recent options is Rockset, a serverless platform for fast SQL analytics on semi-structured and structured data. This week’s episode is also sponsored by Datacoral.

Structured Data 100

Structured Data Cloud SQL Programming Language 100

Big Data Tools

JUNE 8, 2022

I’ll offer my impressions of recent developments in the data engineering space and highlight new ideas from the wider community. News A lot of engineering is about learning new things and keeping a finger on the pulse of new technologies. Here’s what’s happening in the world of data engineering right now.

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Big Data Tools

JUNE 8, 2022

I’ll offer my impressions of recent developments in the data engineering space and highlight new ideas from the wider community. News A lot of engineering is about learning new things and keeping a finger on the pulse of new technologies. Here’s what’s happening in the world of data engineering right now.

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Rockset

JANUARY 3, 2023

This blog compiles real-time data predictions from industry leaders so you know what’s coming in 2023. Confluent’s State of Data in Motion Report found that 97% of companies around the world are using streaming data, making it central to the data landscape.

Data Warehouse 52

Data Warehouse Education Database Data 52

Confluent

JUNE 12, 2019

Confluent Cloud, a fully managed event cloud-native streaming service that extends the value of Apache Kafka ® , is simple, resilient, secure, and performant, allowing you to focus on what is important—building contextual event-driven applications, not infrastructure. Helm Chart configuration for Confluent Cloud.

Cloud 92

Cloud Kafka Healthcare Software Engineer 92

Data Engineering Weekly

APRIL 2, 2023

Contribute to the Rudderstack Transformations Library, Win $1000 RudderStack Transformations lets you customize event data in real-time with your own JavaScript or Python code. Meta: Presto - A Decade of SQL Analytics at Meta Presto and Kafka are the two systems that greatly impacted data infrastructure in the last decade.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Knowledge Hut

JANUARY 22, 2024

In the ever-evolving landscape of technology, where data reigns supreme, the pursuit of mastery in data science, specifically exploring "Data Science Course Fees," has become more than a professional endeavor—it's a journey into the heart of innovation. Welcome to the world of data science.

Data Science 52

Data Science Certification Education Data Lake 52

Big Data Tools

OCTOBER 10, 2022

I am now delighted to have the privilege of returning to the task of collecting for you the most exciting news from the world of data engineering. I’ll offer my impressions of recent developments in the data engineering space and highlight new ideas from the wider community. Greetings from sunny Berlin!

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Big Data Tools

OCTOBER 10, 2022

I am now delighted to have the privilege of returning to the task of collecting for you the most exciting news from the world of data engineering. I’ll offer my impressions of recent developments in the data engineering space and highlight new ideas from the wider community. Greetings from sunny Berlin!

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Confluent

AUGUST 29, 2019

We know that Apache Kafka ® is great when you’re dealing with streams, allowing you to conveniently look at streams as tables. Stream processing engines like KSQL furthermore give you the ability to manipulate all of this fluently. The approach we’ll use works with any Kafka run though.

Kafka 55

Kafka Process Algorithm Cloud 55

Rockset

MARCH 18, 2024

Flink is one of the most popular stream processing technologies, ranked as a top five Apache project and backed by a diverse committer community including Alibaba and Apple. Rockset customers using Flink often share how challenging it is to self-manage Flink for streaming transformations. What is RAG?

Cloud 64

Cloud Building Metadata Kafka 64

DoorDash Engineering

DECEMBER 5, 2023

DoorDash’s Engineering teams revamped Kafka Topic creation by replacing a Terraform/Atlantis based approach with an in-house API, Infra Service. DoorDash’s Real-Time Streaming Platform, or RTSP, team is under the Data Platform organization and manages over 2,500 Kafka Topics across five clusters.

Kafka 90

Kafka Programming Language Metadata Architecture 90

Data Engineering Weekly

MARCH 31, 2024

Intuit: How Intuit data analysts write SQL 2x faster with the internal GenAI tool The productivity increase with GenAI is undeniable, and several startups are trying to solve the Text2SQL generation problem. My key highlight is that Excellent data documentation and “clean data” improve results.

Data Engineering 62

Data Engineering Data Engineer Engineering Scala 62

Snowflake

MARCH 2, 2023

Snowflake enables organizations to be data-driven by offering an expansive set of features for creating performant, scalable, and reliable data pipelines that feed dashboards, machine learning models, and applications. But before data can be transformed and served or shared, it must be ingested from source systems.

Kafka 123

Kafka Data Ingestion Data Pipeline Cloud Storage 123

Confluent

MAY 30, 2019

Following part 1 and part 2 of the Spring for Apache Kafka Deep Dive blog series, here in part 3 we will discuss another project from the Spring team: Spring Cloud Data Flow , which focuses on enabling developers to easily develop, deploy, and orchestrate event streaming pipelines based on Apache Kafka ®.

Kafka 93

Kafka Cloud Data Pipeline PostgreSQL 93

Snowflake

APRIL 19, 2023

Welcome to the third blog post in our series highlighting Snowflake’s data ingestion capabilities, covering the latest on Snowpipe Streaming (currently in public preview) and how streaming ingestion can accelerate data engineering on Snowflake. What is Snowpipe Streaming?

Data Ingestion 57

Data Ingestion Kafka Java Data Pipeline 57

Knowledge Hut

JANUARY 18, 2024

“Big data Analytics” is a phrase that was coined to refer to amounts of datasets that are so large traditional data processing software simply can’t manage them. For example, big data is used to pick out trends in economics, and those trends and patterns are used to predict what will happen in the future.

Data Process 52

Data Process Process Hadoop Scala 52

Confluent

JUNE 26, 2019

I see this pattern coming up more and more in the field in conjunction with Apache Kafka ®. In these projects, microservice architectures use Kafka as an event streaming platform. Apache Kafka – An event streaming platform for microservices. Store streams of events in a fault-tolerant way. Microservices.

Kafka 109

Kafka Designing Architecture ETL Tools 109

Rockset

JUNE 8, 2023

Streaming data adoption continues to accelerate - over 80% of Fortune 100 companies already use Apache Kafka - driven by organizations creating value by putting data to use in real time. Much of this streaming data will land in real-time analytics databases as event streams.

Kafka 52

Kafka Algorithm Database Architecture 52

AltexSoft

OCTOBER 21, 2022

Kafka can continue the list of brand names that became generic terms for the entire type of technology. Similar to Google in web browsing and Photoshop in image processing, it became a gold standard in data streaming, preferred by 70 percent of Fortune 500 companies. What is Kafka? What Kafka is used for.

Kafka 93

Kafka Hadoop ETL Tools Big Data 93

Rockset

MAY 3, 2023

Rockset is a database used for real-time search and analytics on streaming data. In scenarios involving analytics on massive data streams, we’re often asked the maximum throughput and lowest data latency Rockset can achieve and how it stacks up to other databases. Why measure streaming data ingestion?

Data Ingestion 59

Data Ingestion Kafka Database Architecture 59

Confluent

JUNE 11, 2019

For event streaming application developers, it is important to continuously update the streaming pipeline based on the need for changes in the individual applications in the pipeline. It is also important to understand some of the common streaming topologies that streaming developers use to build an event streaming pipeline.

Kafka 84

Kafka Cloud Java MongoDB 84

Rockset

FEBRUARY 24, 2022

Which databases are optimized for ingesting streaming events and analyzing them in real time? We’ll start by evaluating three options for running real-time analytics on AWS Kinesis event streams. About Using Event Data Events are messages that are sent by a system to notify operators or other systems about a change in its domain.

AWS 52

AWS Amazon Web Services Kafka SQL 52

Snowflake

MARCH 30, 2023

A 2016 data science report from data enrichment platform CrowdFlower found that data scientists spend around 80% of their time in data preparation (collecting, cleaning, and organizing of data) before they can even begin to build machine learning (ML) models to deliver business value.

Engineering 81

Engineering Raw Data Data Science Scala 81

Rockset

OCTOBER 11, 2022

Introduction Managing streaming data from a source system, like PostgreSQL, MongoDB or DynamoDB, into a downstream system for real-time analytics is a challenge for many teams. For a system like Elasticsearch , engineers need to have in-depth knowledge of the underlying architecture in order to efficiently ingest streaming data.

Data Ingestion 52

Data Ingestion Kafka Relational Database PostgreSQL 52

Confluent

MAY 22, 2019

In light of this, we’ll share an emerging machine-to-machine (M2M) architecture pattern in which MQTT, Apache Kafka ® , and Scylla all work together to provide an end-to-end IoT solution. The explosive number of devices generating, tracking and sharing data across a variety of networks is overwhelming to most data management solutions.

Kafka 96

Kafka Google Cloud NoSQL Entertainment 96

ProjectPro

JANUARY 19, 2022

Planning to land a successful job as an Azure Data Engineer? Read this blog till the end to learn more about the roles and responsibilities, necessary skillsets, average salaries, and various important certifications that will help you build a successful career as an Azure Data Engineer.

Data Engineering 52

Data Engineering Data Engineer Engineering Scala 52

AltexSoft

NOVEMBER 7, 2022

You can hardly compare data engineering toil with something as easy as breathing or as fast as the wind. The platform went live in 2015 at Airbnb, the biggest home-sharing and vacation rental site, as an orchestrator for increasingly complex data pipelines. How data engineering works. What is Apache Airflow?

PostgreSQL 52

PostgreSQL Metadata Python MySQL 52

Cloudera

AUGUST 20, 2021

How CDP Enables and Accelerates Data Product Ecosystems. A multi-purpose platform focused on diverse value propositions for data products. As a result, CDP-enabled data products can meet multiple and varying functional and non-functional requirements that correspond to product attributes, each fulfilling specific customer needs.

Data Warehouse 94

Data Warehouse Data Cloud Architecture 94

Rockset

NOVEMBER 6, 2019

With event-driven architectures powered by systems like Apache Kafka becoming more prominent, there are now many applications in the modern software stack that make use of events and messages to operate effectively. Supporting separation of data access (read/write) mechanisms like CQRS.

Kafka 40

Kafka Data Lake SQL Hadoop 40

ProjectPro

NOVEMBER 15, 2021

The adaptability and technical superiority of such open-source big data projects make them stand out for community use. As per the surveyors, Big data (35 percent), Cloud computing (39 percent), operating systems (33 percent), and the Internet of Things (31 percent) are all expected to be impacted by open source shortly.

Big Data 52

Big Data Project Metadata Programming Language 52

Rockset

MAY 8, 2023

In this blog, we capture engineering stories from 5 early adopters of vector search- Pinterest, Spotify, eBay, Airbnb and Doordash- who have integrated AI into their applications. In the next sections, we’ll summarize 5 engineering blogs on vector search and highlight key implementation considerations.

Metadata 52

Metadata Algorithm Datasets Google Cloud 52

Rockset

NOVEMBER 1, 2022

This blog outlines best practices from customers I have helped migrate from Elasticsearch to Rockset , reducing risk and avoiding common pitfalls. So popular still today that Rockset engineers use it for our own internal log search functions. In this blog, we distilled their migration journeys into 5 steps.

Database-centric 52

Database-centric Pipeline-centric SQL Aggregated Data 52