Data Engineering Podcast

AUGUST 6, 2022

Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. report having current investments in automation, 85% of data teams plan on investing in automation in the next 12 months.

Machine Learning 100

Machine Learning Database MySQL PostgreSQL 100

Rockset

AUGUST 30, 2021

Apache Kafka has made acquiring real-time data more mainstream, but only a small sliver are turning batch analytics, run nightly, into real-time analytical dashboards with alerts and automatic anomaly detection. But until this release, all these data sources involved indexing the incoming raw data on a record by record basis.

SQL 52

SQL Kafka MongoDB MySQL 52

Striim

NOVEMBER 13, 2023

Introduction In the fast-evolving world of data integration, Striim’s collaboration with Snowflake stands as a beacon of innovation and efficiency. This mode is particularly useful for audit trails or scenarios where preserving the historical sequence of data changes is important.

Data Ingestion 59

Data Ingestion Utilities Data Integration Data 59

Rockset

OCTOBER 11, 2022

Introduction Managing streaming data from a source system, like PostgreSQL, MongoDB or DynamoDB, into a downstream system for real-time analytics is a challenge for many teams. Logstash is an event processing pipeline that ingests and transforms data before sending it to Elasticsearch.

Data Ingestion 52

Data Ingestion Kafka Relational Database PostgreSQL 52

Rockset

APRIL 29, 2019

As opposed to a traditional RDBMS like PostgreSQL, DynamoDB scales horizontally, obviating the need for careful capacity planning, resharding, and database maintenance. The setup requires metrics and monitoring to ensure that it is correctly processing events from the DynamoDB stream and able to write into Elasticsearch.

NoSQL 52

NoSQL PostgreSQL AWS SQL 52

Edureka

FEBRUARY 9, 2023

As the demand for data engineers grows, having a well-written resume that stands out from the crowd is critical. Azure data engineers are essential in the design, implementation, and upkeep of cloud-based data solutions. It is also crucial to have experience with data ingestion and transformation.

Data Engineering 52

Data Engineering Data Engineer Engineering Amazon Web Services 52

Rockset

NOVEMBER 28, 2022

First, CDC theoretically allows companies to analyze and react to data in real time, as it’s generated. It works with existing streaming systems like Apache Kafka, Amazon Kinesis, and Azure Events Hubs, making it easier than ever to build a real-time data pipeline.

Data Warehouse 52

Data Warehouse PostgreSQL MongoDB Data Pipeline 52