Rockset

APRIL 26, 2022

As Kafka Summit is in full swing in London this week and the topic of event streaming is all over my Linkedin feed, I saw a post asking " Is streaming dead? That streaming is rocking and with Kafka Summit this week, I thought it a good time to emphasize the importance of streaming data in today’s modern real-time data stack.

Kafka 52

Kafka Data Warehouse Database Data 52

Data Engineering Podcast

NOVEMBER 26, 2019

__init__ Episode Snuba Blog Post Clickhouse Podcast Episode Disqus Urban Airship HBase Google Bigtable PostgreSQL Redis HyperLogLog Riak Celery RabbitMQ Apache Spark Presto Cassandra Apache Kudu Apache Pinot Apache Druid Flask Apache Kafka Cassandra Tombstone Sentry Blog XML Change Data Capture The intro and outro music is from The Hug by The Freak (..)

Data Warehouse 100

Data Warehouse Building PostgreSQL Kafka 100

Rockset

JULY 28, 2022

As capable as it is, there are still instances where MongoDB alone can't satisfy all of the requirements for an application, so getting a copy of the data into another platform via a change data capture (CDC) solution is required. Debezium It is also possible to capture MongoDB change data capture events using Debezium.

MongoDB 52

MongoDB Kafka NoSQL Data Lake 52

Data Engineering Weekly

MAY 16, 2023

Two-Phase WAP in Real-Time Data Processing The Two-Phase WAP pattern follows the “Fronting Queue” pattern. Since Kafka is almost synonymous with real-time data processing, we often call this a “Fronting Kafka” pattern. The Fronting Kafka pattern follows a two-cluster approach. How to Fix It?

Engineering 110

Engineering Kafka Data Pipeline Data Warehouse 110

Cloudera

SEPTEMBER 17, 2018

These trends and demands lead to stress for existing data warehouse solutions – scale, efficiency, security integrations, IT budgets, ease of access. Cloudera recently launched Cloudera Data Warehouse, a modern data warehousing solution. Here are some highlights: Data Ingest.

Data Warehouse 54

Data Warehouse SQL Portfolio MySQL 54

Confluent

OCTOBER 30, 2020

A fundamental challenge with today’s “data explosion” is finding the best answer to the question, “So where do I put my data?” while avoiding the longer-term problem of data warehouses, […].

Data Architecture 117

Data Architecture Architecture Kafka Building 117

Christophe Blefari

FEBRUARY 24, 2023

When it comes to data storage, the real-time ecosystem has also changed a lot in the last few years and a lot of tooling went out to simplify the burden of managing Kafka clusters, Materialize—a real-time platform—detailed their architecture. Is it Kafka? Often they are right.

Kafka 130

Kafka Data Lake Data Storage Data 130

Christophe Blefari

JANUARY 20, 2024

Data engineering inherits from years of data practices in US big companies. Hadoop initially led the way with Big Data and distributed computing on-premise to finally land on Modern Data Stack — in the cloud — with a data warehouse at the center. Understand Change Data Capture — CDC.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Rockset

MARCH 5, 2020

Apache Kafka has seen broad adoption as the streaming platform of choice for building applications that react to streams of data in real time. In many organizations, Kafka is the foundational platform for real-time event analytics, acting as a central location for collecting event data and making it available in real time.

Kafka 40

Kafka Data Warehouse Data Lake Relational Database 40

Cloudera

JANUARY 20, 2021

Most of what is written though has to do with the enabling technology platforms (cloud or edge or point solutions like data warehouses) or use cases that are driving these benefits (predictive analytics applied to preventive maintenance, financial institution’s fraud detection, or predictive health monitoring as examples) not the underlying data.

Manufacturing 109

Manufacturing Data Warehouse Kafka Retail 109

Knowledge Hut

APRIL 25, 2024

Data warehouses are traditional relational database management systems (RDBMS) that have been enhanced with architectural changes and added functionality to support big data analytics. The two most popular data warehouse systems are Teradata and Oracle Exadata.

Big Data 98

Big Data Technology NoSQL Hadoop 98

Jesse Anderson

DECEMBER 12, 2022

Apache Kafka came in 2011 and gave the industry a much better way to move real-time data. Apache Kafka has its architectural limitations, and Apache Pulsar was released in 2016. Big data projects were given to data scientists and data warehouse teams, where the projects subsequently failed.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Monte Carlo

JANUARY 5, 2024

You know what they always say: data lakehouse architecture is like an onion. …ok, Data lakehouse architecture combines the benefits of data warehouses and data lakes, bringing together the structure and performance of a data warehouse with the flexibility of a data lake. But they should!

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

Monte Carlo

JANUARY 5, 2024

You know what they always say: data lakehouse architecture is like an onion. …ok, Data lakehouse architecture combines the benefits of data warehouses and data lakes, bringing together the structure and performance of a data warehouse with the flexibility of a data lake. But they should!

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

Monte Carlo

DECEMBER 4, 2023

Loading ChatGPT ETL prompts can help write scripts to load data into different databases, data lakes, or data warehouses. I'd like to import this data into my MySQL database into a table called products_table. The data is currently in a pandas DataFrame.

PostgreSQL 52

PostgreSQL Data Lake ETL Tools MySQL 52

Cloudera

JULY 21, 2022

Cloudera users can securely connect Rill to a source of event stream data, such as Cloudera DataFlow , model data into Rill’s cloud-based Druid service, and share live operational dashboards within minutes via Rill’s interactive metrics dashboard or any connected BI solution. Data is made queryable in real time. Apache Hive.

BI 87

BI Digital Media Data Warehouse Kafka 87

Data Engineering Podcast

JULY 27, 2021

RudderStack’s smart customer data pipeline is warehouse-first. It builds your customer data warehouse and your identity graph on your data warehouse, with support for Snowflake, Google BigQuery, Amazon Redshift, and more. RudderStack’s smart customer data pipeline is warehouse-first.

Building 100

Building Management Kafka Data Warehouse 100

Data Engineering Podcast

AUGUST 3, 2021

RudderStack’s smart customer data pipeline is warehouse-first. It builds your customer data warehouse and your identity graph on your data warehouse, with support for Snowflake, Google BigQuery, Amazon Redshift, and more. RudderStack’s smart customer data pipeline is warehouse-first.

Data Lake 130

Data Lake Data Warehouse Hadoop Architecture 130

Knowledge Hut

NOVEMBER 28, 2023

Concepts of IaaS, PaaS, and SaaS are the trend, and big companies expect data engineers to have the relevant knowledge. Kafka Kafka is one of the most desired open-source messaging and streaming systems that allows you to publish, distribute, and consume data streams. ETL is central to getting your data where you need it.

Data Engineering 52

Data Engineering Data Engineer Engineering Generalist 52

Snowflake

SEPTEMBER 14, 2023

Unbound by the limitations of a legacy on-premises solution, its multi-cluster shared data architecture separates compute from storage, allowing data teams to easily scale up and down based on their needs. Data engineers spent 20% of their time on infrastructure issues such as tuning Spark jobs.

Data Architecture 97

Data Architecture Architecture Hadoop Data Warehouse 97

Data Engineering Weekly

NOVEMBER 26, 2023

We saw a fleet of announcements from Data Catalogs tools on how LLM can help to auto-generate documentation [See: How Generative AI Is Making Data Catalogs Smarter ]. I believe the impact of LLM will go further down in the stack with data storage formats in the coming years.

Data Engineering 70

Data Engineering Data Engineer Engineering Kafka 70

Data Engineering Podcast

MARCH 8, 2021

Datafold also helps automate regression testing of ETL code with its Data Diff feature that instantly shows how a change in ETL or BI code affects the produced data, both on a statistical level and down to individual rows and values. RudderStack’s smart customer data pipeline is warehouse-first.

IT 100

IT Data Warehouse MongoDB Kafka 100

Knowledge Hut

MARCH 28, 2024

Role Level: Intermediate Responsibilities Design and develop big data solutions using Azure services like Azure HDInsight, Azure Databricks, and Azure Data Lake Storage. Implement data ingestion, processing, and analysis pipelines for large-scale data sets. Familiarity with ETL tools and techniques for data integration.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Data Engineering Podcast

DECEMBER 4, 2022

Datafold shows how a change in SQL code affects your data, both on a statistical level and down to individual rows and values before it gets merged to production. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows.

Data Lake 100

Data Lake MongoDB MySQL Data Warehouse 100

Confluent

APRIL 6, 2022

Confluent has been offering our customers the opportunity to seamlessly connect their Apache Kafka® topics to Google BigQuery for several years. This helps accelerate data warehouse initiatives by connecting more […].

Google Cloud 62

Google Cloud Kafka Cloud Data Warehouse 62

Data Engineering Podcast

OCTOBER 28, 2021

In the same way that application performance monitoring ensures reliable software and keeps application downtime at bay, Monte Carlo solves the costly problem of broken data pipelines. Start trusting your data with Monte Carlo today! Hightouch is the easiest way to sync data into the platforms that your business teams rely on.

Data Pipeline 100

Data Pipeline SQL Data Warehouse Data Lake 100

Data Engineering Podcast

MAY 11, 2020

You monitor your website to make sure that you’re the first to know when something goes wrong, but what about your data? Tidy Data is the DataOps monitoring platform that you’ve been missing. What motivates you to dedicate so much of your time and enery to Pulsar in particular, and the streaming data ecosystem in general?

Lambda Architecture 100

Lambda Architecture Cloud Kafka Hadoop 100

Data Engineering Podcast

MAY 10, 2021

RudderStack’s smart customer data pipeline is warehouse-first. It builds your customer data warehouse and your identity graph on your data warehouse, with support for Snowflake, Google BigQuery, Amazon Redshift, and more. RudderStack’s smart customer data pipeline is warehouse-first.

PostgreSQL 100

PostgreSQL Data Warehouse Data Pipeline Kafka 100

Christophe Blefari

MARCH 4, 2023

Data ingestion pipeline with Operation Management — At Netflix they annotate video which can lead to thousand of annotation but they need to manage the annotation lifecycle each time the annotation algorithm runs. Some company also call it a lakehouse or a data lake, but the word shift is enough interesting to notice.

Machine Learning 130

Machine Learning AWS Data Data Lake 130

Christophe Blefari

JUNE 16, 2023

Why data consumers do not trust your reporting — It is a good illustration of the data journey manifesto. Stakeholders often notice data issues before the data team does. Data warehouses are mutable, this is one of the many root causes proposed by Lucas. Data Documentation 101: Why?

Programming Language 130

Programming Language SQL PostgreSQL Data 130

Cloudera

JANUARY 19, 2022

Cloudera Data Platform (CDP) scored among the top 10 vendors on all four Analytical Use Cases — Data Warehouse, Logical Data Warehouse, Data Lake and Operational Intelligence in the Critical Capabilities for Cloud Database Management Systems for Analytics Use Cases.

Database 91

Database Cloud Data Warehouse Data Lake 91

Data Engineering Weekly

FEBRUARY 19, 2023

If you’re still heavily reliant on Google for data collection and reporting, now is the perfect time to center your data analytics strategy around your data warehouse. Join our webinar to learn how you can replace GA with analytics on your data cloud.

Data Engineering 52

Data Engineering Data Engineer Engineering Google Cloud 52

Data Engineering Podcast

AUGUST 21, 2022

Select Star’s data discovery platform solves that out of the box, with an automated catalog that includes lineage from where the data originated, all the way to which dashboards rely on it and who is viewing them every day.

Lambda Architecture 100

Lambda Architecture MongoDB Scala MySQL 100

Data Engineering Podcast

NOVEMBER 10, 2021

In the same way that application performance monitoring ensures reliable software and keeps application downtime at bay, Monte Carlo solves the costly problem of broken data pipelines. Start trusting your data with Monte Carlo today! Hightouch is the easiest way to sync data into the platforms that your business teams rely on.

Metadata 100

Metadata Data Warehouse Data Lake BI 100

Rockset

MARCH 19, 2020

Intro In recent years, Kafka has become synonymous with “streaming,” and with features like Kafka Streams, KSQL, joins, and integrations into sinks like Elasticsearch and Druid, there are more ways than ever to build a real-time analytics application around streaming data in Kafka.

Kafka 40

Kafka Database Process SQL 40

ProjectPro

DECEMBER 7, 2021

Generally, data pipelines are created to store data in a data warehouse or data lake or provide information directly to the machine learning model development. Keeping data in data warehouses or data lakes helps companies centralize the data for several data-driven initiatives.

Data Pipeline 52

Data Pipeline Architecture Kafka AWS 52

Data Engineering Podcast

JANUARY 5, 2020

Debezium is an open source platform for reliable change data capture that you can use to build supplemental systems for everything from maintaining audit trails to real-time updates of your data warehouse. What are some of the other options on the market for handling change data capture? Pulsar, Bookkeeper, Pravega)?

Database 100

Database Kafka PostgreSQL MySQL 100