Monte Carlo

DECEMBER 4, 2023

Tune the load process I'm using PostgreSQL to store my company's transactional data. Implement Upserts I'm frequently updating and inserting new records into my PostgreSQL users table. Provide guidance and best practices on specific ETL tools Say you’re new to Apache Kafka. I've heard about the UPSERT functionality.

PostgreSQL 52

PostgreSQL Data Lake ETL Tools MySQL 52

Christophe Blefari

JUNE 16, 2023

Change Data Capture (CDC) with PostgreSQL and ClickHouse — This is a nice vendor post about CDC with Kafka as movement layer (using Debezium). — Marie wrote best practices for establishing complete and reliable data documentation. The post explains well the architecture you need to make it work.

Programming Language 130

Programming Language SQL PostgreSQL Data 130

Data Engineering Podcast

DECEMBER 23, 2018

Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat Your host is Tobias Macey and today I’m interviewing Usman Masood and Derek Nelson about PipelineDB, an open source continuous query engine for PostgreSQL Interview Introduction How did you get involved in the area of data management?

PostgreSQL 100

PostgreSQL Kafka Data Engineering Data Engineer 100

Data Engineering Podcast

APRIL 19, 2020

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA Support Data Engineering Podcast Summary Modern applications frequently require access to real-time data, but building and maintaining the systems that make that possible is a complex and time consuming endeavor.

Building 100

Building PostgreSQL MongoDB SQL 100

Data Engineering Podcast

JANUARY 13, 2019

How have the improvements and new features in the recent releases of PostgreSQL impacted the Timescale product? How have the improvements and new features in the recent releases of PostgreSQL impacted the Timescale product? Have you been able to leverage some of the native improvements to simplify your implementation?

Database 100

Database PostgreSQL SQL MongoDB 100

Data Engineering Podcast

FEBRUARY 11, 2018

release of PostGreSQL had on the design of the project? release of PostGreSQL had on the design of the project? Can you start by explaining what Timescale is and how the project got started? The landscape of time series databases is extensive and oftentimes difficult to navigate. What impact has the 10.0 What impact has the 10.0

PostgreSQL 100

PostgreSQL NoSQL Google Cloud MongoDB 100

Data Engineering Podcast

JANUARY 5, 2020

How has the tight coupling with Kafka impacted the direction and capabilities of Debezium? How has the tight coupling with Kafka impacted the direction and capabilities of Debezium? What are some of the other options on the market for handling change data capture? What, if any, other substrates does Debezium support (e.g.

Database 100

Database Kafka PostgreSQL MySQL 100

Towards Data Science

FEBRUARY 9, 2024

This involves getting data from an API and storing it in a PostgreSQL database. Overview Let’s break down the data pipeline process step-by-step: Data Streaming: Initially, data is streamed from the API into a Kafka topic. The data directory contains the last_processed.json file which is crucial for the Kafka streaming task.

Kafka 75

Kafka Data Engineering Data Engineer PostgreSQL 75

Data Engineering Podcast

MAY 10, 2021

Links Tinybird Carto PostgreSQL Podcast Episode PostGIS Clickhouse Podcast Episode Kafka Tornado Podcast.__init__ Links Tinybird Carto PostgreSQL Podcast Episode PostGIS Clickhouse Podcast Episode Kafka Tornado Podcast.__init__

PostgreSQL 100

PostgreSQL Data Warehouse Data Pipeline Kafka 100

Christophe Blefari

APRIL 28, 2023

AGI ( credits ) Fast News ⚡️ From PostgreSQL to Snowflake: A data migration story — The migration lasted 9 months and included 8 steps. Technologically moving from Airflow batches to Spark running on top of Kafka. This post showcases a lot of it (it uses a XGBoost model).

SQL 100

SQL Food PostgreSQL Data 100

Data Engineering Podcast

APRIL 1, 2018

I understand that your original architecture used RabbitMQ as your ingest mechanism, which you then migrated to Kafka. I understand that your original architecture used RabbitMQ as your ingest mechanism, which you then migrated to Kafka. What was your initial motivation for that change? What was your initial motivation for that change?

Amazon Web Services 100

Amazon Web Services Cloud PostgreSQL Kafka 100

Data Engineering Podcast

MAY 20, 2018

Links Starburst Data Presto Hadapt Hadoop Hive Teradata PrestoCare Cost Based Optimizer ANSI SQL Spill To Disk Tempto Benchto Geospatial Functions Cassandra Accumulo Kafka Redis PostGreSQL The intro and outro music is from The Hug by The Freak Fandango Orchestra / {CC BY-SA]([link] Support Data Engineering Podcast

PostgreSQL 100

PostgreSQL Hadoop SQL Kafka 100

Data Engineering Podcast

AUGUST 6, 2022

To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers Links Milvus Zilliz Linux Foundation/AI & Data MySQL PostgreSQL CockroachDB Pilosa Podcast Episode Pinecone Vector DB Podcast Episode Vector Embedding Reverse Image Search Vector Arithmetic Vector Distance SIGMOD Tensor Rotation Matrix (..)

Machine Learning 100

Machine Learning Database MySQL PostgreSQL 100

Data Engineering Podcast

AUGUST 6, 2017

Contact Information Email @rywalker on Twitter Links Astronomer Kiss Metrics Segment Marketing tools chart Clickstream HIPAA FERPA PCI Mesos Mesos DC/OS Airflow SSIS Marathon Prometheus Grafana Terraform Kafka Spark ELK Stack React GraphQL PostGreSQL MongoDB Ceph Druid Aries Vault Adapter Pattern Docker Kinesis API Gateway Kong AWS Lambda Flink Redshift (..)

PostgreSQL 100

PostgreSQL MongoDB Data Pipeline Kafka 100

Data Engineering Podcast

NOVEMBER 26, 2019

To help other people find the show please leave a review on iTunes and tell your friends and co-workers Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat Links Sentry Podcast.__init__

Data Warehouse 100

Data Warehouse Building PostgreSQL Kafka 100

Big Data Tools

OCTOBER 5, 2021

Kafka 3.0.0 – The Apache Software Foundation needed less than one month to go from Kafka version 3.0.0-rc0 PostgreSQL 14 – Sometimes I forget, but traditional relational databases play a big role in the lives of data engineers. And of course, PostgreSQL is one of the most popular databases. rc0 to the release of 3.0.0.

Data Engineering 52

Data Engineering Data Engineer Engineering Big Data Tools 52

Big Data Tools

OCTOBER 5, 2021

Kafka 3.0.0 – The Apache Software Foundation needed less than one month to go from Kafka version 3.0.0-rc0 PostgreSQL 14 – Sometimes I forget, but traditional relational databases play a big role in the lives of data engineers. And of course, PostgreSQL is one of the most popular databases. rc0 to the release of 3.0.0.

Data Engineering 52

Data Engineering Data Engineer Engineering Big Data Tools 52

Knowledge Hut

NOVEMBER 28, 2023

Kafka Kafka is one of the most desired open-source messaging and streaming systems that allows you to publish, distribute, and consume data streams. Kafka, which is written in Scala and Java, helps you scale your performance in today’s data-driven and disruptive enterprises.

Data Engineering 52

Data Engineering Data Engineer Engineering Generalist 52

Striim

JANUARY 4, 2023

In their example, they use Docker compose to spin up a test instance of their database, a CDC pipeline (using Debezium), Kafka, and the Confluent Schema Registry. Consider the workflow in this following diagram: You can use Striim to move data from a database (PostgreSQL) to a data warehouse (BigQuery).

PostgreSQL 52

PostgreSQL Data Warehouse Data Lake Data 52

Ascend.io

SEPTEMBER 14, 2023

Be it PostgreSQL, MySQL, MongoDB, or Cassandra, Python ensures seamless interactions. Use Case: Storing data with PostgreSQL (example) import psycopg2 conn = psycopg2.connect(dbname="mydb", Use Case: Storing data with PostgreSQL (example) import psycopg2 conn = psycopg2.connect(dbname="mydb",

Data Engineering 52

Data Engineering Data Engineer Python Engineering 52

Data Engineering Podcast

OCTOBER 14, 2019

To help other people find the show please leave a review on iTunes and tell your friends and co-workers Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat Links DataForm YCombinator DBT == Data Build Tool Podcast Episode Fishtown Analytics Typescript Continuous Integration Continuous Delivery BigQuery Snowflake DB (..)

Data Warehouse 100

Data Warehouse PostgreSQL AWS Programming Language 100

Confluent

AUGUST 11, 2020

Building a Cloud ETL Pipeline on Confluent Cloud shows you how to build and deploy a data pipeline entirely in the cloud. However, not all databases can be in the […].

Database 100

Database Kafka Data Pipeline Cloud 100

Data Engineering Podcast

DECEMBER 16, 2019

What are the benefits of using PostgreSQL as the system of record for Marquez? What are the benefits of using PostgreSQL as the system of record for Marquez? Can you explain how Marquez is architected and how the design has evolved since you first began working on it? How is the metadata itself stored and managed in Marquez?

Metadata 100

Metadata PostgreSQL Datasets Data Warehouse 100

Rockset

AUGUST 30, 2021

Apache Kafka has made acquiring real-time data more mainstream, but only a small sliver are turning batch analytics, run nightly, into real-time analytical dashboards with alerts and automatic anomaly detection. But until this release, all these data sources involved indexing the incoming raw data on a record by record basis.

SQL 52

SQL Kafka MongoDB MySQL 52

Data Engineering Podcast

OCTOBER 9, 2018

Links MemSQL NewSQL Microsoft SQL Server St. Links MemSQL NewSQL Microsoft SQL Server St.

PostgreSQL 100

PostgreSQL BI Data Warehouse Machine Learning 100

Striim

NOVEMBER 30, 2022

Some of the sources Striim supports include: Databases: Oracle, Microsoft SQL Server, MySQL, PostgreSQL, etc. With Striim, all your team needs to do is to hit a few clicks for configuration, and an automated pipeline will be created between your source and AWS targets.

AWS 52

AWS Cloud Management Google Cloud 52

Knowledge Hut

SEPTEMBER 6, 2023

Big Data Frameworks : Familiarity with popular Big Data frameworks such as Hadoop, Apache Spark, Apache Flink, or Kafka are the tools used for data processing. Implement ETL & Data Pipelines with Bash, Airflow & Kafka; architect, populate, deploy Data Warehouses; create BI reports & interactive dashboards.

Big Data 52

Big Data Certification Hadoop Scala 52

Rockset

MARCH 10, 2022

A platform such as Apache Kafka/Confluent , Spark or Amazon Kinesis for publishing that stream of event data. Traditionally, this information would be stored in transactional databases — Oracle Database , MySQL , PostgreSQL , etc. because they allow for mutability: Any field stored in these transactional databases is updatable.

Data Analytics 52

Data Analytics Data Warehouse Medical MySQL 52

Cloudera

SEPTEMBER 30, 2021

PostgreSQL 10, 11 and 12 and OracleDB 12c, 19c and 19.9. Add Kafka Service – Required for Atlas if it’s not already installed. If you are running older versions of operating systems, these will also need to be upgraded prior to the cluster upgrade commencing. Step 4b: Upgrading the RDBMS. CDP supports MariaDB 10.2-10.4,

Cloud 72

Cloud PostgreSQL Metadata MySQL 72

Silectis

JANUARY 3, 2021

Kafka Apache Kafka is the Apache Foundation’s open-source software platform for streaming. PostgreSQL A free, open-source relational database management system, also known as Postgres. Hadoop / HDFS Apache’s open-source software framework for processing big data. HDFS stands for Hadoop Distributed File System.

Data Engineering 40

Data Engineering Data Engineer Engineering Hadoop 40

Zalando Engineering

FEBRUARY 6, 2019

Nakadi is a distributed event bus that implements a RESTful API abstraction on top of Kafka-like queues. Project Highlights Lionel Montrieux brought Nakadi to FOSDEM 2019. This is one of the largest open source projects released by Zalando. It is used in production by over a hundred teams daily and handles over 100 TB of data every day.

PostgreSQL 40

PostgreSQL Kafka Project Technology 40

Rockset

OCTOBER 20, 2021

For instance, let’s say you have streaming data coming in from Kafka or Kinesis. PostgreSQL or MySQL). For high velocity data, most commonly coming from data streams, you can roll it up at write-time. This can be helpful when you want to reduce the size of large scale data streams, deduplicate data, or partition your data.

SQL 52

SQL PostgreSQL MongoDB NoSQL 52

Confluent

JANUARY 27, 2021

This a summary of Podium’s technological journey and an example of how our engineering team is tooling ourselves to scale well into the years to come. Here at Podium, we’re […].

Kafka 67

Kafka IT Technology Engineering 67

Rockset

APRIL 29, 2019

As opposed to a traditional RDBMS like PostgreSQL, DynamoDB scales horizontally, obviating the need for careful capacity planning, resharding, and database maintenance. DynamoDB has been one of the most popular NoSQL databases in the cloud since its introduction in 2012.

NoSQL 52

NoSQL PostgreSQL AWS SQL 52

Confluent

MAY 30, 2019

Following part 1 and part 2 of the Spring for Apache Kafka Deep Dive blog series, here in part 3 we will discuss another project from the Spring team: Spring Cloud Data Flow , which focuses on enabling developers to easily develop, deploy, and orchestrate event streaming pipelines based on Apache Kafka ®. The pipe symbol | (i.e.,

Kafka 94

Kafka Cloud Data Pipeline PostgreSQL 94

Zalando Engineering

MARCH 29, 2017

Take a look at our Tech Radar and you’ll see PostgreSQL, Kafka, React, and many other household-name projects listed there. Zalando depends on open source technologies to exist. Many of our projects are now co-developed by Zalandos and talented devs from RedHat, Mozilla, and more.

Scala 40

Scala PostgreSQL Retail Kafka 40

Dataquest

OCTOBER 16, 2019

Of course, a data engineer doesn’t have to know all of these, but this list illustrates just how much there is to do in the world of data engineering.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Science 52