Precisely

JANUARY 9, 2025

And the more sources of data continued to expand, moving beyond mainframes and relational databases to semi-structured and unstructured data sources spanning social feeds, device data, and many other varieties, made it impossible to manage in the same old data warehouse architectures.

Data Integration 119

Data Integration Hadoop Data Warehouse Data Lake 119

Confessions of a Data Guy

JANUARY 12, 2023

Anyone who’s been roaming around the forest of Data Engineering has probably run into many of the newish tools that have been growing rapidly around the concepts of Data Warehouses, Data Lakes, and Lake Houses … the merging of the old relational database functionality with TB and PB level cloud-based file storage systems.

Data Lake 162

Data Lake Relational Database Data Warehouse Data Engineer 162

Cloudera

DECEMBER 4, 2024

All this by making it easier for customers to connect their workloads with Snowflake, Cloudera, and unique AWS services such as Amazon Simple Storage Service (Amazon S3), Amazon Elastic Kubernetes Service (Amazon EKS) , Amazon Relational Database Service (Amazon RDS), Amazon Elastic Compute Cloud (Amazon EC2), Amazon EMR and Amazon Athena.

AWS 89

AWS Raw Data Relational Database Government 89

Data Engineering Weekly

JUNE 1, 2025

This announcement has triggered many interesting conversations about storing metadata in a relational database vs. object storage. With S3 Express One, why not metastore in Express One vs a relational database, which can reduce additional complexity?

Data Engineer 70

Data Engineer Data Engineering Engineering Relational Database 70

KDnuggets

SEPTEMBER 16, 2022

Get up to speed on SQL and relational databases with this free video course.

SQL 146

SQL Database Relational Database 146

ProjectPro

JUNE 6, 2025

You will perform just like a data engineer during the self-paced online courses and learn the fundamental skills needed to work with various tools and relational databases to develop, deploy, and manage organized and unstructured data. You will use SQL statements to query data in Relational Database Management Systems (RDBMS).

Data Engineer 58

Data Engineer Data Engineering Engineering Amazon Web Services 58

ProjectPro

JUNE 6, 2025

FAQs on Graph Databases What is a Graph Database? A graph database is a specialized database designed to efficiently store and query interconnected data. The Key Components of a Graph Database include - Nodes represent entities or objects within the data, such as a person, a place, or a product.

Database 40

Database Database-centric Relational Database MongoDB 40

ProjectPro

JUNE 6, 2025

Amazon RDS and Aurora Serverless are two relational database services provided by AWS. RDS is a fully-managed service that sets up and manages cloud-based database servers, while Aurora Serverless is a relational database engine with a more advanced deployment process that does not require manual management of database servers.

Database 40

Database PostgreSQL MySQL Relational Database 40

ProjectPro

JUNE 6, 2025

The major difference between Sqoop and Flume is that Sqoop is used for loading data from relational databases into HDFS while Flume is used to capture a stream of moving data. The data sources can refer to databases, machine data, web APIs, relational databases, flat files, log files, and RSS (RDF Site Summary) feeds, to name a few.

ETL Tools 40

ETL Tools Hadoop Relational Database Unstructured Data 40

ProjectPro

JUNE 6, 2025

They include relational databases like Amazon RDS for MySQL, PostgreSQL, and Oracle and NoSQL databases like Amazon DynamoDB. Types of AWS Databases AWS provides various database services, such as Relational Databases Non-Relational or NoSQL Databases Other Cloud Databases ( In-memory and Graph Databases).

AWS 40

AWS Database Amazon Web Services MySQL 40

ProjectPro

JUNE 6, 2025

Amazon offers top database services, such as RDS, Aurora , Redshift , DynamoDB, etc., which allow users to create relational, graph, wide-column, and other use-case-specific data models. These databases are completely managed by AWS, relieving users of time-consuming activities like server provisioning, patching, and backup.

Amazon Web Services 40

Amazon Web Services NoSQL Relational Database AWS 40

Snowflake

APRIL 2, 2025

Heres how data teams can benefit from grounding their open lakehouse architectures on Iceberg tables: Higher developer productivity: Iceberg lets developers and data engineers work as if they are using a standard relational database such as Postgres but can scale up to petabytes of data.

Data Lake 97

Data Lake Cloud Storage Metadata Data Warehouse 97

KDnuggets

SEPTEMBER 8, 2023

Understand the relevance of databases in data science. Also learn the fundamentals of relational databases, NoSQL database categories, and more.

Database 137

Database Data Science NoSQL Relational Database 137

Snowflake

JUNE 10, 2025

Updates include: Data integration: With Python DB-API Support (private preview), developers can now use Snowpark to pull data from external relational databases directly into Snowflake. With Snowpark execution, customers have seen an average 5.6x faster performance and 41% cost savings over traditional Spark. [1]

Data Pipeline 83

Data Pipeline SQL Python Building 83

Towards Data Science

JANUARY 7, 2025

Change Data Capture (CDC) is a powerful and efficient tool for transmitting data changes from relational databases such as MySQL and PostgreSQL. Change Data Capture (CDC) SystemExample Diagram (Created using Lucidchart) What is Change DataCapture?

PostgreSQL 84

PostgreSQL MySQL Data Lake Bytes 84

ProjectPro

JUNE 6, 2025

A primary key is a column or set of columns in a relational database management system table that uniquely identifies each record. List a few benefits of using surrogate keys in relational databases. Surrogate keys are a form of primary key and are present in almost all relational database tables.

Data Warehouse 53

Data Warehouse NoSQL PostgreSQL Relational Database 53

ProjectPro

JUNE 6, 2025

Comparison Of AWS Aurora With Other Databases Let us compare AWS Aurora with other databases, such as Amazon RDS, DynamoDB, etc. Amazon Aurora Vs. RDS AWS Aurora and RDS (Relational Database Service) are both cloud-based database services offered by Amazon Web Services.

Database 40

Database Technology Cloud PostgreSQL 40

ProjectPro

JUNE 6, 2025

What are the key considerations for choosing between relational databases and NoSQL databases on AWS? Choosing between relational databases and NoSQL databases on AWS involves considering various factors based on your specific use case and requirements.

Data Engineer 40

Data Engineer Data Engineering Engineering NoSQL 40

Confluent

FEBRUARY 16, 2021

One of the most common relational database systems that connects to Apache Kafka® is Oracle, which often holds highly critical enterprise transaction workloads. While Oracle Database (DB) excels at many […].

Relational Database 128

Relational Database Kafka Database Systems 128

Start Data Engineering

JULY 18, 2020

RDS AWS RDS is a managed service provided by AWS to run a relational database. Go to Services -> RDS Click on Create Database, In the Create Database prompt, choose Standard Create option with PostgreSQL as engine type. We will see how to setup a postgres instance using AWS RDS. Log in to your AWS account.

PostgreSQL 130

PostgreSQL AWS Relational Database Database 130

Confluent

NOVEMBER 2, 2021

What will the next important category of databases look like? For decades, relational databases were the undisputed home of data. They powered everything: from websites to analytics, from customer data […].

Database 121

Database Systems Relational Database Data 121

ProjectPro

JUNE 6, 2025

These sources can have various formats, including written documents, spreadsheets, CSV files, relational databases like Oracle, MySQL, and SQL Server, non-relational databases, and so forth. Extract The extract step of the ETL process entails extracting data from one or more sources.

Project 40

Project Non-relational Database Business Analyst Relational Database 40

Hevo

MARCH 9, 2025

Relational databases like Postgres have been the backbone of enterprise data management for years. However, as data volumes grow and the need for flexibility, scalability, and advanced analytics increases, modern solutions like Apache Iceberg are becoming essential.

Relational Database 52

Relational Database Architecture Data Database 52

ProjectPro

JUNE 6, 2025

Amazon DynamoDB Amazon DynamoDB provides an alternative to relational database systems by using several data types, such as document, graph, key-value, memory, and search. It allows engineers to define dependencies, establish cluster setup, and identify the underlying EC2 instances.

AWS 52

AWS Data Engineer Data Engineering Engineering 52

ProjectPro

JUNE 6, 2025

What features of Microsoft Azure SQL database give it an edge over its competitors? To get answers to all these questions, read our ultimate guide on Azure SQL Database! Did you know that 85% of Fortune 500 companies use Azure SQL database as their relational database service?

Database 58

Database SQL Cloud Data Management 58

Confluent

AUGUST 20, 2019

Traditional relational database systems are ubiquitous in software systems. They are surrounded by a strong ecosystem of tools, such as object-relational mappers and schema migration helpers. A tomicity in relational databases ensures that a transaction either succeeds or fails as a whole.

Kafka 22

Kafka Systems Building Relational Database 22

ProjectPro

JUNE 6, 2025

Depending on the type of database a data engineer is working with, they will use specific software. Below, we mention a few popular databases and the different softwares used for them.

Data Engineer 94

Data Engineer Data Engineering Engineering Amazon Web Services 94

ProjectPro

JUNE 6, 2025

In-memory databases like Redis and Memcached. Relational databases like MySQL and PostgreSQL. Relational database workloads. High-performance databases benefit from the memory optimization of X1e and X1 instances. Workloads requiring a lot of memory, such as real-time analytics and caching servers.

PostgreSQL 40

PostgreSQL MySQL AWS Relational Database 40

Confluent

FEBRUARY 23, 2023

How to use Kafka Streams to aggregate change data capture (CDC) messages from a relational database into transactional messages, powering a scalable microservices architecture.

Kafka 110

Kafka Relational Database Architecture Database 110

Hevo

DECEMBER 20, 2024

PostgreSQL is one of the most popular open-source choices for relational databases. It is loved by engineers for its powerful features, flexibility, efficient data retrieval mechanism, and on top of all its overall performance. However, performance issues can be encountered with the growth in the size of data and complexity of queries.

PostgreSQL 40

PostgreSQL Coding Relational Database Database 40

ProjectPro

JUNE 6, 2025

Common data sources include spreadsheets, databases, JSON data from APIs, Log files, and CSV files. Common destinations include relational databases, analytical data warehouses, or data lakes. Destination refers to a landing area where the data is taken to. Data ingestion refers to moving data from sources to the destination.

Data Ingestion 45

Data Ingestion Data Engineer Data Engineering Project 45

Team Data Science

JANUARY 8, 2021

Data engineering function involve the fundamental understanding of data utilization skills such as coding, python, SQL database, relational database, AWS in the field of big data. It would even be an additional benefit for them to have expertise in computer networking as well.

Data Engineer 130

Data Engineer Data Engineering Engineering Unstructured Data 130

ProjectPro

JUNE 6, 2025

The best part is- you won't merely learn; you will apply your knowledge through hands-on projects and lab simulations, gaining practical expertise with Python libraries , Bash, Apache Spark, ETL tools , and Relational Database Management Systems (RDBMS). Duration The duration of this self-paced course will be four months.

Data Engineer 45

Data Engineer Data Engineering Cloud Engineering 45

ProjectPro

JUNE 6, 2025

You can also access data through non-relational databases such as Apache Cassandra, Apache HBase , Apache Hive, and others like the Hadoop Distributed File System. Presto allows you to query data stored in Hive, Cassandra, relational databases, and even bespoke data storage. However, Trino is not limited to HDFS access.

Big Data 49

Big Data Project Metadata Programming Language 49

ProjectPro

JUNE 6, 2025

This serverless data integration service can automatically and quickly discover structured or unstructured enterprise data when stored in data lakes in Amazon S3, data warehouses in Amazon Redshift, and other databases that are a component of the Amazon Relational Database Service.

AWS 66

AWS Scala Metadata Data Lake 66

ProjectPro

JUNE 6, 2025

Big data operations require specialized tools and techniques since a relational database cannot manage such a large amount of data. This database provides more flexible data storage and retrieval than typical relational databases. SQL MySQL SQL is a relational database. SQL databases scale vertically.

Big Data 40

Big Data Hadoop Relational Database AWS 40

Snowflake

NOVEMBER 11, 2024

SnowConvert is an easy-to-use code conversion tool that accelerates legacy relational database management system (RDBMS) migrations to Snowflake. In addition to free assessments and free table conversions, SnowConvert now supports accurate conversion of database views from Teradata, Oracle or SQL Server for free.

Data Architecture 105

Data Architecture Architecture Data Lake Kafka 105