Christophe Blefari

JUNE 16, 2023

The first advice is about the documentation readers: data team, business users or other stakeholders. Change Data Capture (CDC) with PostgreSQL and ClickHouse — This is a nice vendor post about CDC with Kafka as movement layer (using Debezium). also as a data observability platform, but done differently.

Programming Language 130

Programming Language SQL PostgreSQL Data 130

Christophe Blefari

JANUARY 20, 2024

formats — This is a huge part of data engineering. Picking the right format for your data storage. In this category I recommend also to have a look at data ingestion (Airbyte, Fivetran, etc.), Main technologies around stream are bus messages like Kafka and processing framework like Flink or Spark on top of the bus.

Data Engineering 130

Data Engineering Data Engineer Engineering Hadoop 130

Data Engineering Podcast

DECEMBER 16, 2019

The solution to discoverability and tracking of data lineage is to incorporate a metadata repository into your data platform. The metadata repository serves as a data catalog and a means of reporting on the health and status of your datasets when it is properly integrated into the rest of your tools.

Metadata 100

Metadata PostgreSQL Datasets Data Warehouse 100

Pinterest Engineering

NOVEMBER 22, 2023

Initial Architecture For Goku Short Term Ingestion Figure 1: Old push based ingestion pipeline into GokuS At Pinterest, we have a sidecar metrics agent running on every host that logs the application system metrics time series data points (metric name, tag value pairs, timestamp and value) into dedicated kafka topics.

Database 95

Database Bytes Kafka Architecture 95

Data Engineering Podcast

SEPTEMBER 12, 2021

That leaves DataOps reactive to data quality issues and can make your consumers lose confidence in your data. By connecting to your pipeline orchestrator like Apache Airflow and centralizing your end-to-end metadata, Databand.ai lets you identify data quality issues and their root causes from a single dashboard.

Database 100

Database Metadata Kafka Data Storage 100

Knowledge Hut

NOVEMBER 28, 2023

Concepts of IaaS, PaaS, and SaaS are the trend, and big companies expect data engineers to have the relevant knowledge. Kafka Kafka is one of the most desired open-source messaging and streaming systems that allows you to publish, distribute, and consume data streams. ETL is central to getting your data where you need it.

Data Engineering 52

Data Engineering Data Engineer Engineering Generalist 52

AltexSoft

AUGUST 29, 2023

In 2010, a transformative concept took root in the realm of data storage and analytics — a data lake. The term was coined by James Dixon , Back-End Java, Data, and Business Intelligence Engineer, and it started a new era in how organizations could store, manage, and analyze their data.

Data Lake 52

Data Lake Architecture IT Amazon Web Services 52

Big Data Tools

SEPTEMBER 6, 2021

It hasn’t had its first release yet, but the promise is that it will un-bias your data for you! rc0 – If you like to try new releases of popular products, the time has come to test Kafka 3 and report any issues you find on your staging environment! Reading file metadata is costly because it is an IO operation, which is slow.

Data Engineering 52

Data Engineering Data Engineer Engineering Big Data Tools 52

Rockset

APRIL 18, 2023

Under the hood, Rockset utilizes its Converged Index technology, which is optimized for metadata filtering, vector search and keyword search, supporting sub-second search, aggregations and joins at scale. Feature Generation: Transform and aggregate data during the ingest process to generate complex features and reduce data storage volumes.

Unstructured Data 105

Unstructured Data Metadata Machine Learning SQL 105

Big Data Tools

SEPTEMBER 6, 2021

It hasn’t had its first release yet, but the promise is that it will un-bias your data for you! rc0 – If you like to try new releases of popular products, the time has come to test Kafka 3 and report any issues you find on your staging environment! Reading file metadata is costly because it is an IO operation, which is slow.

Data Engineering 40

Data Engineering Data Engineer Engineering Big Data Tools 40

Monte Carlo

APRIL 24, 2023

Data lakes are useful, flexible data storage repositories that enable many types of data to be stored in its rawest state. Traditionally, after being stored in a data lake, raw data was then often moved to various destinations like a data warehouse for further processing, analysis, and consumption.

Data Lake 52

Data Lake Google Cloud Data Warehouse AWS 52

AltexSoft

JUNE 7, 2021

Master Nodes control and coordinate two key functions of Hadoop: data storage and parallel processing of data. Worker or Slave Nodes are the majority of nodes used to store data and run computations according to instructions from a master node. and keeps track of storage capacity, a volume of data being transferred, etc.

Big Data Tools 98

Big Data Tools Hadoop Big Data Database-centric 98

AltexSoft

JULY 29, 2022

No matter the actual size, each cluster accommodates three functional layers — Hadoop distributed file systems for data storage, Hadoop MapReduce for processing, and Hadoop Yarn for resource management. Yet, its pool of supporters definitely stands out if compared with other Big Data platforms. Let’s see why.

Hadoop 59

Hadoop Big Data Google Cloud NoSQL 59

ProjectPro

NOVEMBER 15, 2021

It was built from the ground up for interactive analytics and can scale to the size of Facebook while approaching the speed of commercial data warehouses. Presto allows you to query data stored in Hive, Cassandra, relational databases, and even bespoke data storage. CMAK is developed to help the Kafka community.

Big Data 52

Big Data Project Metadata Programming Language 52

AltexSoft

SEPTEMBER 23, 2021

Data integration layer holds any transformations required to make the data digestible for end users. This often involves such operations as data harmonization, mastering, and enrichment with metadata. Storage layer corresponds to the needs of database management and data modeling. Azure Data Factory.

Architecture 52

Architecture Data Lake Unstructured Data Data Warehouse 52

AltexSoft

NOVEMBER 10, 2021

In a nutshell, the lakehouse system leverages low-cost storage to keep large volumes of data in its raw formats just like data lakes. At the same time, it brings structure to data and empowers data management features similar to those in data warehouses by implementing the metadata layer on top of the store.

Architecture 52

Architecture Data Lake Data Warehouse Metadata 52

Silectis

JANUARY 3, 2021

Data Architecture Data architecture is a composition of models, rules, and standards for all data systems and interactions between them. Data Catalog An organized inventory of data assets relying on metadata to help with data management. HDFS stands for Hadoop Distributed File System.

Data Engineering 40

Data Engineering Data Engineer Engineering Hadoop 40

ProjectPro

SEPTEMBER 16, 2021

NoSQL databases are the new-age solutions to distributed unstructured data storage and processing. The speed, scalability, and fail-over safety offered by NoSQL databases are needed in the current times in the wake of Big Data Analytics and Data Science technologies. Hence, writes in Hbase are operation intensive.

NoSQL 52

NoSQL Database Hadoop Big Data 52

AltexSoft

JULY 18, 2023

This module can ingest live data streams from multiple sources, including Apache Kafka , Apache Flume , Amazon Kinesis , or Twitter, splitting them into discrete micro-batches. Netflix leverages Spark Streaming and Kafka for near real-time movie recommendations. Dependence on external storage systems: A weakness?

Big Data 52

Big Data Data Process Process Hadoop 52

phData: Data Engineering

AUGUST 4, 2023

Data storage is a vital aspect of any Snowflake Data Cloud database. Within Snowflake, data can either be stored locally or accessed from other cloud storage systems. Snowflake hides user data objects and makes them accessible only through SQL queries through the compute layer.

Cloud Storage 52

Cloud Storage Google Cloud Amazon Web Services Data Storage 52

AltexSoft

MARCH 14, 2023

Additionally, this modularity can help prevent vendor lock-in, giving organizations more flexibility and control over their data stack. Many components of a modern data stack (such as Apache Airflow, Kafka, Spark, and others) are open-source and free. But this distinction has been blurred with the era of cloud data warehouses.

IT 59

IT Data Warehouse Data Governance Data Lake 59

Monte Carlo

JANUARY 12, 2024

The company has refactored their Kafka connector and made it so that when data lands in Snowflake it is queryable immediately resulting in a 10x lower latency. On the data lake side, Databricks has launched Unity Catalog to help bring more metadata, structure, and governance to data assets.

Big Data 59

Big Data Data Analytics Data Science Data Lake 59

Monte Carlo

AUGUST 19, 2023

We’ll cover: What is a data platform? Below, we share what the “basic” data platform looks like and list some hot tools in each space (you’re likely using several of them): The modern data platform is composed of five critical foundation layers. Data Storage and Processing The first layer?

Building 52

Building BI Data Lake Data Governance 52

AltexSoft

SEPTEMBER 21, 2023

In the hospitality industry context, a single document could represent one hotel room’s data, including attributes like room number, type, price, amenities, and availability status. Each document has unique metadata fields like index , type , and id that help identify its storage location and nature.

Engineering 52

Engineering NoSQL Programming Language Java 52

Monte Carlo

JUNE 14, 2023

In this post, we will help you quickly level up your overall knowledge of data pipeline architecture by reviewing: Table of Contents What is data pipeline architecture? Why is data pipeline architecture important? This is frequently referred to as a 5 or 7 layer (depending on who you ask) data stack like in the image below.

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

Confluent

SEPTEMBER 20, 2019

As a distributed system for collecting, storing, and processing data at scale, Apache Kafka ® comes with its own deployment complexities. To simplify all of this, different providers have emerged to offer Apache Kafka as a managed service. Before Confluent Cloud was announced , a managed service for Apache Kafka did not exist.

Kafka 21

Kafka Management Cloud AWS 21

AltexSoft

OCTOBER 30, 2021

Data engineer’s integral task is building and maintaining data infrastructure — the system managing the flow of data from its source to destination. This typically includes setting up two processes: an ETL pipeline , which moves data, and a data storage (typically, a data warehouse ), where it’s kept.

Data Engineering 98

Data Engineering Data Engineer Engineering Machine Learning 98

Snowflake

MARCH 30, 2023

Use Snowflake’s native Kafka Connector to configure Kafka topics into Snowflake tables. Snowflake can also ingest external tables from on-premise s data sources via S3-compliant data storage APIs.

Engineering 83

Engineering Raw Data Data Science Scala 83

ProjectPro

NOVEMBER 22, 2021

StructType is a collection of StructField objects that determines column name, column data type, field nullability, and metadata. To define the columns, PySpark offers the pyspark.sql.types import StructField class, which has the column name (String), column type (DataType), nullable column (Boolean), and metadata (MetaData).

Hadoop 52

Hadoop Python Datasets Metadata 52

ProjectPro

AUGUST 11, 2021

The data warehouse layer consists of the relational database management system (RDBMS) that contains the cleaned data and the metadata, which is data about the data. The RDBMS can either be directly accessed from the data warehouse layer or stored in data marts designed for specific enterprise departments.

Data Lake 52

Data Lake Data Warehouse Cloud Hadoop 52

ProjectPro

NOVEMBER 30, 2021

Flat Files: CSV, TXT, and Excel spreadsheets are standard text file formats for storing data. Nontechnical users can easily access these data formats without installing data science software. SQL RDBMS: The SQL database is a trendy data storage where we can load our processed data.

Process 52

Process Data Pipeline Data Warehouse AWS 52

ProjectPro

MARCH 23, 2015

Get FREE Access to Data Analytics Example Codes for Data Cleaning, Data Munging, and Data Visualization The main motive of the Hadoop 2.0 Earlier there was one Hadoop NameNode for maintaining the tree hierarchy of the HDFS files and tracking the data storage in the cluster. With Hadoop 2.0, With Hadoop 2.0,

Hadoop 40

Hadoop Big Data Architecture Metadata 40

ProjectPro

SEPTEMBER 26, 2021

Becoming a Big Data Engineer - The Next Steps Big Data Engineer - The Market Demand An organization’s data science capabilities require data warehousing and mining, modeling, data infrastructure, and metadata management. Most of these are performed by Data Engineers.

Big Data 52

Big Data Data Engineering Data Engineer Engineering 52

ProjectPro

MAY 30, 2016

It stores the application data and file system metadata separately. Application data is stored on severs known as DataNodes and file system metadata is stored on dedicated servers called NameNodes. HDFS uses a master slave architecture.

Hadoop 40

Hadoop Metadata Big Data Portfolio 40

ProjectPro

NOVEMBER 7, 2016

A Deep Dive Into The Hadoop Architecture - HDFS, Yarn, and MapReduce Hadoop follows a master-slave architecture design for data storage and distributed processing using HDFS and MapReduce. The slave nodes in the Hadoop physical architecture are the other machines in the Hadoop cluster that store data and perform complex computations.

Hadoop 40

Hadoop Architecture IT Big Data 40

ProjectPro

DECEMBER 10, 2021

The service provider's data center hosts the underlying infrastructure, software, and app data. Azure Redis Cache is an in-memory data storage, or cache system, based on Redis that boosts the flexibility and efficiency of applications that rely significantly on backend data stores. Explain Azure Redis Cache.

BI 52

BI Cloud Computing SQL Database 52

ProjectPro

JULY 30, 2021

There are many cloud computing job roles like Cloud Consultant, Cloud reliability engineer, cloud security engineer, cloud infrastructure engineer, cloud architect, data science engineer that one can make a career transition to. PaaS packages the platform for development and testing along with data, storage, and computing capability.

Cloud Computing 40

Cloud Computing Cloud Amazon Web Services AWS 40