Scribd Technology

JANUARY 14, 2025

In a recent session with the Delta Lake project I was able to share the work led Kuntal Basu and a number of other people to dramatically improve the efficiency and reliability of our online data ingestion pipeline. as they take you behind the scenes of Scribds data ingestion setup.

Data Ingestion 40

Data Ingestion AWS Cloud Architecture 40

Cloudyard

JUNE 6, 2023

Snowflake Output Happy 0 0 % Sad 0 0 % Excited 0 0 % Sleepy 0 0 % Angry 0 0 % Surprise 0 0 % The post Data Ingestion with Glue and Snowpark appeared first on Cloudyard. Technical Implementation: GLUE Job.

Data Ingestion 94

Data Ingestion AWS Data Database 94

Hevo

FEBRUARY 23, 2025

Organizations generate tons of data every second, yet 80% of enterprise data remains unstructured and unleveraged (Unstructured Data). Organizations need data ingestion and integration to realize the complete value of their data assets.

Data Ingestion 40

Data Ingestion Data Integration Unstructured Data Raw Data 40

Hevo

FEBRUARY 23, 2025

Organizations generate tons of data every second, yet 80% of enterprise data remains unstructured and unleveraged (Unstructured Data). Organizations need data ingestion and integration to realize the complete value of their data assets.

Data Ingestion 40

Data Ingestion Data Integration Unstructured Data Raw Data 40

databricks

DECEMBER 10, 2024

Data engineering teams are frequently tasked with building bespoke ingestion solutions for myriad custom, proprietary, or industry-specific data sources. Many teams find that.

Data Ingestion 105

Data Ingestion Python Data Data Engineering 105

Analytics Vidhya

MARCH 7, 2023

Introduction Apache Flume is a tool/service/data ingestion mechanism for gathering, aggregating, and delivering huge amounts of streaming data from diverse sources, such as log files, events, and so on, to centralized data storage. Flume is a tool that is very dependable, distributed, and customizable.

Data Ingestion 223

Data Ingestion Data Storage Hadoop Data 223

KDnuggets

SEPTEMBER 11, 2024

Learn how to create a data science pipeline with a complete structure.

Data Science 105

Data Science Data Ingestion Process Data 105

Confluent

JANUARY 22, 2024

The new fully managed BigQuery Sink V2 connector for Confluent Cloud offers streamlined data ingestion and cost-efficiency. Learn about the Google-recommended Storage Write API and OAuth 2.0 support.

Data Ingestion 78

Data Ingestion Cloud Management Data 78

databricks

MARCH 29, 2024

Overview In the competitive world of professional hockey, NHL teams are always seeking to optimize their performance. Advanced analytics has become increasingly important.

Data Ingestion 111

Data Ingestion Management Data Entertainment 111

Snowflake

MARCH 2, 2023

This solution is both scalable and reliable, as we have been able to effortlessly ingest upwards of 1GB/s throughput.” Rather than streaming data from source into cloud object stores then copying it to Snowflake, data is ingested directly into a Snowflake table to reduce architectural complexity and reduce end-to-end latency.

Kafka 137

Kafka Data Ingestion Data Pipeline Cloud Storage 137

Analytics Vidhya

FEBRUARY 20, 2023

Introduction Azure data factory (ADF) is a cloud-based data ingestion and ETL (Extract, Transform, Load) tool. The data-driven workflow in ADF orchestrates and automates data movement and data transformation.

Data Ingestion 188

Data Ingestion Data Cloud Cloud Computing 188

DataKitchen

NOVEMBER 5, 2024

You have typical data ingestion layer challenges in the bronze layer: lack of sufficient rows, delays, changes in schema, or more detailed structural/quality problems in the data. Data missing or incomplete at various stages is another critical quality issue in the Medallion architecture.

Architecture 78

Architecture Raw Data Pipeline-centric Data Ingestion 78

DataKitchen

JUNE 6, 2025

Every data team member needs to interact with data quality testing, but in different ways depending on their responsibilities and expertise: Data Ingestion Teams: Data ingestion specialists use quality tests to identify errors in source data before it propagates downstream.

Data Ingestion 73

Data Ingestion Data Governance Data Government 73

Cloudera

DECEMBER 9, 2024

Image: The Smart Logistics Assistant demonstrates how Cloudera AI Inference and solution pattern can streamline operations with real-time data, enhancing decision-making and efficiency. ReadyFlows(NiFi) : Clouderas ReadyFlows are pre-designed data pipelines for various use cases, reducing complexity in data ingestion and transformation.

Professional Services 74

Professional Services Data Ingestion Manufacturing Retail 74

Rockset

MARCH 1, 2023

When you deconstruct the core database architecture, deep in the heart of it you will find a single component that is performing two distinct competing functions: real-time data ingestion and query serving. When data ingestion has a flash flood moment, your queries will slow down or time out making your application flaky.

Data Ingestion 144

Data Ingestion Database Cloud Storage Architecture 144

Cloudera

DECEMBER 4, 2024

For more than a decade, Cloudera has been an ardent supporter and committee member of Apache NiFi, long recognizing its power and versatility for data ingestion, transformation, and delivery.

Data Pipeline 93

Data Pipeline Data Ingestion Data Preparation Architecture 93

Snowflake

NOVEMBER 12, 2024

Siemens leverages Hybrid Tables in its Data Ingestion Engine to overcome concurrency challenges and improve data quality and consistency for its critical ERP replication process,” says Henrique Dias, Service Manager and Data Architect at Siemens AG.

Food 95

Food Metadata Education AWS 95

KDnuggets

JULY 29, 2024

Learn to build the end-to-end data science pipelines from data ingestion to data visualization using Pandas pipe method.

Data Science 136

Data Science Building Data Ingestion Data 136

Snowflake

JUNE 13, 2024

But at Snowflake, we’re committed to making the first step the easiest — with seamless, cost-effective data ingestion to help bring your workloads into the AI Data Cloud with ease. Like any first step, data ingestion is a critical foundational block. Ingestion with Snowflake should feel like a breeze.

Data Ingestion 113

Data Ingestion MySQL PostgreSQL Data Pipeline 113

Data Engineering Weekly

MARCH 5, 2025

While the Iceberg itself simplifies some aspects of data management, the surrounding ecosystem introduces new challenges: Small File Problem (Revisited): Like Hadoop, Iceberg can suffer from small file problems. Data ingestion tools often create numerous small files, which can degrade performance during query execution.

Hadoop 59

Hadoop Metadata Data Ingestion Data Governance 59

DataKitchen

JULY 8, 2025

And while each individual team may do their part, the usability of the data as a whole breaks down. Let’s break the problem down through the DataOps lens: Data ingestion teams often encounter inconsistent and poorly labeled inputs. Data engineers stitch it together but don’t always check for business logic failures.

Data Ingestion 64

Data Ingestion Datasets Data Data Engineering 64

Cloudyard

NOVEMBER 12, 2024

This approach not only minimizes costs but also maximizes efficiency by performing essential operations only when new data is available. This use case will walk through the setup of a Snowflake task called LOAD_ORDER_DATA, which performs automated data ingestion and validation.

Data Ingestion 52

Data Ingestion Cloud Process Building 52

ProjectPro

JUNE 6, 2025

1) Build an Uber Data Analytics Dashboard This data engineering project idea revolves around analyzing Uber ride data to visualize trends and generate actionable insights. Project Idea : Build a data pipeline to ingest data from APIs like CoinGecko or Kaggle’s crypto datasets.

Data Engineering 59

Data Engineering Data Engineer Project Engineering 59

Data Engineering Weekly

JUNE 30, 2025

The Data Platform Fundamentals Guide Learn the fundamental concepts to build a data platform in your organization.

Data Engineering 52

Data Engineering Data Engineer Engineering Kafka 52

Data Engineering Weekly

JUNE 1, 2025

[link] Sponsored: The Data Platform Fundamentals Guide Learn the fundamental concepts to build a data platform in your organization. The architecture highlights one industry challenge: We still need a specialized OLAP engine like Clickhouse for real-time query capabilities and the typical data ingestion flowing to Lakehouses.

Data Engineering 72

Data Engineering Data Engineer Engineering Relational Database 72

Data Engineering Weekly

JANUARY 8, 2025

Apache Hudi's unique differentiators, such as its ability to handle complex data operations asynchronously, set it apart. For example, Hudi excels in scenarios requiring large-scale data ingestion with transactional guarantees, a feature critical for the finance, healthcare, and retail industries.

Data Lake 64

Data Lake Datasets Retail Data Ingestion 64

KDnuggets

SEPTEMBER 1, 2023

This article describes a large-scale data warehousing use case to provide reference for data engineers who are looking for log analytic solutions. It introduces the log processing architecture and real-case practice in data ingestion, storage, and queries.

Data Ingestion 126

Data Ingestion Data Engineering Data Engineer Architecture 126

Pinterest Engineering

APRIL 7, 2025

In the case during the instance migration, even though the measured network throughput was well below the baseline bandwidth, we still see TCP retransmits to spike during bulk data ingestion into EC2. In the database service, the application reads data (e.g.

AWS 68

AWS Bytes Data Ingestion Database 68

Cloudyard

DECEMBER 18, 2024

Handling feed files in data pipelines is a critical task for many organizations. These files, often stored in stages such as Amazon S3 or Snowflake internal stages, are the backbone of data ingestion workflows. Without a proper archival strategy, these files can clutter staging areas, leading to operational challenges.

Retail 52

Retail Data Ingestion AWS Data Pipeline 52

ProjectPro

JUNE 6, 2025

Here’s the breakdown of the core layers - Data Ingestion: The ingestion layer handles transferring data from various sources into the data lake. It supports batch processing for large amounts of data and real-time streaming for continuous data. into Azure Data Lake Storage.

Data Lake 40

Data Lake Building Hadoop Raw Data 40

Snowflake

MARCH 31, 2025

A set of CPU- and GPU-specific images, pre-installed with the latest and most popular libraries and frameworks (PyTorch, XGBoost, LightGBM, scikit-learn and many more ) supporting ML development, so data scientists can simply spin up a Snowflake Notebook and dive right into their work.

Healthcare 93

Healthcare Medical Government Electronics 93

ProjectPro

JUNE 6, 2025

Apache NiFi Apache NiFi is a commonly used open-source data integration tool for data routing, transformation, and system mediation. NiFi's user-friendly interface allows users to design complex data flows effortlessly, making it an excellent choice for data ingestion and routing tasks.

Data Pipeline 40

Data Pipeline Building Retail Data Ingestion 40

Hevo

JUNE 13, 2025

You are probably here because you are either using Apache NiFi for your data integration workflows or just exploring it as an option because of its visual interface, open-source nature, and flow-based design. It is a solid tool for many data ingestion tasks, especially if you’re working with on-premise systems or batch-based pipelines.

Data Ingestion 40

Data Ingestion Data Integration Designing Systems 40

Snowflake

APRIL 9, 2025

SoFlo Solar SoFlo Solars SolarSync platform uses real-time AI data analytics and ML to transform underperforming residential solar systems into high-uptime clean energy assets, providing homeowners with savings while creating a virtual power plant network that delivers measurable value to utilities and grid operators.

Pharmaceutical 98

Pharmaceutical Manufacturing Data Ingestion SQL 98

KDnuggets

APRIL 29, 2022

Top-rated data science tracks consist of multiple project-based courses covering all aspects of data. It includes an introduction to Python/R, data ingestion & manipulation, data visualization, machine learning, and reporting.

Data Science 111

Data Science Data Ingestion Machine Learning Python 111

Data Engineering Podcast

NOVEMBER 20, 2022

report having current investments in automation, 85% of data teams plan on investing in automation in the next 12 months. The Ascend Data Automation Cloud provides a unified platform for data ingestion, transformation, orchestration, and observability. In fact, while only 3.5% That’s where our friends at Ascend.io

Data Lake 130

Data Lake MongoDB Data Ingestion MySQL 130

Snowflake

OCTOBER 3, 2023

We are excited to announce the availability of data pipelines replication, which is now in public preview. In the event of an outage, this powerful new capability lets you easily replicate and failover your entire data ingestion and transformations pipelines in Snowflake with minimal downtime.

Data Pipeline 112

Data Pipeline Management Data Ingestion Data 112