Data Pipeline ETL Tools Events Metadata 
Towards Data Science
APRIL 6, 2023
Today’s post follows the same philosophy: fitting local and cloud pieces together to build a data pipeline. And, when it comes to data engineering solutions, it’s no different: They have databases, ETL tools, streaming platforms, and so on — a set of tools that makes our life easier (as long as you pay for them).
ProjectPro
FEBRUARY 16, 2023
The process of data extraction from source systems, processing it for data transformation, and then putting it into a target data system is known as ETL, or Extract, Transform, and Load. ETL has typically been carried out utilizing data warehouses and on-premise ETL tools.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
AltexSoft
AUGUST 29, 2023
Instead of relying on traditional hierarchical structures and predefined schemas, as in the case of data warehouses, a data lake utilizes a flat architecture. This structure is made efficient by data engineering practices that include object storage. Watch our video explaining how data engineering works.
dbt Developer Hub
NOVEMBER 28, 2021
In my days as a data consultant and now as a member of the dbt Labs Solutions Architecture team, I’ve frequently seen Airflow, dbt Core & dbt Cloud ( via the official provider ) blended as needed, based on the needs of a specific data pipeline, or a team’s structure and skillset. Let’s export log events into BigQuery. “I
dbt Developer Hub
NOVEMBER 3, 2022
Let’s discuss how to convert events from an event-driven microservice architecture into relational tables in a warehouse like Snowflake. We use Snowflake as our data warehouse where we build dashboards both for internal use and for customers. This data would become the main dbt sources used by our report models in BI.
ProjectPro
OCTOBER 28, 2015
Some of the common challenges with data ingestion in Hadoop are parallel processing, data quality, machine data on a higher scale of several gigabytes per minute, multiple source ingestion, real-time ingestion and scalability. Need for Apache Sqoop How Apache Sqoop works? Need for Flume How Apache Flume works?
AltexSoft
OCTOBER 21, 2022
The technology was written in Java and Scala in LinkedIn to solve the internal problem of managing continuous data flows. This scenario involves three main characters — publishers, subscribers, and a message or event broker. A subscriber is a receiving program such as an end-user app or business intelligence tool.
Expert insights. Personalized for you.
79 
Let's personalize your content