Towards Data Science

FEBRUARY 6, 2024

This generalisation makes their data models complex and cryptic and require domain expertise. Even harder to manage, a common setup within large organisations is to have several instances of these systems with some underlaying processes in charge of transmitting data among them, which could lead to duplications, inconsistencies, and opacity.

Systems 76

Systems Raw Data Metadata Data Cleanse 76

Christophe Blefari

APRIL 21, 2023

A lot of data teams embraced dbt, or at least the SQL with engineering practices to transform data in cloud data warehouses. It is interesting to read this post jointly with the future of data engineer at Meta. Data Economy 💰 Betterdata raises $1.65m seed round. Synthetic data are AI generated data.

Raw Data 130

Raw Data Data Datasets SQL 130

Hevo

FEBRUARY 21, 2023

As data volumes continue to grow, organizations seek ways to make sense of it all, and data warehouses are at the center. BigQuery is a popular cloud-based data warehouse that allows for powerful analytics and querying at scale. This is […]

Raw Data 52

Raw Data Data Warehouse Data Cloud 52

Ascend.io

JANUARY 2, 2024

Getting your hands on the right data at the right time is the lifeblood of any forward-thinking company. But let’s be honest, creating effective, robust, and reliable data pipelines, the ones that feed your company’s reporting and analytics, is no walk in the park. What Is a Data Pipeline? But our journey doesn’t end there.

Data Pipeline 52

Data Pipeline Building Raw Data Data Warehouse 52

Ascend.io

NOVEMBER 21, 2023

The emergence of cloud data warehouses, offering scalable and cost-effective data storage and processing capabilities, initiated a pivotal shift in data management methodologies. This article revisits the foundational elements of ELT, exploring what it is, how it reshaped data strategies, and how it works.

Raw Data 52

Raw Data Data Warehouse Data Cleanse Data Integration 52

Data Engineering Podcast

DECEMBER 11, 2021

In order to make it easier for developers to build customer profiles in a way that respects their privacy Serge Huber helped to create the Apache Unomi framework as an open source customer data platform. Missing data? Start trusting your data with Monte Carlo today! Struggling with broken pipelines? Stale dashboards?

Data Warehouse 100

Data Warehouse Raw Data Data Lake BI 100

Cloudera

JANUARY 20, 2021

Most of what is written though has to do with the enabling technology platforms (cloud or edge or point solutions like data warehouses) or use cases that are driving these benefits (predictive analytics applied to preventive maintenance, financial institution’s fraud detection, or predictive health monitoring as examples) not the underlying data.

Manufacturing 111

Manufacturing Data Warehouse Kafka Retail 111

Cloud Academy

JUNE 7, 2022

How Do We Transform and Model Data at Cloud Academy? “Data is the new gold”: a common phrase over the last few years. For all organizations, data and information have become crucial to making good decisions for the future and having a clear understanding of how they’re making progress — or otherwise.

Cloud 52

Cloud Data Warehouse Raw Data Business Intelligence 52

DataKitchen

AUGUST 19, 2021

Below is our fourth post (4 of 5) on combining data mesh with DataOps to foster innovation while addressing the challenges of a decentralized architecture. We’ve covered the basic ideas behind data mesh and some of the difficulties that must be managed. Below is a discussion of a data mesh implementation in the pharmaceutical space.

Pharmaceutical 98

Pharmaceutical Data Lake Data Warehouse Raw Data 98

The Modern Data Company

MAY 10, 2023

In our previous post, The Pros and Cons of Leading Data Management and Storage Solutions , we untangled the differences among data lakes, data warehouses, data lakehouses, data hubs, and data operating systems. What factors are most important when building a data management ecosystem?

Data Management 52

Data Management Data Lake Management Data Warehouse 52

The Modern Data Company

MAY 10, 2023

In our previous post, The Pros and Cons of Leading Data Management and Storage Solutions , we untangled the differences among data lakes, data warehouses, data lakehouses, data hubs, and data operating systems. What factors are most important when building a data management ecosystem?

Data Management 52

Data Management Data Lake Management Data Warehouse 52

The Modern Data Company

MAY 10, 2023

In our previous post, The Pros and Cons of Leading Data Management and Storage Solutions , we untangled the differences among data lakes, data warehouses, data lakehouses, data hubs, and data operating systems. What factors are most important when building a data management ecosystem?

Data Management 52

Data Management Data Lake Management Data Warehouse 52

AltexSoft

AUGUST 29, 2023

In 2010, a transformative concept took root in the realm of data storage and analytics — a data lake. The term was coined by James Dixon , Back-End Java, Data, and Business Intelligence Engineer, and it started a new era in how organizations could store, manage, and analyze their data. What is a data lake?

Data Lake 52

Data Lake Architecture IT Amazon Web Services 52

Knowledge Hut

JANUARY 29, 2024

In today's data-driven world, where information reigns supreme, businesses rely on data to guide their decisions and strategies. However, the sheer volume and complexity of raw data from various sources can often resemble a chaotic jigsaw puzzle. What Is Data Wrangling? Why Is Data Wrangling Important?

Raw Data 52

Raw Data Data Mining Data Preparation Structured Data 52

Snowflake

APRIL 5, 2023

Two different data modeling approaches—dimensional data modeling and Data Vault—each have their own pros and cons. Modernizing a data warehouse with Snowflake Data Cloud is a smart investment that can provide significant benefits to businesses of all sizes, today more than ever as data models become ever more complex.

Data Warehouse 52

Data Warehouse Healthcare Unstructured Data Metadata 52

Ascend.io

AUGUST 31, 2023

In the dynamic world of data, many professionals are still fixated on traditional patterns of data warehousing and ETL, even while their organizations are migrating to the cloud and adopting cloud-native data services. Modern platforms like Redshift , Snowflake , and BigQuery have elevated the data warehouse model.

Data Lake 52

Data Lake ETL Tools Data Warehouse Data Pipeline 52

DareData

JULY 5, 2023

Learn how we build data lake infrastructures and help organizations all around the world achieving their data goals. In today's data-driven world, organizations are faced with the challenge of managing and processing large volumes of data efficiently.

Data Lake 52

Data Lake Building Raw Data ETL Tools 52

Knowledge Hut

JANUARY 18, 2024

Data Science has risen to become one of the world's topmost emerging multidisciplinary approaches in technology. Recruiters are hunting for people with data science knowledge and skills these days. Data Scientists collect, analyze, and interpret large amounts of data. Choose data sets.

Data Science 98

Data Science Business Analyst ETL Method Data Architect 98

Data Engineering Weekly

FEBRUARY 26, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. I believe Data Contract is a technology solution to bring organizational change.

Data Engineering 57

Data Engineering Data Engineer Engineering Raw Data 57

Towards Data Science

MAY 3, 2023

Data Engineering Learn about slow change dimensions (SCD) and how to implement SCD Type 2 in VDK Photo by Joshua Sortino on Unsplash Data is the backbone of any organization, and in today’s fast-paced world, it is crucial to keep track of its versions. They store and manage current and historical data in a data warehouse.

Data Lake 80

Data Lake Data SQL Data Warehouse 80

Monte Carlo

MAY 28, 2024

A data ingestion architecture is the technical blueprint that ensures that every pulse of your organization’s data ecosystem brings critical information to where it’s needed most. Ensuring all relevant data inputs are accounted for is crucial for a comprehensive ingestion process.

Data Ingestion 52

Data Ingestion Architecture Designing Hadoop 52

Monte Carlo

APRIL 24, 2023

Data lakes are useful, flexible data storage repositories that enable many types of data to be stored in its rawest state. Traditionally, after being stored in a data lake, raw data was then often moved to various destinations like a data warehouse for further processing, analysis, and consumption.

Data Lake 52

Data Lake Google Cloud Data Warehouse AWS 52

ProjectPro

FEBRUARY 16, 2023

ETL is a critical component of success for most data engineering teams, and with teams harnessing it with the power of AWS, the stakes are higher than ever. Data Engineers and Data Scientists require efficient methods for managing large databases, which is why centralized data warehouses are in high demand.

AWS 52

AWS Data Management ETL Tools Management 52

Workfall

JULY 4, 2023

Reading Time: 8 minutes In the world of data engineering, a mighty tool called DBT (Data Build Tool) comes to the rescue of modern data workflows. Imagine a team of skilled data engineers on an exciting quest to transform raw data into a treasure trove of insights. These guards are tests in DBT.

Data Warehouse 69

Data Warehouse SQL PostgreSQL Database 69

Knowledge Hut

JANUARY 30, 2024

In today's world, where data rules the roost, data extraction is the key to unlocking its hidden treasures. As someone deeply immersed in the world of data science, I know that raw data is the lifeblood of innovation, decision-making, and business progress. What is data extraction?

ETL Tools 52

ETL Tools Database-centric Data Mining Data Cleanse 52

Data Engineering Podcast

DECEMBER 16, 2019

The solution to discoverability and tracking of data lineage is to incorporate a metadata repository into your data platform. The metadata repository serves as a data catalog and a means of reporting on the health and status of your datasets when it is properly integrated into the rest of your tools.

Metadata 100

Metadata PostgreSQL Datasets Data Warehouse 100

ProjectPro

JULY 22, 2016

The desire to save every bit and byte of data for future use, to make data-driven decisions is the key to staying ahead in the competitive world of business operations. For the same cost, organizations can now store 50 times as much data as in a Hadoop data lake than in a data warehouse.

Data Lake 40

Data Lake Data Warehouse Hadoop Unstructured Data 40

Hevo

MARCH 31, 2023

dbt is a data transformation tool used by data engineers to process raw data within data warehouses. While dbt is a robust tool for transformation, you should be aware of the dbt commands to harness the power of dbt. Mastering dbt commands will let you run complex asks on dbt projects.

Raw Data 52

Raw Data Data Warehouse Data Engineering Data Engineer 52

Knowledge Hut

JUNE 30, 2023

The demand for experienced data engineers continuously expands in today's data-driven environment. Books on data engineering serve as essential resources to guide you through the vast terrain of data engineering. What is Data Engineering? Who are Data Engineers?

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Hepta Analytics

FEBRUARY 14, 2022

DE Zoomcamp 2.2.1 – Introduction to Workflow Orchestration Following last weeks blog , we move to data ingestion. We already had a script that downloaded a csv file, processed the data and pushed the data to postgres database. This week, we got to think about our data ingestion design. It is called Idempotency.

Data Ingestion 52

Data Ingestion Data Engineering Data Engineer Engineering 52

Hevo

APRIL 27, 2023

In today’s data-driven era, you have more raw data than ever before. However, to leverage the power of big data, you need to convert raw data into valuable insights for informed decision-making. ” While they may sound […]

Raw Data 52

Raw Data Big Data Data Data Warehouse 52

ProjectPro

FEBRUARY 16, 2023

The demand for skilled data engineers who can build, maintain, and optimize large data infrastructures does not seem to slow down any sooner. At the heart of these data engineering skills lies SQL that helps data engineers manage and manipulate large amounts of data. of data engineer job postings on Indeed?

Data Engineering 52

Data Engineering Data Engineer SQL Engineering 52

Grouparoo

DECEMBER 14, 2021

The ETL data integration process has been around for decades and is an integral part of data analytics today. In this article, we’ll look at what goes on in the ETL process and some modern variations that are better suited to our modern, data-driven society. ETL data pipelines can be built using a variety of approaches.

Process 52

Process Raw Data Data Warehouse Data Pipeline 52

Knowledge Hut

JUNE 28, 2023

In our data-driven world, our lives are governed by big data. The TV shows we watch, the social media we follow, the news we read, and even the optimized routes we take to work are all influenced by the power of big data analytics. The answer lies in the strategic utilization of business intelligence for data mining (BI).

Data Mining 52

Data Mining Business Intelligence BI Datasets 52

Ascend.io

FEBRUARY 23, 2024

You are a data professional. Exciting, isn’t it Yet, embarking on the AI adoption journey introduces a series of challenges, with one of the most significant being the readiness of your data platform. In this article, we outline the essential prerequisites for an AI data platform.

Cloud Storage 52

Cloud Storage Data Ingestion Machine Learning Algorithm 52

Workfall

SEPTEMBER 18, 2023

Reading Time: 9 minutes Imagine your data as pieces of a complex puzzle scattered across different platforms and formats. This is where the power of data integration comes into play. Meet Airbyte, the data magician that turns integration complexities into child’s play. In this blog, we will cover: What is Airbyte?

Data Pipeline 93

Data Pipeline Raw Data Data Schemas Healthcare 93

ProjectPro

DECEMBER 7, 2021

Data pipelines are a significant part of the big data domain, and every professional working or willing to work in this field must have extensive knowledge of them. As data is expanding exponentially, organizations struggle to harness digital information's power for different business use cases. What is a Big Data Pipeline?

Data Pipeline 52

Data Pipeline Architecture Kafka AWS 52