Data Engineering Weekly

DECEMBER 29, 2022

Data catalogs are the most expensive data integration systems you never intended to build. Data Catalog as a passive web portal to display metadata requires significant rethinking to adopt modern data workflow, not just adding “modern” in its prefix. The modern(?)

Metadata 62

Metadata Data Warehouse ETL Tools Data Workflow 62

Monte Carlo

APRIL 27, 2023

Empowered by the rise of the modern data stack , leading companies like Warner Music Group , Chime and Petsmart are now turning to the Composable CDP to unlock a modular customer data architecture that centers on the cloud data warehouse. Why not build one single store of customer data to activate from centrally?

Data Warehouse 72

Data Warehouse Data Collection Architecture Data Storage 72

Knowledge Hut

JUNE 30, 2023

The essential theories, procedures, and equipment for creating trustworthy and effective data systems are covered in this book. It explores subjects including data modeling, data pipelines, data integration, and data quality, offering helpful advice on organizing and implementing reliable data solutions.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Cloudera

SEPTEMBER 12, 2022

Combining and analyzing both structured and unstructured data is a whole new challenge to come to grips with, let alone doing so across different infrastructures. Both obstacles can be overcome using modern data architectures, specifically data fabric and data lakehouse. Unified data fabric.

Unstructured Data 74

Unstructured Data Data Lake Data Architecture Data 74

Knowledge Hut

NOVEMBER 17, 2023

Data Modeling The process of creating a logical and physical data model for a system is known as data modeling. Understanding data modeling concepts like entity-relationship diagrams, data normalization, and data integrity is a requirement for an Azure Data Engineer.

Data Engineering 52

Data Engineering Data Engineer Engineering Scala 52

Ascend.io

DECEMBER 21, 2022

The last three years have seen a remarkable change in data infrastructure. Now, data teams are embracing a new approach: reverse ETL. Cloud data warehouses, such as Snowflake and BigQuery, have made it simpler than ever to combine all of your data into one location. Make your data operational.

ETL Tools 52

ETL Tools ETL System Data Warehouse Data Consolidation 52

Knowledge Hut

OCTOBER 30, 2023

This data and reports are generated and developed by Power BI developers. A Power BI developer is a business intelligence personnel who thoroughly understands business intelligence, data integration, data warehousing, modeling, database administration, and technical aspects of BI systems.

BI 52

BI Business Intelligence Data Cleanse Business Analyst 52

Data Engineering Podcast

AUGUST 12, 2019

Fivetran is a platform that does the hard work for you and replicates information from your source systems into whichever data warehouse you use. Upcoming events include the O’Reilly AI Conference, the Strata Data Conference, and the combined events of the Data Architecture Summit and Graphorum.

Media 100

Media Data Warehouse Data Big Data 100

phData: Data Engineering

JULY 18, 2023

With the birth of cloud data warehouses, data applications, and generative AI , processing large volumes of data faster and cheaper is more approachable and desired than ever. First up, let’s dive into the foundation of every Modern Data Stack, a cloud-based data warehouse.

Data Warehouse 52

Data Warehouse Pipeline-centric Government Data 52

Monte Carlo

DECEMBER 8, 2023

Let’s dive deeper into the definitions of data provenance and data lineage, what they’re used for, and why they’re so important for data teams. Table of Contents What is data provenance? The role of data provenance in ensuring data integrity and authenticity Data provenance use cases What is data lineage?

Metadata 52

Metadata Data Data Warehouse Government 52

Monte Carlo

DECEMBER 8, 2023

Let’s dive deeper into the definitions of data provenance and data lineage, what they’re used for, and why they’re so important for data teams. Table of Contents What is data provenance? The role of data provenance in ensuring data integrity and authenticity Data provenance use cases What is data lineage?

Metadata 52

Metadata Data Data Warehouse Government 52

Monte Carlo

FEBRUARY 9, 2023

Over the past several years, data warehouses have evolved dramatically, but that doesn’t mean the fundamentals underpinning sound data architecture needs to be thrown out the window. What is a Data Vault model? Pie Insurance , a leading small business insurtech, leverages a data vault 2.0

Architecture 52

Architecture Raw Data Metadata Data Warehouse 52

Silectis

JANUARY 22, 2021

In this demo, we’ll show you how native tools in the Magpie data engineering platform play well with Snowflake, ultimately, allowing your team to do more in a centralized data engineering environment. This is an important piece of a modern and reliable data architecture.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Data Engineering Podcast

JULY 22, 2019

We have partnered with organizations such as O’Reilly Media, Dataversity, and the Open Data Science Conference. Upcoming events include the O’Reilly AI Conference, the Strata Data Conference, and the combined events of the Data Architecture Summit and Graphorum. What has been your experience in that regard?

Data Lake 100

Data Lake Hadoop Data Architecture 100

AltexSoft

FEBRUARY 11, 2023

This specialist works closely with people on both business and IT sides of a company to understand the current needs of the stakeholders and help them unlock the full potential of data. To get a better understanding of a data architect’s role, let’s clear up what data architecture is.

Data Architect 87

Data Architect Certification Generalist Big Data 87

Monte Carlo

JULY 27, 2023

The most common use case data quality engineers support are: Analytical dashboards : Mentioned in 56% of job postings Machine learning or data science teams : Mentioned in 34% of postings Gen AI : Mentioned in one job posting (but really emphatically).

Engineering 52

Engineering Healthcare Scala Data Warehouse 52

ProjectPro

FEBRUARY 16, 2023

Knowing SQL helps data engineers optimize data infrastructures for better performance and efficiency and also develop more effective data models and data warehousing solutions. Data integration will become highly significant as the amount of data globally grows in volume, variety, and complexity.

Data Engineering 52

Data Engineering Data Engineer SQL Engineering 52

Silectis

JANUARY 3, 2021

Big Data Processing In order to extract value or insights out of big data, one must first process it using big data processing software or frameworks, such as Hadoop. Big Query Google’s cloud data warehouse. Data Catalog An organized inventory of data assets relying on metadata to help with data management.

Data Engineering 40

Data Engineering Data Engineer Engineering Hadoop 40

Ascend.io

FEBRUARY 28, 2024

This process reduces noise in the data, which is crucial for the effectiveness of AI algorithms, especially in complex predictive models and deep learning applications. By automating these tasks, data engineers can focus on more complex aspects of data architecture and strategy, improving productivity and efficiency.

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

Striim

OCTOBER 4, 2022

Use Striim to power your real-time data architecture In the past, there was some hesitation toward adopting real-time systems unless they were absolutely necessary. Over time, improvements in the price and performance of memory and CPUs have made real-time data processing significantly more affordable for large-scale adoption.

Media 52

Media Data Data Architecture Data Warehouse 52

AltexSoft

MARCH 30, 2023

What is Databricks Databricks is an analytics platform with a unified set of tools for data engineering, data management , data science, and machine learning. It combines the best elements of a data warehouse, a centralized repository for structured data, and a data lake used to host large amounts of raw data.

Scala 64

Scala Data Lake BI Google Cloud 64

AltexSoft

NOVEMBER 10, 2021

The pun being obvious, there’s more to that than just a new term: Data lakehouses combine the best features of both data lakes and data warehouses and this post will explain this all. What is a data lakehouse? Data warehouse vs data lake vs data lakehouse: What’s the difference.

Architecture 52

Architecture Data Lake Data Warehouse Metadata 52

Monte Carlo

APRIL 20, 2023

Solutions with automated data lineage capabilities constantly update these graphs and illustrate them as nodes and edges, or in other words, the objects through which the data travels and the relationship between them. This is one of the most frequent data lineage use cases leveraged by Vox. It’s just easy validation.

Data Warehouse 52

Data Warehouse BI Data Government 52

Databand.ai

DECEMBER 13, 2022

1) Joseph Machado Senior Data Engineer at LinkedIn Joseph is an experienced data engineer, holding a Master’s degree in Electrical Engineering from Columbia University and having spent time on the teams at Annalect, Narrativ, and most recently LinkedIn. You can also watch the video recording.

Data Engineering 52

Data Engineering Data Engineer Engineering AWS 52

ProjectPro

JANUARY 31, 2022

As the demand for big data grows, an increasing number of businesses are turning to cloud data warehouses. The cloud is the only platform to handle today's colossal data volumes because of its flexibility and scalability. Launched in 2014, Snowflake is one of the most popular cloud data solutions on the market.

Architecture 52

Architecture IT Data Warehouse Amazon Web Services 52

ProjectPro

DECEMBER 7, 2021

Generally, data pipelines are created to store data in a data warehouse or data lake or provide information directly to the machine learning model development. Keeping data in data warehouses or data lakes helps companies centralize the data for several data-driven initiatives.

Data Pipeline 52

Data Pipeline Architecture Kafka AWS 52

AltexSoft

OCTOBER 30, 2021

Data engineer’s integral task is building and maintaining data infrastructure — the system managing the flow of data from its source to destination. This typically includes setting up two processes: an ETL pipeline , which moves data, and a data storage (typically, a data warehouse ), where it’s kept.

Data Engineering 98

Data Engineering Data Engineer Engineering Machine Learning 98

U-Next

AUGUST 10, 2022

Develop your data architecture: They design, develop, and manage data structures systematically, even while maintaining them in line with business needs. Automate Workflows: Data Engineers go into the data to identify processes that may be automated to remove manual involvement.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Science 52

Striim

NOVEMBER 17, 2023

Striim serves as a real-time data integration platform that seamlessly and continuously moves data from diverse data sources to destinations such as cloud databases, messaging systems, and data warehouses, making it a vital component in modern data architectures.

Machine Learning 52

Machine Learning Data Process PostgreSQL Process 52

dbt Developer Hub

FEBRUARY 7, 2022

This aggregation process requires an analytics warehouse, as all of these things need to be synced together outside of the application database itself to incorporate other data sources (billing / events information, past touchpoints in the CRM, etc). You most definitely deserve a jaffle. ?

Data Warehouse 52

Data Warehouse Datasets Data SQL 52

phData: Data Engineering

AUGUST 11, 2023

Data engineering models can help public health systems identify isolated cases before they explode into a health crisis. In Australia, the government’s healthcare branch uses Data Integration Partnership for Australia (DIPA) to identify adverse events. There is potential to revolutionize the industry by 2030.

Healthcare 52

Healthcare Data Engineering Data Engineer Hospitality 52

ProjectPro

MAY 31, 2021

For such scenarios, data-driven integration becomes less comfortable, so you must prefer event-based data integration. This project will teach you how to design and implement an event-based data integration pipeline on the Google Cloud Platform by processing data using DataFlow.

Big Data 52

Big Data Coding Project Hadoop 52

Databand.ai

JANUARY 24, 2023

Supporting all of this requires a modern infrastructure and data architecture with appropriate governance. DataOps helps ensure organizations make decisions based on sound data. This step of the ladder includes efforts like master data management and data integration. Enter DataOps.

Retail 52

Retail Banking Government Manufacturing 52

AltexSoft

MAY 12, 2023

Data ingestion tools are software applications or services designed to collect, import, and process data from various sources into a central data storage system or repository. There are several widely used unstructured data storage solutions such as data lakes (e.g., Build data architecture.

Unstructured Data 59

Unstructured Data NoSQL Hadoop Data Lake 59

ProjectPro

SEPTEMBER 26, 2021

Big Data Engineer performs a multi-faceted role in an organization by identifying, extracting, and delivering the data sets in useful formats. A Big Data Engineer also constructs, tests, and maintains the Big Data architecture. Your organization will use internal and external sources to port the data.

Big Data 52

Big Data Data Engineering Data Engineer Engineering 52

Meltano

DECEMBER 7, 2022

ELT is becoming the default choice for data architectures and yet, many best practices focus primarily on “T”: the transformations. But the extract and load phase is where data quality is determined for transformation and beyond. Both of which will delete data that is present in the source system.

Raw Data 52

Raw Data Metadata Data Database 52

Monte Carlo

SEPTEMBER 16, 2020

While our ability to collect, store, aggregate, and visualize this data has largely kept up with the needs of modern data teams (think: domain-oriented data meshes , cloud warehouses , data visualization tools , and data modeling solutions ), the mechanics behind data quality and data integrity has lagged.

Software Engineer 52

Software Engineer Software Engineering Data Data Pipeline 52