phData: Data Engineering

SEPTEMBER 19, 2023

With many data modeling methodologies and processes available, choosing the right approach can be daunting. This blog will guide you through the best data modeling methodologies and processes for your data lake, helping you make informed decisions and optimize your data management practices. What is a Data Lake?

Data Lake 52

Data Lake Process Metadata Data Warehouse 52

Snowflake

NOVEMBER 30, 2023

At TCS , we help companies shift their enterprise data warehouse (EDW) platforms to the cloud as well as offering IT services. We’re extremely familiar with just how tricky a cloud migration can be, especially when it involves moving historical business data. Use separate data warehouses for cost-effective data loading.

Data Warehouse 98

Data Warehouse Banking Data Cloud 98

Workfall

JUNE 12, 2023

So, embrace the power of Change Data Capture, and embark on a captivating journey where the magic of real-time data awaits. In this blog, we will cover: What Is CDC and Its Benefits? It bridges gaps in data ecosystems, ensuring consistency and synchronisation across systems. Where Is CDC Used and Who Uses It?

Telecommunication 80

Telecommunication Metadata Healthcare Finance 80

Lyft Engineering

OCTOBER 3, 2023

In this post we will define data quality at a high-level and explore our motivation to achieve better data quality. We will then introduce our in-house product, Verity, and showcase how it serves as a central platform for ensuring data quality in our Hive Data Warehouse. What and Where is Data Quality?

Big Data 84

Big Data Metadata Data Warehouse Data 84

Data Engineering Podcast

MARCH 22, 2021

For analytical systems, the only way to provide this reliably is by implementing change data capture (CDC). Unfortunately, this is a non-trivial undertaking, particularly for teams that don’t have extensive experience working with streaming data and complex distributed systems. What are the alternatives to CDC?

Data Warehouse 100

Data Warehouse Metadata Data Lake Hadoop 100

Cloudera

AUGUST 31, 2021

Cloudera and Accenture demonstrate strength in their relationship with an accelerator called the Smart Data Transition Toolkit for migration of legacy data warehouses into Cloudera Data Platform. Accenture’s Smart Data Transition Toolkit . Are you looking for your data warehouse to support the hybrid multi-cloud?

Data Warehouse 80

Data Warehouse Database-centric Metadata Cloud 80

Data Engineering Weekly

MARCH 10, 2024

Pradheep Arjunan - Shared insights on AZ's journey from on-prem to the cloud data warehouses. Google: Croissant- a metadata format for ML-ready datasets Google Research introduced Croissant, a new metadata format designed to make datasets ML-ready by standardizing the format, facilitating easier use in machine learning projects.

Data Engineering 52

Data Engineering Data Engineer Engineering Datasets 52

Cloudera

DECEMBER 10, 2020

Why worry about costs with cloud-native data warehousing? Have you been burned by the unexpected costs of a cloud data warehouse? If not, before adopting a cloud data warehouse, consider the true costs of a cloud-native data warehouse. These costs impede the adoption of cloud-native data warehouses.

Data Warehouse 98

Data Warehouse Cloud Storage Metadata Data 98

Netflix Tech

DECEMBER 3, 2022

The dataflow migration command is a special feature, developed single handedly by Stephen Huenneke , to fully automate the communication and tracking of a data warehouse table changes. It lets you create YAML formatted mock data files based on selected tables, columns and a few rows of data from the Netflix data warehouse.

Data Pipeline 143

Data Pipeline Scala Metadata Food 143

Databand.ai

JULY 19, 2023

Data Lineage Tools: Key Capabilities and 5 Notable Solutions Ryan Yackel July 19, 2023 What Are Data Lineage Tools? Data lineage tools provide a visual representation of your data’s journey across multiple systems and transformations. It provides context for data, making it easier to understand and manage.

Pipeline-centric 52

Pipeline-centric Data Governance Metadata Government 52

Cloudera

JUNE 18, 2022

These lakes power mission critical large scale data analytics, business intelligence (BI), and machine learning use cases, including enterprise data warehouses. In recent years, the term “data lakehouse” was coined to describe this architectural pattern of tabular analytics over data in the data lake.

Data Lake 90

Data Lake Data Warehouse BI SQL 90

Cloudera

JANUARY 19, 2022

Gartner® recognized Cloudera in three recent reports – Magic Quadrant for Cloud Database Management Systems (DBMS), Critical Capabilities for Cloud Database Management Systems for Analytical Use Cases and Critical Capabilities for Cloud Database Management Systems for Operational Use Cases. Get started with CDP.

Database 87

Database Cloud Data Warehouse Data Lake 87

Data Engineering Weekly

JULY 9, 2023

link] Capital One: Democratizing machine learning It is an exciting blog post + video interview from Capital One focusing on the people and technology aspect of democratizing the machine learning practice across the org. The blog narrates the optimization and cluster architectural changes by ByteDance on top of ClickHouse.

Data Engineering 52

Data Engineering Data Engineer Engineering Lambda Architecture 52

Data Engineering Weekly

MAY 16, 2023

In the first part of this series, we talked about design patterns for data creation and the pros & cons of each system from the data contract perspective. In the second part, we will focus on architectural patterns to implement data quality from a data contract perspective. Why is Data Quality Expensive?

Engineering 110

Engineering Kafka Data Pipeline Data Warehouse 110

Cloudera

JULY 14, 2023

Depending on the size and usage patterns of the data, several different strategies could be pursued to achieve a successful migration. In this blog, I will describe a few strategies one could undertake for various use cases. This will be discussed in a later blog. Relatively fast as the underlying data files are kept in place.

Metadata 69

Metadata Data Warehouse Big Data Ecosystem Java 69

Databand.ai

AUGUST 30, 2023

Data Cataloging Data cataloging is the process of organizing and categorizing data assets to make them easily discoverable and understandable for data teams. This enables data teams to quickly and easily find the data they need for their analytics projects. In this article: Why Are DataOps Tools Important?

Data Cleanse 52

Data Cleanse Data Pipeline Data Ingestion Data Validation 52

Maxime Beauchemin

JANUARY 20, 2017

In relation to previously existing roles , the data engineering field could be thought of as a superset of business intelligence and data warehousing that brings more elements from software engineering. Sure, there’s a need to abstract the complexity of data processing, computation and storage.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130

Monte Carlo

FEBRUARY 9, 2023

Over the past several years, data warehouses have evolved dramatically, but that doesn’t mean the fundamentals underpinning sound data architecture needs to be thrown out the window. While data vault has many benefits, it is a sophisticated and complex methodology that can present challenges to data quality.

Architecture 52

Architecture Raw Data Metadata Data Warehouse 52

ProjectPro

FEBRUARY 8, 2023

Do ETL and data integration activities seem complex to you? Read this blog to understand everything about AWS Glue that makes it one of the most popular data integration solutions in the industry. Did you know the global big data market will likely reach $268.4 Businesses are leveraging big data now more than ever.

AWS 98

AWS Scala Metadata Data Lake 98

Monte Carlo

JANUARY 23, 2024

In short, Maxime argues that to effectively scale data science and analytics in the future, data teams needed a specialized engineer to manage ETL, build pipelines, and scale data infrastructure. “We The data warehouse is the mirror of the organization in many ways. Data debt or data pipeline chaos.

Data Engineering 98

Data Engineering Data Engineer Engineering Data 98

Cloudyard

MARCH 4, 2023

DCDW Architecture Above all, Architecture was divided into three Business layers: Firstly,Agile Data ingestion : Heterogeneous Source System fed the data into Cloud. Respective Cloud would consume/Store the data in bucket or containers. Load the data AS-IS into Snowflake called RAW layer.

Architecture 52

Architecture Cloud Metadata Data Ingestion 52

Cloudera

SEPTEMBER 29, 2022

Data Mesh is a concept used to help scale a company’s data footprint in a manageable way. It is a set of rubrics around people, process, and technology choices that allow for companies to scale their data systems. . Data mesh conceptual hierarchy. The data itself is then treated as a product. Miro: [link].

Architecture 73

Architecture Data Architecture Metadata Data Warehouse 73

Data Engineering Weekly

FEBRUARY 18, 2024

[link] Joe Reis: Everything Ends - My Journey With the Modern Data Stack Joe writes another excellent retrospect for Modern Data Stack, walking down memory lane of the early and golden days of the Modern Data Stack. We are so over the Big Data Era to Modern Data Stack.

Data Engineering 62

Data Engineering Data Engineer Engineering Data 62

Cloudera

OCTOBER 11, 2022

In the article, Bret Greenstein, data, analytics and AI partner at PwC identifies that, “No matter how organizations move toward scaling AI in the coming year, it’s important to understand the significant differences between using AI as a ‘proof of concept’ and scaling those efforts.” But it isn’t just aggregating data for models.

Data Science 90

Data Science Aggregated Data Data Consulting 90

Cloudera

NOVEMBER 12, 2020

Your sunk costs are minimal and if a workload or project you are supporting becomes irrelevant, you can quickly spin down your cloud data warehouses and not be “stuck” with unused infrastructure. Cloud deployments for suitable workloads gives you the agility to keep pace with rapidly changing business and data needs.

Cloud 102

Cloud Data Warehouse Banking Data 102

Data Engineering Weekly

OCTOBER 30, 2022

Luis Velasco: Data Contracts - The Mesh Glue I'm excited about the Schemata & Data contract because it perfectly stitches catalogs, data quality, and glossary. The current state of these systems is inherently passive systems. The author explains how to dump the history of blockchains into S3.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Ingestion 52

Databand.ai

JULY 28, 2022

In general, it makes it possible to identify errors in data, reduce the risk associated with system and process changes, and increase trust in data. All of these are essential at a time when data plays such an integral role in business outcomes and decision-making. This technique focuses directly on the data (vs.

Metadata 52

Metadata Data Lake Datasets Data Warehouse 52

Data Engineering Weekly

JULY 2, 2023

🌟 Submit Your Conference Talk Proposal Here LinkedIn: Declarative Data Pipelines with Hoptimator Data pipeline often requires multiple hop through the system to serve the end customers. LinkedIn write about Hoptimator for auto generated Flink pipeline with multiple stages of systems.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

DataKitchen

AUGUST 3, 2021

The past decades of enterprise data platform architectures can be summarized in 69 words. First-generation – expensive, proprietary enterprise data warehouse and business intelligence platforms maintained by a specialized team drowning in technical debt. DDD divides a system or model into smaller subsystems called domains.

Pharmaceutical 98

Pharmaceutical Data Lake Data Architecture Scala 98

Cloudera

SEPTEMBER 17, 2020

By enabling their event analysts to monitor and analyze events in real time, as well as directly in their data visualization tool, and also rate and give feedback to the system interactively, they increased their data to insight productivity by a factor of 10. . Our solution: Cloudera Data Visualization.

Unstructured Data 80

Unstructured Data Data Warehouse Pharmaceutical MySQL 80

Databand.ai

JUNE 28, 2023

“Observability” has become a bit of a buzzword so it’s probably best to define it: Data observability is the blanket term for monitoring and improving the health of data within applications and systems like data pipelines. Data observability vs. monitoring: what is the difference?

Data Pipeline 98

Data Pipeline Data Engineering Data Engineer Engineering 98

Rockset

MAY 8, 2023

In this blog, we capture engineering stories from 5 early adopters of vector search- Pinterest, Spotify, eBay, Airbnb and Doordash- who have integrated AI into their applications. In the next sections, we’ll summarize 5 engineering blogs on vector search and highlight key implementation considerations.

Metadata 52

Metadata Algorithm Datasets Google Cloud 52

Data Engineering Weekly

DECEMBER 4, 2022

The author narrates why the data models are still important for managing data assets' structure, content, and relationships but also need to keep agility in mind to bring business velocity. The article highlights the challenges of maintaining data models in a world where SQL data warehouses are no longer the primary data platform.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

Monte Carlo

MARCH 3, 2021

Our SOC 2 designation means that Monte Carlo has designed a set of internal controls, systems, policies, and procedures that meet industry best practices for protecting our customer data. Specifically, with Monte Carlo, data never leaves your environment — full stop. Until then, here’s wishing you no data downtime!

Metadata 52

Metadata Data Lake Business Intelligence Certification 52

DoorDash Engineering

APRIL 12, 2023

Challenges of ad-hoc SQLs Our initial goal with Curie was to standardize the analysis methodologies and simplify the experiment analysis process for data scientists. Core Data Models / Semantics We placed a strong emphasis on identifying the most comprehensive and effective core data models for users to create their own metrics.

SQL 82

SQL Metadata Raw Data Government 82

Workfall

AUGUST 14, 2023

Two powerful tools that have emerged to simplify the management of data pipelines are DBT (Data Build Tool) and Airflow. These tools, when combined, form a dynamic duo that streamlines and scales data engineering workflows. So, let’s dive into the world of DBT and Airflow and discover how they revolutionize data pipelines.

Data Pipeline 62

Data Pipeline Raw Data Data Database 62

Maxime Beauchemin

AUGUST 28, 2017

This leads to systemic, stupid errors that waste hours. Consensus seeking Whether you think that old-school data warehousing concepts are fading or not, the quest to achieve conformed dimensions and conformed metrics is as relevant as it ever was. Data engineers are many degrees removed from those who are “moving the needle”.

Data Engineering 174

Data Engineering Data Engineer Engineering Software Engineer 174