The Modern Data Company

APRIL 15, 2024

The term “Data Product” has become a buzzword, often misused or overstretched. But at its core, a Data Product is much more than just data. This blend ensures that a Data Product is informative, actionable, and adaptable to various needs.

Government 52

Government Metadata Data Data Lake 52

Databand.ai

AUGUST 30, 2023

Poor data quality: The lack of automation and data governance in legacy architectures can lead to data quality issues, such as incomplete, inaccurate, or duplicate data. They include the various databases, applications, APIs, and external systems from which data is collected and ingested.

Architecture 52

Architecture Data Ingestion Data Governance Data Cleanse 52

Data Engineering Podcast

SEPTEMBER 23, 2018

Using the metaphor of a museum curator carefully managing the precious resources on display and in the vaults, he discusses the various layers of an enterprise data strategy. Can you walk through the stages of an ideal lifecycle for data within the context of an organizations uses for it?

Data Lake 100

Data Lake Data Warehouse Data Architecture Architecture 100

AltexSoft

FEBRUARY 11, 2023

In the article, we explore the role of a data architect, discuss the responsibilities and required skills, and share what kind of companies may need such a specialist. What is a data architect? Data architecture is the organization and design of how data is collected, transformed, integrated, stored, and used by a company.

Data Architect 87

Data Architect Certification Generalist Big Data 87

Cloudera

SEPTEMBER 15, 2022

The gathering in 2022 marked the sixteenth year for top data and analytics professionals to come to the MIT campus to explore current and future trends. A key area of focus for the symposium this year was the design and deployment of modern data platforms. See below. .

Data Lake 81

Data Lake Analytics Application Cloud Storage Architecture 81

Data Engineering Podcast

JUNE 19, 2022

Summary Metadata is the lifeblood of your data platform, providing information about what is happening in your systems. In order to level up their value a new trend of active metadata is being implemented, allowing use cases like keeping BI reports up to date, auto-scaling your warehouses, and automated data governance.

Metadata 130

Metadata MongoDB Scala MySQL 130

Monte Carlo

MAY 14, 2024

At its core, a table format is a sophisticated metadata layer that defines, organizes, and interprets multiple underlying data files. Table formats incorporate aspects like columns, rows, data types, and relationships, but can also include information about the structure of the data itself.

Data Lake 52

Data Lake Metadata Hadoop Data Governance 52

Cloudera

AUGUST 21, 2020

Data-in-motion is predominantly about streaming data so enterprises typically have two different ways or binary ways of looking at data. The governance aspect is perhaps even more important and businesses need to be able to understand where the data comes from.

Banking 61

Banking Kafka Cloud Storage Government 61

Cloudera

AUGUST 4, 2021

And if data security tops IT concerns, data governance should be their second priority. Not only is it critical to protect data, but data governance is also the foundation for data-driven businesses and maximizing value from data analytics. But it’s still not easy. But it’s still not easy.

Data Governance 84

Data Governance Government Metadata Data Security 84

Monte Carlo

AUGUST 25, 2023

That’s why it’s essential for teams to choose the right architecture for the storage layer of their data stack. But, the options for data storage are evolving quickly. Different vendors offering data warehouses, data lakes, and now data lakehouses all offer their own distinct advantages and disadvantages for data teams to consider.

Data Lake 52

Data Lake Data Warehouse Unstructured Data Raw Data 52

Monte Carlo

NOVEMBER 14, 2023

Databricks announced that Delta tables metadata will also be compatible with the Iceberg format, and Snowflake has also been moving aggressively to integrate with Iceberg. It is designed to be easily queryable with SQL even for large analytic tables (we’re talking petabytes of data). How Apache Iceberg tables structure metadata.

Data Lake 64

Data Lake Metadata Data Warehouse SQL 64

Data Engineering Podcast

NOVEMBER 26, 2023

Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics.

Architecture 147

Architecture Data Lake High Quality Data SQL 147

Precisely

JANUARY 25, 2024

To achieve that, IT leaders must understand the various methods and tools available to them, and how to design a manageable roadmap that balances costs, risks, and benefits. Cloud-based DevOps provides a modern, agile environment for developing and maintaining applications and services that interact with the organization’s mainframe data.

Metadata 69

Metadata Data Governance Relational Database Government 69

AltexSoft

AUGUST 22, 2022

In this post, we’ll attempt to explain the idea behind a data fabric, its architectural building blocks, the benefits it brings, and ways to approach its implementation. What is data fabric? to provide a unified view of all enterprise data. Data fabric architecture example. Support for different data types and use cases.

Architecture 52

Architecture Metadata Data Lake Machine Learning 52

AltexSoft

NOVEMBER 10, 2021

The pun being obvious, there’s more to that than just a new term: Data lakehouses combine the best features of both data lakes and data warehouses and this post will explain this all. What is a data lakehouse? Data warehouse vs data lake vs data lakehouse: What’s the difference.

Architecture 52

Architecture Data Lake Data Warehouse Metadata 52

Cloudera

MARCH 23, 2022

Iceberg is designed to be open and engine agnostic allowing datasets to be shared. The table information (such as schema, partition) is stored as part of the metadata (manifest) file separately, making it easier for applications to quickly integrate with the tables and the storage formats of their choice. 3: Open Performance.

Metadata 90

Metadata Data Architecture BI Machine Learning 90

Monte Carlo

JUNE 14, 2023

In this post, we will help you quickly level up your overall knowledge of data pipeline architecture by reviewing: Table of Contents What is data pipeline architecture? Why is data pipeline architecture important? What is data pipeline architecture? Zero ETL is a bit of a misnomer.

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

Ascend.io

JULY 19, 2023

But something about data mesh feels different, doesn’t it? For one, data mesh tackles the real headaches caused by an overburdened data lake and the annoying game of tag that’s too often played between the people who make data, the ones who use it, and everyone else caught in the middle.

Data Governance 40

Data Governance Government Metadata Data 40

The Modern Data Company

JANUARY 17, 2023

It can improve the data quality, discoverability, and observability of the data in your CDP. A Data Operating System Can Save Your CDP DataOS is the only operating system on the market designed to work with every tool, app, and data source within your data ecosystem. The result?

Metadata 59

Metadata Government Data Lake Data 59

AltexSoft

NOVEMBER 23, 2021

If the transformation step comes after loading (for example, when data is consolidated in a data lake or a data lakehouse ), the process is known as ELT. You can learn more about how such data pipelines are built in our video about data engineering. Self-service capabilities for all business users.

Process 69

Process Data Lake Metadata Data Warehouse 69

AltexSoft

JUNE 26, 2023

Robotic process automation Robotic process automation, or RPA is a type of software designed to perform repetitive and tedious daily operations otherwise carried out by humans. Relational vs non-relational databases As we mentioned above, relational or SQL databases are designed for structured or tabular data.

Data Collection 52

Data Collection Machine Learning Unstructured Data Non-relational Database 52

AltexSoft

SEPTEMBER 23, 2021

One of the innovative ways to address this problem is to build a data hub — a platform that unites all your information sources under a single umbrella. This article explains the main concepts of a data hub, its architecture, and how it differs from data warehouses and data lakes. What is Data Hub?

Architecture 52

Architecture Data Lake Unstructured Data Data Warehouse 52

AltexSoft

JULY 29, 2022

a runtime environment (sandbox) for classic business intelligence (BI), advanced analysis of large volumes of data, predictive maintenance , and data discovery and exploration; a store for raw data; a tool for large-scale data integration ; and. a suitable technology to implement data lake architecture.

Hadoop 59

Hadoop Big Data Google Cloud NoSQL 59

Snowflake

MARCH 30, 2023

Based on Tecton blog So is this similar to data engineering pipelines into a data lake/warehouse? Let’s explore that last point a little more closely… Data versioning Raw vault stores the business process output in its applied state as raw hub, link, and satellite tables.

Engineering 83

Engineering Raw Data Data Science Scala 83

AltexSoft

OCTOBER 8, 2021

They are applied to retrieve data from the source systems, perform transformations when necessary, and load it into a target system ( data mart , data warehouse, or data lake). So, why is data integration such a big deal? Connections to both data warehouses and data lakes are possible in any case.

Data Integration 52

Data Integration Hadoop Data Warehouse Data Lake 52

ProjectPro

NOVEMBER 15, 2021

Delta Lake Source: Github Delta Lake is an open-source project that allows you to create a Lakehouse design based on data lakes. On top of existing data lakes like S3, ADLS, GCS, and HDFS, Delta Lake enables ACID transactions, scalable metadata handling, and unifies streaming and batch data processing.

Big Data 52

Big Data Project Metadata Programming Language 52

ProjectPro

JANUARY 31, 2022

It also offers a unique architecture that allows users to quickly build tables and begin querying data without administrative or DBA involvement. Snowflake is a cloud-based data platform that provides excellent manageability regarding data warehousing, data lakes, data analytics, etc. What Does Snowflake Do?

Architecture 52

Architecture IT Data Warehouse Amazon Web Services 52

ProjectPro

AUGUST 24, 2021

In this project, you will explore the usage of Databricks Spark on Azure with Spark SQL and build this data pipeline. Upload it to Azure Data lake storage manually. Create a Data Factory pipeline to ingest files. Also, you get a chance to explore Azure Databricks, Data Factory, and Storage services.

Data Engineering 52

Data Engineering Data Engineer Coding Project 52

ProjectPro

NOVEMBER 30, 2021

Transform Data Once the data is validated, we apply transformations to remove duplicates, perform cleansing, standardization, business rule application, perform data integrity checks, data governance, use aggregations, and many more. Stage Data Data that has been transformed is stored in this layer.

Process 52

Process Data Pipeline Data Warehouse AWS 52

RandomTrees

FEBRUARY 6, 2024

Over the years, the field of data engineering has seen significant changes and paradigm shifts driven by the phenomenal growth of data and by major technological advances such as cloud computing, data lakes, distributed computing, containerization, serverless computing, machine learning, graph database, etc.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

AltexSoft

MAY 12, 2023

Unstructured data , on the other hand, is unpredictable and has no fixed schema, making it more challenging to analyze. Without a fixed schema, the data can vary in structure and organization. There are several widely used unstructured data storage solutions such as data lakes (e.g., Hadoop, Apache Spark).

Unstructured Data 59

Unstructured Data NoSQL Hadoop Data Lake 59

Ascend.io

APRIL 7, 2023

Data products are easily searchable and shareable among business units, allowing any stakeholder to use them as-is or combine them with products from other domains to create more sophisticated analytics, models, and customer-facing solutions. The overarching philosophy of a data mesh is that it’s designed with data products in mind.

Metadata 52

Metadata Datasets Data Governance Government 52

Cloudera

OCTOBER 7, 2021

Introduction to the Data Mesh Architecture. The concept of the data mesh architecture is not entirely new; Its conceptual origins are rooted in the microservices architecture, its design principles (i.e., In the Enterprise Data Management realm, such a data domain is called an Authoritative Data Domain (ADD).

Architecture 123

Architecture Metadata Government Kafka 123

Cloudera

OCTOBER 11, 2021

Customers can now seamlessly automate migration to Cloudera’s Hybrid Data Platform — Cloudera Data Platform (CDP) to dynamically auto-scale cloud services with Cloudera Data Engineering (CDE) integration with Modak Nabu. Customers using Modak Nabu with CDP today have deployed Data Lakes and.

Data Engineering 92

Data Engineering Data Engineer Cloud Engineering 92

Monte Carlo

JANUARY 23, 2024

It’s harder to gain consensus on governance – and that’s OK In a previous era of data engineering, data team structure was very much centralized, with data engineers and tech-savvy analysts serving as the “librarians” of the data for the entire company. What are data engineering skills?

Data Engineering 98

Data Engineering Data Engineer Engineering Data 98

Precisely

OCTOBER 16, 2023

This shift has influenced how cloud resources are designed and marketed, focusing on easy access, modularity, and straightforward deployment. The data-driven enterprise : Another key trend centers around a deeper understanding of data and its vital importance in creating and sustaining competitive advantage.

Cloud 72

Cloud Cloud Computing Metadata Data Governance 72

Monte Carlo

JANUARY 23, 2023

When it comes to the data community, there’s always a debate broiling about something— and right now “data mesh vs data lake” is right at the top of that list. In this post we compare and contrast the data mesh vs data lake to illustrate the benefits of each and help discover what’s right for your data platform.

Data Lake 52

Data Lake Unstructured Data Business Intelligence Architecture 52