Data Engineering Podcast

MARCH 5, 2023

TimeXtender takes a holistic approach to data integration that focuses on agility rather than fragmentation. By bringing all the layers of the data stack together, TimeXtender helps you build data solutions up to 10 times faster and saves you 70-80% on costs. But don't worry, there is a better way.

Building 147

Building Database Design Metadata Machine Learning 147

Monte Carlo

MAY 14, 2024

At its core, a table format is a sophisticated metadata layer that defines, organizes, and interprets multiple underlying data files. Table formats incorporate aspects like columns, rows, data types, and relationships, but can also include information about the structure of the data itself.

Data Lake 52

Data Lake Metadata Hadoop Data Governance 52

Data Engineering Podcast

NOVEMBER 20, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. The biggest challenge with modern data systems is understanding what data you have, where it is located, and who is using it.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Data Engineering Podcast

MARCH 25, 2023

In this episode they explain why streaming architectures are so challenging, how they have designed Grainite to be robust and scalable, and how you can start using it today to build your streaming data applications without all of the operational headache. But don't worry, there is a better way.

MySQL 130

MySQL Python Architecture Machine Learning 130

Netflix Tech

NOVEMBER 14, 2023

In this context, managing the data, especially when it arrives late, can present a substantial challenge! In this three-part blog post series, we introduce you to Psyberg , our incremental data processing framework designed to tackle such challenges! Psyberg: The Game Changer!

Data Engineering 105

Data Engineering Data Engineer Engineering Metadata 105

Data Engineering Weekly

DECEMBER 10, 2023

link] Evidently: ML system design - 300 case studies to learn from An amazing compilation of ML system design articles from various companies. Key aspects of Psyberg: The initialization phase computes a range of data that needs reprocessing for a pipeline based on parameters. Stores metadata to utilize later.

Data Engineering 76

Data Engineering Data Engineer Engineering Metadata 76

Data Engineering Podcast

MARCH 18, 2023

TimeXtender takes a holistic approach to data integration that focuses on agility rather than fragmentation. By bringing all the layers of the data stack together, TimeXtender helps you build data solutions up to 10 times faster and saves you 70-80% on costs. But don't worry, there is a better way.

Data Security 130

Data Security Machine Learning Data Science Metadata 130

Data Engineering Podcast

DECEMBER 25, 2022

Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan's active metadata capabilities. Missing data? Again, be prepared to have metadata challenges especially. Struggling with broken pipelines? Stale dashboards?

Building 100

Building Metadata Business Intelligence Data Lake 100

RandomTrees

FEBRUARY 6, 2024

Transformation: Shaping Data for the Future: LLMs facilitate standardizing date formats with precision and translation of complex organizational structures into logical database designs, streamline the definition of business rules, automate data cleansing, and propose the inclusion of external data for a more complete analytical view.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

ProjectPro

FEBRUARY 8, 2023

Do ETL and data integration activities seem complex to you? Read this blog to understand everything about AWS Glue that makes it one of the most popular data integration solutions in the industry. Did you know the global big data market will likely reach $268.4 Businesses are leveraging big data now more than ever.

AWS 98

AWS Scala Metadata Data Lake 98

Ascend.io

FEBRUARY 28, 2024

Engineers ensure the availability of clean, structured data, a necessity for AI systems to learn from patterns, make accurate predictions, and automate decision-making processes. Through the design and maintenance of efficient data pipelines , data engineers facilitate the seamless flow and accessibility of data for AI processing.

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

Striim

JUNE 11, 2024

Think of it as a game-changing solution designed to safeguard transactions and maintain customer confidence. Furthermore, AI integrates data from multiple sources, including transaction histories, device metadata, and geolocation information, constructing comprehensive profiles for each user.

MongoDB 52

MongoDB MySQL Utilities Algorithm 52

Workfall

JUNE 12, 2023

It ensures that organisations stay at the forefront by capturing every twist and turn in the data landscape. With CDC by their side, organisations unlock the power of informed decision-making, safeguard data integrity, and enable lightning-fast analytics. CDC also plays a crucial role in data integration and ETL processes.

Telecommunication 80

Telecommunication Metadata Healthcare Finance 80

know.bi

JUNE 15, 2022

Installation size : Apache Hop used to be a lot smaller than Pentaho Data Integration. where a significant size reduction was achieved by cutting out the Big Data plugin. Performance : Apache Hop is a lot faster than Pentaho Data Integration. In contrast, Pentaho Data Integration is open source in name.

Metadata 52

Metadata Data Integration Cloud Storage Project 52

Precisely

MAY 23, 2024

Key takeaways: Quickly adapt to market changes by easily adding new data sources and targets, ensuring your IT landscape evolves at the pace of your business. Gain a competitive edge with real-time data integration, crucial for time-sensitive decisions and actions in fraud detection and customer interactions.

Data Integration 64

Data Integration Metadata Telecommunication Data Governance 64

Data Engineering Podcast

MARCH 22, 2021

What are the factors that you need to consider when deciding whether to implement a CDC system for a given data integration? How does CDC fit into a broader data platform, particularly where there are likely to be other data integration pipelines in operation? What are the barriers to entry?

Data Warehouse 100

Data Warehouse Metadata Data Lake Hadoop 100

phData: Data Engineering

SEPTEMBER 19, 2023

Data modeling methodologies are systematic approaches used to design and define the structure and relationships of data within a system. They provide a framework for organizing and representing data elements, attributes, and relationships. Want to learn more about data governance?

Data Lake 52

Data Lake Process Metadata Data Warehouse 52

Data Engineering Podcast

FEBRUARY 13, 2022

Prophecy provides an easy-to-use visual interface to design & deploy data pipelines on Apache Spark & Apache Airflow. Now all the data users can use software engineering best practices – git, tests and continuous deployment with a simple to use visual designer. How does it work? How does it work?

Building 100

Building Hadoop Data Pipeline Metadata 100

Data Engineering Weekly

DECEMBER 29, 2022

Data catalogs are the most expensive data integration systems you never intended to build. Data Catalog as a passive web portal to display metadata requires significant rethinking to adopt modern data workflow, not just adding “modern” in its prefix. It makes rolling out the data catalogs.

Metadata 62

Metadata Data Warehouse ETL Tools Data Workflow 62

Monte Carlo

FEBRUARY 21, 2023

Despite its prevalence, data can be messy, siloed, ungovernable, and inaccessible—especially to the non-technical employees who rely on it. Enter data fabric: a data management architecture designed to serve the needs of the business, not just those of data engineers. Table of Contents What is a data fabric?

Data Architecture 52

Data Architecture Architecture Metadata Unstructured Data 52

Monte Carlo

FEBRUARY 21, 2023

Despite its prevalence, data can be messy, siloed, ungovernable, and inaccessible—especially to the non-technical employees who rely on it. Enter data fabric: a data management architecture designed to serve the needs of the business, not just those of data engineers. Table of Contents What is a data fabric?

Data Architecture 52

Data Architecture Architecture Metadata Unstructured Data 52

Towards Data Science

FEBRUARY 6, 2024

ERP and CRM systems are designed and built to fulfil a broad range of business processes and functions. This generalisation makes their data models complex and cryptic and require domain expertise. There are a bunch of Data Integration tools that offer a UI to simplify the ingestion.

Systems 76

Systems Raw Data Metadata Data Cleanse 76

Knowledge Hut

JULY 24, 2023

But why do we need data integrity in a DBMS? In addition to data entered by users, database systems typically store large amounts of data. The system stores metadata about data which makes it easier to find and retrieve data. This information is independent, but all this information is related.

Database Design 52

Database Design Relational Database Metadata Database 52

The Modern Data Company

APRIL 15, 2024

The term “Data Product” has become a buzzword, often misused or overstretched. But at its core, a Data Product is much more than just data. This blend ensures that a Data Product is informative, actionable, and adaptable to various needs.

Government 52

Government Metadata Data Data Lake 52

Monte Carlo

NOVEMBER 14, 2023

Databricks announced that Delta tables metadata will also be compatible with the Iceberg format, and Snowflake has also been moving aggressively to integrate with Iceberg. It is designed to be easily queryable with SQL even for large analytic tables (we’re talking petabytes of data). Image courtesy of Dremio.

Data Lake 64

Data Lake Metadata Data Warehouse SQL 64

Snowflake

APRIL 5, 2023

To keep the data model in good form, Snowflake allows primary key, foreign key, unique key, and “not null” constraints. However, we may verify the entity relationship and data integrity in the data model using Snowflake Tasks. What is dimensional data modeling? This gives a Data Vault the upperhand in this case.

Data Warehouse 52

Data Warehouse Healthcare Unstructured Data Metadata 52

Precisely

MAY 23, 2024

Key takeaways: Quickly adapt to market changes by easily adding new data sources and targets, ensuring your IT landscape evolves at the pace of your business. Gain a competitive edge with real-time data integration, crucial for time-sensitive decisions and actions in fraud detection and customer interactions.

Data Integration 52

Data Integration Metadata Telecommunication Data Governance 52

Ascend.io

APRIL 26, 2023

They drain engineering teams of their efficiency while robbing stakeholders of the just-in-time data they need to inform and accelerate their growth. Intelligent data pipelines anticipate and ease the tasks with which engineers drive business change rather than be consumed by busywork and keeping the lights on.

Data Pipeline 52

Data Pipeline Datasets Metadata Data 52

Data Engineering Podcast

JANUARY 15, 2022

Prophecy provides an easy-to-use visual interface to design & deploy data pipelines on Apache Spark & Apache Airflow. Now all the data users can use software engineering best practices – git, tests and continuous deployment with a simple to use visual designer. How does it work? How does it work?

Engineering 100

Engineering Electronics ETL Tools Data Pipeline 100

AltexSoft

FEBRUARY 11, 2023

In the article, we explore the role of a data architect, discuss the responsibilities and required skills, and share what kind of companies may need such a specialist. What is a data architect? To get a better understanding of a data architect’s role, let’s clear up what data architecture is.

Data Architect 87

Data Architect Certification Generalist Big Data 87

Towards Data Science

NOVEMBER 4, 2023

Back in October, I wrote about the rise of the Data Engineer, the role, its challenges, responsibilities, daily routine and how to become successful in this field. The data engineering landscape is constantly changing but major trends seem to remain the same. So here are a few things to consider that can help us answer these questions.

Data Engineering 79

Data Engineering Data Engineer Engineering BI 79

Monte Carlo

FEBRUARY 9, 2023

Pie Insurance , a leading small business insurtech, leverages a data vault 2.0 architecture (with some minor deviations) to achieve their data integration objectives around scalability and use of metadata. “A Foundational – Data vault provides a solid foundation that allows for fast “speed-to-insights.”

Architecture 52

Architecture Raw Data Metadata Data Warehouse 52

Monte Carlo

NOVEMBER 6, 2023

Odds are you’re not starved for data; you’re buried in it. Also known as data catalogs, data discovery tools are specialized software solutions designed to navigate, identify, and visualize data sources within an organization.

Metadata 40

Metadata Unstructured Data Government Data Governance 40

Towards Data Science

MARCH 5, 2024

Summary ∘ Embrace data modeling best practices ∘ Master data operations for cost-effectiveness ∘ Design for efficiency and avoid unnecessary data persistence Disclaimer : BigQuery is a product which is constantly being developed, pricing might change at any time and this article is based on my own experience.

Bytes 67

Bytes Google Cloud Cloud Storage Utilities 67

Databand.ai

AUGUST 30, 2023

DataOps , short for data operations, is an emerging discipline that focuses on improving the collaboration, integration, and automation of data processes across an organization. By using DataOps tools, organizations can break down silos, reduce time-to-insight, and improve the overall quality of their data analytics processes.

Data Cleanse 52

Data Cleanse Data Pipeline Data Ingestion Data Validation 52

Grouparoo

FEBRUARY 9, 2022

The testing process is often performed during the initial setup of a data warehouse after new data sources are added to a pipeline and after data integration and migration projects. ETL testing can be challenging since most ETL systems process large volumes of heterogeneous data. Validate data sources.

Process 52

Process ETL System Data Warehouse Metadata 52

Databand.ai

AUGUST 30, 2023

Faster data processing: By automating data workflows and leveraging modern data processing technologies, DataOps architecture accelerates data ingestion, transformation, and analysis. They include the various databases, applications, APIs, and external systems from which data is collected and ingested.

Architecture 52

Architecture Data Ingestion Data Governance Data Cleanse 52