Data Engineering Podcast

SEPTEMBER 25, 2022

In this episode Tom Baeyens explains their reasons for creating a new syntax for expressing and validating checks for data assets and processes, as well as how to incorporate it into your own projects. Atlan is the metadata hub for your data ecosystem. What are the ways that reliability is measured for data assets?

Building 100

Building Metadata MongoDB MySQL 100

Knowledge Hut

MAY 9, 2024

Data Architect ScyllaDB Data architects play a crucial role in designing an organization's data management framework by assessing data sources and integrating them into a centralized plan. Average Annual Salary of Data Modeler A data modeler can earn $126,811 annually.

Data Science 52

Data Science Data Mining Data Architect Programming Language 52

Data Engineering Podcast

SEPTEMBER 4, 2022

In this episode founders Barr Moses and Lior Gavish rejoin the show to reflect on the evolution and adoption of data observability technologies and the capabilities that are being introduced as the broader ecosystem adopts the practices. Atlan is the metadata hub for your data ecosystem.

IT 100

IT Metadata MongoDB MySQL 100

Data Engineering Podcast

JULY 31, 2022

In this episode Ernie Ostic shares the approach that he and his team at Manta are taking to build a complete view of data lineage across the various data systems in your organization and the useful applications of that information in the work of every data stakeholder. Atlan is the metadata hub for your data ecosystem.

IT 100

IT Metadata MongoDB MySQL 100

Data Engineering Podcast

DECEMBER 19, 2021

Summary Building a well managed data ecosystem for your organization requires a holistic view of all of the producers, consumers, and processors of information. The team at Metaphor are building a fully connected metadata layer to provide both technical and social intelligence about your data.

Systems 100

Systems Building Metadata Data Warehouse 100

Data Engineering Podcast

JULY 10, 2022

They also discuss how they have established a guild system for training and supporting data professionals in the organization. Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities.

Metadata 100

Metadata MongoDB MySQL Data Pipeline 100

Data Engineering Podcast

MAY 22, 2022

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode.

Machine Learning 130

Machine Learning Data Engineering Data Engineer Cloud 130

Data Engineering Podcast

JULY 3, 2022

She also discusses her views on the role of the data lakehouse as a building block for these architectures and the ongoing influence that it will have as the technology matures. Atlan is the metadata hub for your data ecosystem. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Architecture 100

Architecture Metadata MongoDB MySQL 100

Data Engineering Podcast

JUNE 5, 2022

In this episode Sean Falconer explains the idea of a data privacy vault and how this new architectural element can drastically reduce the potential for making a mistake with how you manage regulated or personally identifiable information. Atlan is the metadata hub for your data ecosystem.

Data Security 100

Data Security Metadata MongoDB Scala 100

Data Engineering Podcast

NOVEMBER 20, 2021

In this episode he shares the journey that he and his team at Elementl have taken to understand the state of the ecosystem and how they can provide a foundational layer for a holistic data platform. Start trusting your data with Monte Carlo today! How has that shifted the way that engineers approach pipeline design and composition?

Data Warehouse 100

Data Warehouse Data Lake BI Business Intelligence 100

AltexSoft

AUGUST 29, 2023

In 2010, a transformative concept took root in the realm of data storage and analytics — a data lake. The term was coined by James Dixon , Back-End Java, Data, and Business Intelligence Engineer, and it started a new era in how organizations could store, manage, and analyze their data.

Data Lake 52

Data Lake Architecture IT Amazon Web Services 52

phData: Data Engineering

SEPTEMBER 19, 2023

With many data modeling methodologies and processes available, choosing the right approach can be daunting. This blog will guide you through the best data modeling methodologies and processes for your data lake, helping you make informed decisions and optimize your data management practices. What is a Data Lake?

Data Lake 52

Data Lake Process Metadata Data Warehouse 52

AltexSoft

FEBRUARY 11, 2023

In the article, we explore the role of a data architect, discuss the responsibilities and required skills, and share what kind of companies may need such a specialist. What is a data architect? To get a better understanding of a data architect’s role, let’s clear up what data architecture is.

Data Architect 87

Data Architect Certification Generalist Big Data 87

Snowflake

APRIL 5, 2023

Dimensional models have a specific structure and organize the data to simplify complex data structures, and the denormalized data improves performance. Dimensional model What is the Data Vault modeling approach? The Data Vault modeling approach has a design suited for large, complex data integration.

Data Warehouse 52

Data Warehouse Healthcare Unstructured Data Metadata 52

Silectis

JANUARY 3, 2021

If you’re new to data engineering or are a practitioner of a related field, such as data science, or business intelligence, we thought it might be helpful to have a handy list of commonly used terms available for you to get up to speed. Big Data Large volumes of structured or unstructured data.

Data Engineering 40

Data Engineering Data Engineer Engineering Hadoop 40

Monte Carlo

JUNE 28, 2022

Over the past several years, cloud data lakes like Databricks have gotten so powerful (and popular) that according to Mordor Intelligence , the data lake market is expected to grow from $3.74 Traditionally, data lakes held raw data in its native format and were known for their flexibility, speed, and open source ecosystem.

Data Lake 52

Data Lake Metadata AWS Data Warehouse 52

Knowledge Hut

JULY 26, 2023

Database applications have become vital in current business environments because they enable effective data management, integration, privacy, collaboration, analysis, and reporting. It includes the tools and functionality required to create, store, retrieve, and modify data in a database. What are Database Applications?

Database 52

Database NoSQL Telecommunication MongoDB 52

Data Engineering Podcast

DECEMBER 29, 2022

Rehgan Avon co-founded AlignAI to help address this challenge through a more purposeful platform designed to collect and distribute the knowledge of how and why data is used in a business. Atlan is the metadata hub for your data ecosystem. Missing data? Struggling with broken pipelines? Stale dashboards?

Management 130

Management Metadata Business Intelligence Data Lake 130

Data Engineering Podcast

OCTOBER 23, 2022

In this episode Shane Gibson shares practical advice and insights from his years of experience as a consultant and engineer working in data about how to adopt agile principles in your data work so that you can move faster and provide more value to the business, while building systems that are maintainable and adaptable.

Project 130

Project Metadata MongoDB MySQL 130

Data Engineering Podcast

DECEMBER 25, 2022

In order to condense that acquired knowledge into a format that is useful to everyone Scott Hirleman turns the tables in this episode and asks Tobias about the tactical and strategic aspects of his experiences applying those lessons to the work of building a data platform from scratch. Atlan is the metadata hub for your data ecosystem.

Building 100

Building Metadata Business Intelligence Data Lake 100

Data Engineering Podcast

DECEMBER 18, 2022

In this episode Abe Gong brings his experiences with the Great Expectations project and community to discuss the technical and organizational considerations involved in implementing these constraints to your data workflows. Atlan is the metadata hub for your data ecosystem. Missing data? Struggling with broken pipelines?

Metadata 130

Metadata Business Intelligence Data Lake BI 130

Data Engineering Podcast

NOVEMBER 20, 2022

He explains the constraints that he and his team are faced with and the various challenges that they have overcome to build useful data products on top of a legacy platform where they don’t control the end-to-end systems. Atlan is the metadata hub for your data ecosystem. Closing Announcements Thank you for listening!

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Data Engineering Podcast

JUNE 26, 2022

In this episode Isaac Brodsky explains how the Unfolded platform is architected, their experience joining the team at Foursquare, and how you can start using it for analyzing your spatial data today. Atlan is the metadata hub for your data ecosystem. What are some of the core challenges of working with spatial data?

Datasets 130

Datasets Unstructured Data Metadata MongoDB 130

Data Engineering Podcast

NOVEMBER 27, 2022

Summary The data ecosystem has been growing rapidly, with new communities joining and bringing their preferred programming languages to the mix. This has led to inefficiencies in how data is stored, accessed, and shared across process and system boundaries. Atlan is the metadata hub for your data ecosystem. Missing data?

Data Process 100

Data Process Process Metadata Business Intelligence 100

Cloudera

AUGUST 30, 2022

Agencies are plagued by a wide range of data formats and storage environments—legacy systems, databases, on-premises applications, citizen access portals, innumerable sensors and devices, and more—that all contribute to a siloed ecosystem and the data management challenge. . Modern data architectures.

Data Architecture 77

Data Architecture Architecture Data Lake NoSQL 77

Snowflake

NOVEMBER 2, 2023

Over the years, the technology landscape for data management has given rise to various architecture patterns, each thoughtfully designed to cater to specific use cases and requirements. Customers that require a hybrid of these to support many different tools and languages have built a data lakehouse.

Data Lake 91

Data Lake Data Warehouse Cloud Unstructured Data 91

Data Engineering Podcast

JANUARY 15, 2022

Summary Applications of data have grown well beyond the venerable business intelligence dashboards that organizations have relied on for decades. Prophecy provides an easy-to-use visual interface to design & deploy data pipelines on Apache Spark & Apache Airflow. How does it work?

Engineering 100

Engineering Electronics ETL Tools Data Pipeline 100

AltexSoft

NOVEMBER 10, 2021

Well, there’s a new phenomenon in data management that received the name of a data lakehouse. The pun being obvious, there’s more to that than just a new term: Data lakehouses combine the best features of both data lakes and data warehouses and this post will explain this all. Data warehouse.

Architecture 52

Architecture Data Lake Data Warehouse Metadata 52

AltexSoft

OCTOBER 30, 2021

Data scientists need to have precise domain knowledge to extract the most relevant features for a model to use. Feature engineering is one of the most important and time consuming parts of a data scientist’s job. Data scientists are well versed in algorithms and data-related problems to be able to make a solid choice.

Data Engineering 98

Data Engineering Data Engineer Engineering Machine Learning 98

Monte Carlo

AUGUST 25, 2023

This same structure aids in maintaining data quality and simplifies how users interact with and understand the data. One advantage of data warehouses is their integrated nature. As fully managed solutions, data warehouses are designed to offer ease of construction and operation.

Data Lake 52

Data Lake Data Warehouse Unstructured Data Raw Data 52

Maxime Beauchemin

AUGUST 28, 2017

Historically, people used the pejorative term “ data silo ” to designate issues related to heterogenous analytics that would be scattered across platforms or use incompatible referential. Silos naturally spawn into existence as projects get started, teams drift and inevitably as acquisitions occur.

Data Engineering 174

Data Engineering Data Engineer Engineering Software Engineer 174

ProjectPro

SEPTEMBER 26, 2021

Becoming a Big Data Engineer - The Next Steps Big Data Engineer - The Market Demand An organization’s data science capabilities require data warehousing and mining, modeling, data infrastructure, and metadata management. Most of these are performed by Data Engineers.

Big Data 52

Big Data Data Engineering Data Engineer Engineering 52

Cloudera

JANUARY 21, 2021

With CDW, as an integrated service of CDP, your line of business gets immediate resources needed for faster application launches and expedited data access, all while protecting the company’s multi-year investment in centralized data management, security, and governance.

IT 93

IT Data Lake Data Warehouse Cloud Storage 93

AltexSoft

SEPTEMBER 23, 2021

A data warehouse (DW) is a unified storage for all corporate data. The main purpose of a DW is to enable analytics: It is designed to source raw historical data, apply transformations, and store it in a structured format. Data hub architecture. Enrichment with metadata is another important thing.

Architecture 52

Architecture Data Lake Unstructured Data Data Warehouse 52

AltexSoft

JULY 29, 2022

Developed in 2006 by Doug Cutting and Mike Cafarella to run the web crawler Apache Nutch, it has become a standard for Big Data analytics. According to the study by the Business Application Research Center (BARC), Hadoop found intensive use as. a suitable technology to implement data lake architecture. Small file problem.

Hadoop 59

Hadoop Big Data Google Cloud NoSQL 59

ProjectPro

JANUARY 31, 2022

This is the reason why we need Data Warehouses. What is Snowflake Data Warehouse? A Data Warehouse is a central information repository that enables Data Analytics and Business Intelligence (BI) activities. Snowflake is a data warehousing platform that runs on the cloud.

Architecture 52

Architecture IT Data Warehouse Amazon Web Services 52

AltexSoft

JULY 18, 2023

Whether you’re a data scientist, software engineer, or big data enthusiast, get ready to explore the universe of Apache Spark and learn ways to utilize its strengths to the fullest. Maintained by the Apache Software Foundation, Apache Spark is an open-source, unified engine designed for large-scale data analytics.

Big Data 52

Big Data Data Process Process Hadoop 52