ProjectPro

DECEMBER 7, 2021

Data pipelines are a significant part of the big data domain, and every professional working or willing to work in this field must have extensive knowledge of them. As data is expanding exponentially, organizations struggle to harness digital information's power for different business use cases.

Data Pipeline 52

Data Pipeline Architecture Kafka AWS 52

Data Engineering Podcast

APRIL 26, 2021

Summary The reason for collecting, cleaning, and organizing data is to make it usable by the organization. One of the most common and widely used methods of access is through a business intelligence dashboard. Modern Data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Business Intelligence 100

Business Intelligence Data Warehouse Hadoop Data Pipeline 100

Data Engineering Podcast

OCTOBER 22, 2021

Summary The market for business intelligence has been going through an evolutionary shift in recent years. Lightdash has fully embraced that shift by building an entire open source business intelligence framework that is powered by dbt models. And don’t forget to thank them for their continued support of this show!

Engineering 100

Engineering Business Intelligence BI Data Warehouse 100

Data Engineering Podcast

NOVEMBER 4, 2021

Summary The precursor to widespread adoption of cloud data warehouses was the creation of customer data platforms. Acting as a centralized repository of information about how your customers interact with your organization they drove a wave of analytics about how to improve products based on actual usage data.

Data Warehouse 100

Data Warehouse Business Intelligence ETL System Data Lake 100

Monte Carlo

JUNE 2, 2024

Every day, an uncountable amount of data flows through millions of businesses. Data Infrastructure Engineers are the professionals who ensure that this data flows smoothly and reliably. But what does a data infrastructure engineer do exactly? Table of Contents What is a Data Infrastructure Engineer?

Engineering 52

Engineering Amazon Web Services Data Science AWS 52

Data Engineering Podcast

DECEMBER 4, 2021

Summary Hiring data professionals is challenging for a multitude of reasons, and as with every interview process there is a potential for bias to creep in. The whole process of hiring is an important organizational skill to cultivate and this is an interesting exploration of the specific challenges involved in finding data professionals.

Data Warehouse 100

Data Warehouse Data Lake BI Business Intelligence 100

Data Engineering Podcast

JULY 17, 2022

Summary There are extensive and valuable data sets that are available outside the bounds of your organization. Whether that data is public, paid, or scraped it requires investment and upkeep to acquire and integrate it with your systems. Atlan is the metadata hub for your data ecosystem.

Data Management 130

Data Management Management Metadata MongoDB 130

Data Engineering Podcast

JULY 3, 2022

Summary The ecosystem for data tools has been going through rapid and constant evolution over the past several years. These technological shifts have brought about corresponding changes in data and platform architectures for managing data and analytical workflows. Atlan is the metadata hub for your data ecosystem.

Architecture 100

Architecture Metadata MongoDB MySQL 100

Data Engineering Podcast

DECEMBER 19, 2021

Summary Building a well managed data ecosystem for your organization requires a holistic view of all of the producers, consumers, and processors of information. The team at Metaphor are building a fully connected metadata layer to provide both technical and social intelligence about your data. Struggling with broken pipelines?

Systems 100

Systems Building Metadata Data Warehouse 100

Data Engineering Podcast

JUNE 19, 2022

Summary Data analysis is a valuable exercise that is often out of reach of non-technical users as a result of the complexity of data systems. Atlan is the metadata hub for your data ecosystem. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Metadata 130

Metadata Unstructured Data MongoDB MySQL 130

Data Engineering Podcast

JANUARY 1, 2022

Summary This has been an active year for the data ecosystem, with a number of new product categories and substantial growth in existing areas. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Missing data?

Data Warehouse 100

Data Warehouse Data Lake SQL Hadoop 100

Data Engineering Podcast

JULY 10, 2022

Summary Building a data platform is a journey, not a destination. In this episode Inbar Yogev and Lior Winner share the journey that they and their teams at Riskified have been on for their data platform. They also discuss how they have established a guild system for training and supporting data professionals in the organization.

Metadata 100

Metadata MongoDB MySQL Data Pipeline 100

Data Engineering Podcast

DECEMBER 11, 2021

In order to make it easier for developers to build customer profiles in a way that respects their privacy Serge Huber helped to create the Apache Unomi framework as an open source customer data platform. Struggling with broken pipelines? Missing data? Start trusting your data with Monte Carlo today! Stale dashboards?

Data Warehouse 100

Data Warehouse Raw Data Data Lake BI 100

Advancing Analytics: Data Engineering

JUNE 21, 2019

Ah the ETL (Extract-Transform-Load) Window, the schedule by which the Business Intelligence developer sets their clock, the nail-biting nightly period during which the on-call support hopes their phone won’t ring. It’s a cornerstone of the data warehousing approach… and we shouldn’t have one. There, I said it.

Data Warehouse 52

Data Warehouse Business Intelligence Data Data Validation 52

Data Engineering Podcast

JUNE 12, 2022

Summary Building a well rounded and effective data team is an iterative process, and the first hire can set the stage for future success or failure. Trupti Natu has been the first data hire multiple times and gone through the process of building teams across the different stages of growth.

Metadata 100

Metadata Unstructured Data Business Intelligence MongoDB 100

Knowledge Hut

DECEMBER 26, 2023

According to the World Economic Forum, the amount of data generated per day will reach 463 exabytes (1 exabyte = 10 9 gigabytes) globally by the year 2025. Thus, almost every organization has access to large volumes of rich data and needs “experts” who can generate insights from this rich data.

Data Science 97

Data Science BI Business Intelligence Data Mining 97

Data Engineering Podcast

OCTOBER 9, 2022

Summary The core of any data platform is the centralized storage and processing layer. For many that is a data warehouse, but in order to support a diverse and constantly changing set of uses and technologies the data lakehouse is a paradigm that offers a useful balance of scale and cost, with performance and ease of use.

Metadata 100

Metadata AWS MongoDB MySQL 100

Data Engineering Podcast

JUNE 5, 2022

Summary The best way to make sure that you don’t leak sensitive data is to never have it in the first place. The team at Skyflow decided that the second best way is to build a storage system dedicated to securely managing your sensitive information and making it easy to integrate with your applications and data systems.

Data Security 100

Data Security Metadata MongoDB Scala 100

Data Engineering Podcast

NOVEMBER 10, 2021

Summary A significant source of friction and wasted effort in building and integrating data management systems is the fragmentation of metadata across various tools. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform.

Metadata 100

Metadata Data Warehouse Data Lake BI 100

Ascend.io

AUGUST 31, 2023

In the dynamic world of data, many professionals are still fixated on traditional patterns of data warehousing and ETL, even while their organizations are migrating to the cloud and adopting cloud-native data services. This unchanging schema forms the foundation for all queries and business intelligence.

Data Lake 52

Data Lake ETL Tools Data Warehouse Data Pipeline 52

Knowledge Hut

DECEMBER 29, 2023

The market for analytics is flourishing, as is the usage of the phrase Data Science. Professionals from a variety of disciplines use data in their day-to-day operations and feel the need to understand cutting-edge technology to get maximum insights from the data, therefore contributing to the growth of the organization.

Data Science 98

Data Science Data Mining Deep Learning Programming Language 98

Data Engineering Podcast

NOVEMBER 20, 2021

Summary The technology for scaling storage and processing of data has gone through massive evolution over the past decade, leaving us with the ability to work with massive datasets at the cost of massive complexity. Struggling with broken pipelines? Missing data? Start trusting your data with Monte Carlo today!

Data Warehouse 100

Data Warehouse Data Lake BI Business Intelligence 100

Data Engineering Podcast

DECEMBER 21, 2020

Summary Data governance is a term that encompasses a wide range of responsibilities, both technical and process oriented. One of the more complex aspects is that of access control to the data assets that an organization is responsible for managing. If you hand a book to a new data engineer, what wisdom would you add to it?

Data Governance 130

Data Governance Government Data Lake Banking 130

Data Engineering Podcast

JULY 31, 2022

Summary Data lineage is the roadmap for your data platform, providing visibility into all of the dependencies for any report, machine learning model, or data warehouse table that you are working with. Atlan is the metadata hub for your data ecosystem.

IT 100

IT Metadata MongoDB MySQL 100

Data Engineering Podcast

MAY 8, 2022

Summary Many of the events, ideas, and objects that we try to represent through data have a high degree of connectivity in the real world. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Struggling with broken pipelines?

Database 100

Database Data Lake BI Business Intelligence 100

Knowledge Hut

JUNE 30, 2023

The demand for experienced data engineers continuously expands in today's data-driven environment. Books on data engineering serve as essential resources to guide you through the vast terrain of data engineering. What is Data Engineering? Who are Data Engineers? Who are Data Engineers?

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Data Engineering Podcast

MAY 15, 2022

Summary Designing a data platform is a complex and iterative undertaking which requires accounting for many conflicting needs. Designing a platform that relies on a data lake as its central architectural tenet adds additional layers of difficulty. Struggling with broken pipelines? Missing data? Stale dashboards?

Data Lake 100

Data Lake Building BI Architecture 100

Monte Carlo

AUGUST 31, 2023

Data is a priority for your CEO, as it often is for digital-first companies, and she is fluent in the latest and greatest business intelligence tools. But what happens when the data is wrong? What about a frantic email from your CTO about “duplicate data” in a business intelligence dashboard?

Data Lake 52

Data Lake Data Pipeline Unstructured Data Data Warehouse 52

Data Engineering Podcast

JANUARY 11, 2021

Summary As data professionals we have a number of tools available for storing, processing, and analyzing data. We also have tools for collaborating on software and analysis, but collaborating on data is still an underserved capability. And don’t forget to thank them for their continued support of this show!

PostgreSQL 100

PostgreSQL Python Data Lake Computer Science 100

Knowledge Hut

APRIL 25, 2023

The tremendous growth in data generation, then the rise in data engineer jobs - there’s no arguing the fact that the big data industry is at its best pace and you, as an aspiring data engineer, have a lot to learn and make out of it - including some tools! While they go about it - enter big data data engineer tools.

Data Engineering 52

Data Engineering Data Engineer Engineering Google Cloud 52

Monte Carlo

MARCH 28, 2023

What is stale data? Stale data is data that hasn’t been updated at the frequency interval required for its productive use. Now it’s more about we have such a breadth of data—how do we manage it and make it more easily discoverable and more easily searchable? How does stale data happen and what are the consequences?

IT 52

IT Machine Learning Data Business Intelligence 52

DataKitchen

AUGUST 3, 2021

The data mesh design pattern breaks giant, monolithic enterprise data architectures into subsystems or domains, each managed by a dedicated team. DataOps helps the data mesh deliver greater business agility by enabling decentralized domains to work in concert. . But first, let’s define the data mesh design pattern.

Pharmaceutical 98

Pharmaceutical Data Lake Data Architecture Scala 98

Silectis

JANUARY 3, 2021

If you’re new to data engineering or are a practitioner of a related field, such as data science, or business intelligence, we thought it might be helpful to have a handy list of commonly used terms available for you to get up to speed. Big Data Large volumes of structured or unstructured data.

Data Engineering 40

Data Engineering Data Engineer Engineering Hadoop 40

AltexSoft

AUGUST 29, 2023

In 2010, a transformative concept took root in the realm of data storage and analytics — a data lake. The term was coined by James Dixon , Back-End Java, Data, and Business Intelligence Engineer, and it started a new era in how organizations could store, manage, and analyze their data. What is a data lake?

Data Lake 52

Data Lake Architecture IT Amazon Web Services 52

Striim

FEBRUARY 10, 2023

According to a recent study by KX, US businesses could see a total revenue uplift of $2.6 trillion through investment in real-time data analytics. From telecommunication to retail, businesses are harnessing the power of data analytics to optimize operations and drive growth.

Data Analytics 52

Data Analytics Telecommunication Insurance Google Cloud 52

Knowledge Hut

JUNE 26, 2023

Welcome to the world of data engineering, where the power of big data unfolds. If you're aspiring to be a data engineer and seeking to showcase your skills or gain hands-on experience, you've landed in the right spot. Therefore, the greatest thing you can do as a novice is to work on some real-time data engineering initiatives.

Data Engineering 98

Data Engineering Data Engineer Coding Project 98

Knowledge Hut

FEBRUARY 27, 2023

A data engineer is a key member of an enterprise data analytics team and is responsible for handling, leading, optimizing, evaluating, and monitoring the acquisition, storage, and distribution of data across the enterprise. Data Engineers indulge in the whole data process, from data management to analysis.

Data Engineering 52

Data Engineering Data Engineer Database-centric Pipeline-centric 52