Data Engineering Podcast

AUGUST 28, 2022

Summary The dream of every engineer is to automate all of their tasks. For data engineers, this is a monumental undertaking. Orchestration engines are one step in that direction, but they are not a complete solution. Atlan is the metadata hub for your data ecosystem.

Data Engineering 100

Data Engineering Data Engineer MongoDB Metadata 100

Knowledge Hut

DECEMBER 28, 2023

Human society in 2023 is a digital world, and its fuel - its currency - is data. Today, organizations seek skilled professionals who can harness data’s power to drive informed decisions. As technology evolves, cloud platforms have emerged as the cornerstone of modern data management. Who is an Azure Data Engineer?

Certification 52

Certification Data Engineering Data Engineer Engineering 52

Knowledge Hut

JUNE 26, 2023

Welcome to the world of data engineering, where the power of big data unfolds. If you're aspiring to be a data engineer and seeking to showcase your skills or gain hands-on experience, you've landed in the right spot. What are Data Engineering Projects?

Data Engineering 98

Data Engineering Data Engineer Coding Project 98

Hepta Analytics

FEBRUARY 14, 2022

DE Zoomcamp 2.2.1 – Introduction to Workflow Orchestration Following last weeks blog , we move to data ingestion. We already had a script that downloaded a csv file, processed the data and pushed the data to postgres database. This week, we got to think about our data ingestion design. It is called Idempotency.

Data Ingestion 52

Data Ingestion Data Engineering Data Engineer Engineering 52

Silectis

JANUARY 3, 2021

If you’re new to data engineering or are a practitioner of a related field, such as data science, or business intelligence, we thought it might be helpful to have a handy list of commonly used terms available for you to get up to speed. Big Data Large volumes of structured or unstructured data.

Data Engineering 40

Data Engineering Data Engineer Engineering Hadoop 40

Knowledge Hut

JUNE 30, 2023

The demand for experienced data engineers continuously expands in today's data-driven environment. Books on data engineering serve as essential resources to guide you through the vast terrain of data engineering. What is Data Engineering? Who are Data Engineers?

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Knowledge Hut

APRIL 25, 2023

The tremendous growth in data generation, then the rise in data engineer jobs - there’s no arguing the fact that the big data industry is at its best pace and you, as an aspiring data engineer, have a lot to learn and make out of it - including some tools! What are Data Engineering Tools?

Data Engineering 52

Data Engineering Data Engineer Engineering Google Cloud 52

Monte Carlo

SEPTEMBER 1, 2022

Data engineering is a relatively young profession, even for the tech space. To put it in perspective, front-end engineering has twice the number of years in industry maturity. That’s because not only is data engineering talent in high demand, but the very nature of handling big data necessitates prioritization and focus.

Data Engineering 52

Data Engineering Data Engineer Engineering Datasets 52

Data Engineering Podcast

MAY 8, 2022

Summary Many of the events, ideas, and objects that we try to represent through data have a high degree of connectivity in the real world. TigerGraph is a leading database that offers a highly scalable and performant native graph engine for powering graph analytics and machine learning. Missing data? Stale dashboards?

Database 100

Database Data Lake BI Business Intelligence 100

Data Engineering Podcast

AUGUST 21, 2022

Summary The position of Chief Data Officer (CDO) is relatively new in the business world and has not been universally adopted. In this episode Tracy Daniels, CDO of Truist, shares her journey into the position, her responsibilities, and her relationship to the data professionals in her organization.

Metadata 100

Metadata MongoDB MySQL Data Lake 100

Knowledge Hut

DECEMBER 29, 2023

The market for analytics is flourishing, as is the usage of the phrase Data Science. Professionals from a variety of disciplines use data in their day-to-day operations and feel the need to understand cutting-edge technology to get maximum insights from the data, therefore contributing to the growth of the organization.

Data Science 98

Data Science Data Mining Deep Learning Programming Language 98

Databand.ai

DECEMBER 13, 2022

The Top 25 Data Engineering Influencers and Content Creators on LinkedIn Ryan Yackel 2022-12-13 10:23:19 Interested in data engineering? LinkedIn is full of influencers sharing new ideas and sparking conversations on all kinds of topics, and data engineering is no exception. You’ve come to the right place.

Data Engineering 52

Data Engineering Data Engineer Engineering AWS 52

Data Engineering Podcast

DECEMBER 4, 2021

Summary Hiring data professionals is challenging for a multitude of reasons, and as with every interview process there is a potential for bias to creep in. The whole process of hiring is an important organizational skill to cultivate and this is an interesting exploration of the specific challenges involved in finding data professionals.

Data Warehouse 100

Data Warehouse Data Lake BI Business Intelligence 100

Monte Carlo

JANUARY 5, 2024

You know what they always say: data lakehouse architecture is like an onion. …ok, Data lakehouse architecture combines the benefits of data warehouses and data lakes, bringing together the structure and performance of a data warehouse with the flexibility of a data lake. But they should!

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

Knowledge Hut

APRIL 26, 2024

In the age of big data processing, how to store these terabytes of data surfed over the internet was the key concern of companies until 2010. Now that the issue of storage of big data has been solved successfully by Hadoop and various other frameworks, the concern has shifted to processing these data.

Data Science 52

Data Science Machine Learning Hadoop Programming Language 52

Monte Carlo

JANUARY 5, 2024

You know what they always say: data lakehouse architecture is like an onion. …ok, Data lakehouse architecture combines the benefits of data warehouses and data lakes, bringing together the structure and performance of a data warehouse with the flexibility of a data lake. But they should!

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

Data Engineering Podcast

NOVEMBER 4, 2021

Summary The precursor to widespread adoption of cloud data warehouses was the creation of customer data platforms. Acting as a centralized repository of information about how your customers interact with your organization they drove a wave of analytics about how to improve products based on actual usage data. Missing data?

Data Warehouse 100

Data Warehouse Business Intelligence ETL System Data Lake 100

Cloudera

JUNE 18, 2022

Cloudera customers run some of the biggest data lakes on earth. These lakes power mission critical large scale data analytics, business intelligence (BI), and machine learning use cases, including enterprise data warehouses. On data warehouses and data lakes.

Data Lake 93

Data Lake Data Warehouse BI SQL 93

Data Engineering Podcast

OCTOBER 28, 2021

Summary Streaming data systems have been growing more capable and flexible over the past few years. In this episode Eric Sammer discusses the shortcomings of the current set of streaming engines and how they force engineers to work at an extremely low level of abstraction. Missing data? Struggling with broken pipelines?

Data Pipeline 100

Data Pipeline SQL Data Warehouse Data Lake 100

Data Engineering Podcast

SEPTEMBER 11, 2022

Summary Data engineering systems are complex and interconnected with myriad and often opaque chains of dependencies. In order to turn this into a tractable problem one approach is to define and enforce contracts between producers and consumers of data. Atlan is the metadata hub for your data ecosystem.

Systems 100

Systems Metadata Building MongoDB 100

Data Engineering Podcast

DECEMBER 21, 2020

Summary Data governance is a term that encompasses a wide range of responsibilities, both technical and process oriented. One of the more complex aspects is that of access control to the data assets that an organization is responsible for managing. If you hand a book to a new data engineer, what wisdom would you add to it?

Data Governance 130

Data Governance Government Data Lake Banking 130

Data Engineering Podcast

SEPTEMBER 4, 2022

Summary Data observability is a product category that has seen massive growth and adoption in recent years. Monte Carlo is in the vanguard of companies who have been enabling data teams to observe and understand their complex data systems. Atlan is the metadata hub for your data ecosystem.

IT 100

IT Metadata MongoDB MySQL 100

Data Engineering Podcast

OCTOBER 9, 2022

Summary The core of any data platform is the centralized storage and processing layer. For many that is a data warehouse, but in order to support a diverse and constantly changing set of uses and technologies the data lakehouse is a paradigm that offers a useful balance of scale and cost, with performance and ease of use.

Metadata 100

Metadata AWS MongoDB MySQL 100

Data Engineering Podcast

DECEMBER 11, 2021

In order to make it easier for developers to build customer profiles in a way that respects their privacy Serge Huber helped to create the Apache Unomi framework as an open source customer data platform. Missing data? Start trusting your data with Monte Carlo today! Struggling with broken pipelines? Stale dashboards?

Data Warehouse 100

Data Warehouse Raw Data Data Lake BI 100

Data Engineering Podcast

JANUARY 1, 2022

Summary This has been an active year for the data ecosystem, with a number of new product categories and substantial growth in existing areas. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Missing data?

Data Warehouse 100

Data Warehouse Data Lake SQL Hadoop 100

Data Engineering Podcast

DECEMBER 19, 2021

Summary Building a well managed data ecosystem for your organization requires a holistic view of all of the producers, consumers, and processors of information. The team at Metaphor are building a fully connected metadata layer to provide both technical and social intelligence about your data. Missing data?

Systems 100

Systems Building Metadata Data Warehouse 100

Data Engineering Podcast

NOVEMBER 10, 2021

Summary A significant source of friction and wasted effort in building and integrating data management systems is the fragmentation of metadata across various tools. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform.

Metadata 100

Metadata Data Warehouse Data Lake BI 100

Data Engineering Podcast

NOVEMBER 20, 2021

Summary The technology for scaling storage and processing of data has gone through massive evolution over the past decade, leaving us with the ability to work with massive datasets at the cost of massive complexity. Missing data? Start trusting your data with Monte Carlo today! Struggling with broken pipelines? Stale dashboards?

Data Warehouse 100

Data Warehouse Data Lake BI Business Intelligence 100

Monte Carlo

AUGUST 31, 2023

Data is a priority for your CEO, as it often is for digital-first companies, and she is fluent in the latest and greatest business intelligence tools. But what happens when the data is wrong? What about a frantic email from your CTO about “duplicate data” in a business intelligence dashboard?

Data Lake 52

Data Lake Data Pipeline Unstructured Data Data Warehouse 52

Scott Logic

APRIL 22, 2024

Amazon’s goal in this space is to develop AI tools that are able to be used by non-technical users, so that organisations are not hamstrung by any bottlenecks caused by the number of data scientists they have available. In particular, with SageMaker Canvas, it’s possible to create a machine learning model entirely graphically.

BI 97

BI Software Engineer Software Engineering Algorithm 97

Data Engineering Podcast

JANUARY 11, 2021

Summary As data professionals we have a number of tools available for storing, processing, and analyzing data. We also have tools for collaborating on software and analysis, but collaborating on data is still an underserved capability. And don’t forget to thank them for their continued support of this show!

PostgreSQL 100

PostgreSQL Python Data Lake Computer Science 100

Ascend.io

AUGUST 31, 2023

In the dynamic world of data, many professionals are still fixated on traditional patterns of data warehousing and ETL, even while their organizations are migrating to the cloud and adopting cloud-native data services. This unchanging schema forms the foundation for all queries and business intelligence.

Data Lake 52

Data Lake ETL Tools Data Warehouse Data Pipeline 52

DataKitchen

AUGUST 3, 2021

The data mesh design pattern breaks giant, monolithic enterprise data architectures into subsystems or domains, each managed by a dedicated team. DataOps helps the data mesh deliver greater business agility by enabling decentralized domains to work in concert. . But first, let’s define the data mesh design pattern.

Pharmaceutical 98

Pharmaceutical Data Lake Data Architecture Scala 98

Knowledge Hut

DECEMBER 22, 2023

It was released as a standalone product in July 2015 after adding more features including enterprise-level data connectivity and security options, apart from its original Excel features like Power Query, Power Pivot, and Power View. Microsoft developed it and combines business analytics, data visualization, and best practices.

BI 98

BI Business Intelligence Machine Learning SQL 98

Knowledge Hut

DECEMBER 26, 2023

In recent years, with the advent of technology, data has been considered to be a valuable asset in both large-scale and small-scale organizations. Data as a resource requires skilled professionals to be collected, interpreted, and stored safely. The demand for data scientists and career prospects has skyrocketed in the job market.

Data Science 52

Data Science Scala Programming Language R (Programming) 52

Snowflake

JUNE 20, 2023

In this edition, we’re digging into cybersecurity with Parth Shah, Co-Founder and Head of Product at Dassana , as he discusses the power of operationalizing security data, why you need to consider security data lakes, and how Snowflake gave Dassana an agility upgrade. What’s the coolest thing you’re doing with data?

Data Lake 57

Data Lake Business Intelligence Cloud Technology 57

ProjectPro

JUNE 9, 2021

This blog on Data Science vs. Data Engineering presents a detailed comparison between the two domains. vs. What does a Data Engineer do? Are you a Data Scientist or a Data Engineer? Is data engineering more important than data science?

Data Engineering 52

Data Engineering Data Engineer Engineering Amazon Web Services 52