Simon Späti

MARCH 9, 2021

This post focuses on practical data pipelines with examples from web-scraping real-estates, uploading them to S3 with MinIO, Spark and Delta Lake, adding some Data Science magic with Jupyter Notebooks, ingesting into Data Warehouse Apache Druid, visualising dashboards with Superset and managing everything with Dagster.

Data Engineering 130

Data Engineering Data Engineer Engineering Project 130

Data Engineering Podcast

OCTOBER 28, 2021

He also explains why he started Decodable to address that limitation and the work that he and his team have done to let data engineers build streaming pipelines entirely in SQL. Start trusting your data with Monte Carlo today! Hightouch is the easiest way to sync data into the platforms that your business teams rely on.

Data Pipeline 100

Data Pipeline SQL Data Warehouse Data Lake 100

Towards Data Science

DECEMBER 4, 2023

A Glossary with Use Cases for First-Timers in Data Engineering An happy Data Engineer at work Are you a data engineering rookie interested in knowing more about modern data infrastructures? In this guide Data Engineering meets Formula 1. Data models are built around business needs.

Data Engineering 80

Data Engineering Data Engineer Engineering Data Lake 80

ProjectPro

JANUARY 31, 2023

Want to know who is a business intelligence engineer, what does a business intelligence engineer do, and how these BI engineers turn mountains of data into actionable insights? This growth indicates that the field of business intelligence has a lot of career prospects.

Business Intelligence 52

Business Intelligence Engineering BI Data Warehouse 52

Data Engineering Podcast

MAY 1, 2022

He describes how the platform is architected, the challenges related to selling cloud technologies into enterprise organizations, and how you can adopt Matillion for your own workflows to reduce the maintenance burden of data integration workflows. Struggling with broken pipelines? Missing data? Stale dashboards?

Data Warehouse 100

Data Warehouse Data Integration Cloud Google Cloud 100

Knowledge Hut

DECEMBER 26, 2023

Data Engineering is typically a software engineering role that focuses deeply on data – namely, data workflows, data pipelines, and the ETL (Extract, Transform, Load) process. What is Data Science? What are the roles and responsibilities of a Data Engineer? And many more.

Data Engineering 98

Data Engineering Data Engineer Engineering Pipeline-centric 98

Data Engineering Podcast

DECEMBER 26, 2021

Summary Data Engineering is still a relatively new field that is going through a continued evolution as new technologies are introduced and new requirements are understood. In this episode Maxime Beauchemin returns to revisit what it means to be a data engineer and how the role has changed over the past 5 years.

Data Engineering 100

Data Engineering Data Engineer Engineering Data Warehouse 100

Knowledge Hut

MAY 3, 2023

With companies increasingly relying on data-driven insights to make informed decisions, there has never been a greater need for skilled specialists who can manage and evaluate vast amounts of data. The roles of data analyst and data engineer have emerged as two of the most in-demand professions in today's job market.

Data Engineering 98

Data Engineering Data Engineer Engineering Data Cleanse 98

Data Engineering Podcast

DECEMBER 18, 2022

In this episode Abe Gong brings his experiences with the Great Expectations project and community to discuss the technical and organizational considerations involved in implementing these constraints to your data workflows. Start trusting your data with Monte Carlo today! Start trusting your data with Monte Carlo today!

Metadata 130

Metadata Business Intelligence Data Lake BI 130

Knowledge Hut

DECEMBER 21, 2023

Data Science is the world's most rapidly growing sector and data engineers are at the forefront. In this article, we will understand the promising data engineer career outlook and what it takes to succeed in this role. What is Data Engineering? What are the Data Engineer Career Opportunities?

Data Engineering 52

Data Engineering Data Engineer Engineering MongoDB 52

Monte Carlo

JUNE 14, 2023

In this post, we will help you quickly level up your overall knowledge of data pipeline architecture by reviewing: Table of Contents What is data pipeline architecture? Why is data pipeline architecture important? What is data pipeline architecture? What is data pipeline architecture?

Data Pipeline 52

Data Pipeline Architecture Data Lake Data Warehouse 52

Data Engineering Podcast

DECEMBER 29, 2022

In this episode she shares the strategic and tactical elements of how to make more effective use of the technical and organizational resources that are available to you for getting work done with data. Data engineers struggling with unreliable data need look no further than Monte Carlo, the leading end-to-end Data Observability Platform!

Management 130

Management Metadata Business Intelligence Data Lake 130

Data Engineering Podcast

SEPTEMBER 19, 2021

In this episode Jillian Rowe shares her experience of working in the field and supporting teams of scientists and analysts with the data infrastructure that they need to get their work done. This is a fascinating exploration of the collaboration between data professionals and scientists. Missing data? Stale dashboards?

Data Engineering 100

Data Engineering Data Engineer Engineering Data Lake 100

Data Engineering Podcast

JANUARY 15, 2023

In this episode Srujan Akula explains how the system is implemented and how you can start using it today with your existing data systems. Data engineers struggling with unreliable data need look no further than Monte Carlo, the leading end-to-end Data Observability Platform!

Coding 130

Coding Building PostgreSQL Data Lake 130

Data Engineering Podcast

JANUARY 8, 2023

Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data and analytics leaders, 2023 is your year to sharpen your leadership skills, refine your strategies and lead with purpose. Start trusting your data with Monte Carlo today! Missing data?

PostgreSQL 130

PostgreSQL Data Lake Data Warehouse BI 130

Ascend.io

APRIL 28, 2023

Data pipelines are having a moment — at least, that is, within the data world. That’s because as more and more businesses are adopting a data-driven mindset, the movement of data into and within organizations has never been a bigger priority.

Data Pipeline 52

Data Pipeline Data Data Warehouse Data Engineering 52

Monte Carlo

FEBRUARY 6, 2023

So, you’re planning a cloud data warehouse migration. But be warned, a warehouse migration isn’t for the faint of heart. As you probably already know if you’re reading this, a data warehouse migration is the process of moving data from one warehouse to another. A worthy quest to be sure.

Data Warehouse 52

Data Warehouse AWS Data Validation Data 52

Knowledge Hut

JUNE 30, 2023

The demand for experienced data engineers continuously expands in today's data-driven environment. Books on data engineering serve as essential resources to guide you through the vast terrain of data engineering. What is Data Engineering? Who are Data Engineers?

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Monte Carlo

APRIL 24, 2023

During data ingestion, raw data is extracted from sources and ferried to either a staging server for transformation or directly into the storage level of your data stack—usually in the form of a data warehouse or data lake. There are two primary types of raw data. Missed Nishith’s 5 considerations?

Data Pipeline 52

Data Pipeline Building Data Ingestion BI 52

Monte Carlo

AUGUST 25, 2023

Different vendors offering data warehouses, data lakes, and now data lakehouses all offer their own distinct advantages and disadvantages for data teams to consider. So let’s get to the bottom of the big question: what kind of data storage layer will provide the strongest foundation for your data platform?

Data Lake 52

Data Lake Data Warehouse Unstructured Data Raw Data 52

Knowledge Hut

JUNE 26, 2023

Welcome to the world of data engineering, where the power of big data unfolds. If you're aspiring to be a data engineer and seeking to showcase your skills or gain hands-on experience, you've landed in the right spot. What are Data Engineering Projects?

Data Engineering 98

Data Engineering Data Engineer Coding Project 98

Edureka

FEBRUARY 9, 2023

Azure Data Engineering is a rapidly growing field that involves designing, building, and maintaining data processing systems using Microsoft Azure technologies. Contents: What is the role of an Azure Data Engineer?

Data Engineering 52

Data Engineering Data Engineer Engineering Amazon Web Services 52

Data Engineering Podcast

NOVEMBER 27, 2022

In this episode Wes McKinney shares the ways that Arrow and its related projects are improving the efficiency of data systems and driving their next stage of evolution. Data engineers struggling with unreliable data need look no further than Monte Carlo, the leading end-to-end Data Observability Platform!

Data Process 100

Data Process Process Metadata Business Intelligence 100

Knowledge Hut

APRIL 25, 2023

The tremendous growth in data generation, then the rise in data engineer jobs - there’s no arguing the fact that the big data industry is at its best pace and you, as an aspiring data engineer, have a lot to learn and make out of it - including some tools! What are Data Engineering Tools?

Data Engineering 52

Data Engineering Data Engineer Engineering Google Cloud 52

Data Engineering Podcast

APRIL 26, 2021

One of the most common and widely used methods of access is through a business intelligence dashboard. In this episode Maxime Beauchemin discusses how data engineers can use Superset to provide self service access to data and deliver analytics. RudderStack’s smart customer data pipeline is warehouse-first.

Business Intelligence 100

Business Intelligence Data Warehouse Hadoop Data Pipeline 100

ProjectPro

DECEMBER 7, 2021

Data pipelines are a significant part of the big data domain, and every professional working or willing to work in this field must have extensive knowledge of them. Table of Contents What is a Data Pipeline? The Importance of a Data Pipeline What is an ETL Data Pipeline?

Data Pipeline 52

Data Pipeline Architecture Kafka AWS 52

Data Engineering Podcast

NOVEMBER 20, 2022

Summary The majority of blog posts and presentations about data engineering and analytics assume that the consumers of those efforts are internal business users accessing an environment controlled by the business. Data engineers don’t enjoy writing, maintaining, and modifying ETL pipelines all day, every day.

Systems 130

Systems Metadata Data Pipeline MongoDB 130

Silectis

JANUARY 3, 2021

If you’re new to data engineering or are a practitioner of a related field, such as data science, or business intelligence, we thought it might be helpful to have a handy list of commonly used terms available for you to get up to speed. Big Data Large volumes of structured or unstructured data.

Data Engineering 40

Data Engineering Data Engineer Engineering Hadoop 40

Knowledge Hut

FEBRUARY 27, 2023

A data engineer is a key member of an enterprise data analytics team and is responsible for handling, leading, optimizing, evaluating, and monitoring the acquisition, storage, and distribution of data across the enterprise. Data Engineers indulge in the whole data process, from data management to analysis.

Data Engineering 52

Data Engineering Data Engineer Database-centric Pipeline-centric 52

Data Engineering Podcast

OCTOBER 22, 2021

Summary The market for business intelligence has been going through an evolutionary shift in recent years. One of the driving forces for that change has been the rise of analytics engineering powered by dbt. Are you bored with writing scripts to move data into SaaS tools like Salesforce, Marketo, or Facebook Ads?

Engineering 100

Engineering Business Intelligence BI Data Warehouse 100

Knowledge Hut

JUNE 20, 2023

Let's find out the differences between a data scientist and a machine learning engineer below to make an informative decision. Data Engineer vs Machine Learning Engineer While there are similarities between a data engineer and a machine learning engineer, both play a key role in the technological world.

Machine Learning 52

Machine Learning Data Engineering Data Engineer Engineering 52

Data Engineering Podcast

DECEMBER 25, 2022

Summary Five years of hosting the Data Engineering Podcast has provided Tobias Macey with a wealth of insight into the work of building and operating data systems at a variety of scales and for myriad purposes. Start trusting your data with Monte Carlo today! If this resonates with you, you’re not alone.

Building 100

Building Metadata Business Intelligence Data Lake 100

Knowledge Hut

DECEMBER 28, 2023

Its comprehensive suite of services can handle data at scale. It’s no surprise that the demand for certified Azure data engineers has skyrocketed. Today, Azure Data Engineer certification is an invaluable asset for those looking to excel in the field of data engineering. Who is an Azure Data Engineer?

Certification 52

Certification Data Engineering Data Engineer Engineering 52

dbt Developer Hub

APRIL 30, 2023

My next gig was in consulting where I bootstrapped my way into data engineering and had to learn the whole gamut below. Armed with them, I finally saw the magic of the modern data stack and what problems it could solve. Armed with them, I finally saw the magic of the modern data stack and what problems it could solve.

Data Engineering 64

Data Engineering Data Engineer Engineering Finance 64

Data Engineering Podcast

NOVEMBER 4, 2021

Summary The precursor to widespread adoption of cloud data warehouses was the creation of customer data platforms. Acting as a centralized repository of information about how your customers interact with your organization they drove a wave of analytics about how to improve products based on actual usage data.

Data Warehouse 100

Data Warehouse Business Intelligence ETL System Data Lake 100

Knowledge Hut

DECEMBER 29, 2023

Data Engineering is the most demanding and fruitful career option, as most companies rely on their data to make significant growth decisions. First, they gather all the data from different resources and segregate it into fruitful data sets. The highlight feature of this data engineer toolkit is spark streaming.

Data Engineering 40

Data Engineering Data Engineer Engineering BI 40

Databand.ai

DECEMBER 13, 2022

The Top 25 Data Engineering Influencers and Content Creators on LinkedIn Ryan Yackel 2022-12-13 10:23:19 Interested in data engineering? LinkedIn is full of influencers sharing new ideas and sparking conversations on all kinds of topics, and data engineering is no exception. You’ve come to the right place.

Data Engineering 52

Data Engineering Data Engineer Engineering AWS 52