Simon Späti

MARCH 9, 2021

This post focuses on practical data pipelines with examples from web-scraping real-estates, uploading them to S3 with MinIO, Spark and Delta Lake, adding some Data Science magic with Jupyter Notebooks, ingesting into Data Warehouse Apache Druid, visualising dashboards with Superset and managing everything with Dagster.

Data Engineering 189

Data Engineering Data Engineer Engineering Project 189

Data Engineering Weekly

FEBRUARY 18, 2024

Our hope is only with the amazing community of data practitioners who constantly support us. One thing I learned while writing Data Engineering Weekly is that persistence and consistency are the keys to success. We are so over the Big Data Era to Modern Data Stack. Elevate your data skills!

Data Engineering 62

Data Engineering Data Engineer Engineering Data 62

Knowledge Hut

MARCH 28, 2024

Data engineering is one of them. According to AnalytixLabs , the data science market is expected to be worth USD 230.80 All these numbers point to one thing–increased job roles and careers, especially when we talk about data engineering jobs in Azure, which are on the rise every year. Let’s get started.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Data Engineering Weekly

MARCH 10, 2024

Big thanks to our insightful speakers, Hareshkumar Selvakumar - Talks about his work on Data Products for PayPal. Pradheep Arjunan - Shared insights on AZ's journey from on-prem to the cloud data warehouses. Data engineers build the systems that store and process sensitive information.

Data Engineering 52

Data Engineering Data Engineer Engineering Datasets 52

Towards Data Science

FEBRUARY 24, 2023

The Chaos Data-Engineering Manifesto Another lesson we can learn from software engineers: break stuff to make it more reliable. While this idea isn’t completely foreign to data engineering, it can certainly be described as an extremely uncommon practice. Data is different. It’s terrifying. I’m afraid so.

Data Engineering 98

Data Engineering Data Engineer Engineering Software Engineer 98

Towards Data Science

AUGUST 19, 2023

How I made the transition to an analytics engineer Photo by Campaign Creators on Unsplash A few years ago, I was at a point where I was feeling unfulfilled in my career. I had been working in data engineering for three years and the initial excitement of starting in the world of tech had faded.

Data Engineering 86

Data Engineering Data Engineer Engineering SQL 86

Databand.ai

JUNE 28, 2023

Data Pipeline Observability: A Model For Data Engineers Eitan Chazbani June 29, 2023 Data pipeline observability is your ability to monitor and understand the state of a data pipeline at any time. We believe the world’s data pipelines need better data observability. To measure, but not track.

Data Pipeline 98

Data Pipeline Data Engineering Data Engineer Engineering 98

Simon Späti

MARCH 9, 2021

This post focuses on practical data pipelines with examples from web-scraping real-estates, uploading them to S3 with MinIO, Spark and Delta Lake, adding some Data Science magic with Jupyter Notebooks, ingesting into Data Warehouse Apache Druid, visualising dashboards with Superset and managing everything with Dagster.

Data Engineering 130

Data Engineering Data Engineer Engineering Project 130

Analytics Vidhya

FEBRUARY 21, 2023

This is where data warehousing is a critical component of any business, allowing companies to store and manage vast amounts of data. It provides the necessary foundation for businesses to […] The post Understanding the Basics of Data Warehouse and its Structure appeared first on Analytics Vidhya.

Data Warehouse 206

Data Warehouse IT Data Collection Data 206

Analytics Vidhya

FEBRUARY 2, 2023

Now, businesses are looking for different types of data storage to store and manage their data effectively. Organizations can collect millions of data, but if they’re lacking in storing that data, those efforts […] The post A Comprehensive Guide to Data Lake vs. Data Warehouse appeared first on Analytics Vidhya.

Data Lake 202

Data Lake Data Warehouse Data Storage Data 202

Knowledge Hut

OCTOBER 31, 2023

Two of the most well-known cloud service providers, Amazon Web Services (AWS) and Microsoft Azure, provide reliable data engineering solutions. Often, aspiring data engineers must choose between two options: AWS data engineer or Azure data engineer.

AWS 52

AWS Data Engineering Data Engineer Engineering 52

Knowledge Hut

MARCH 13, 2024

Wondering what is a big data engineer? As the name suggests, Big Data is associated with ‘big’ data, which hints at something big in the context of data. Big data forms one of the pillars of data science. Big data has been a hot topic in the IT sector for quite a long time.

Big Data 52

Big Data Data Engineering Data Engineer Engineering 52

Knowledge Hut

MARCH 13, 2024

Wondering what is a big data engineer? As the name suggests, Big Data is associated with ‘big’ data, which hints at something big in the context of data. Big data forms one of the pillars of data science. Big data has been a hot topic in the IT sector for quite a long time.

Big Data 52

Big Data Data Engineering Data Engineer Engineering 52

Data Science Blog: Data Engineering

DECEMBER 23, 2022

Warum Data Engineering der Data Science in Bedeutung und Berufschancen längst die Show stiehlt, dabei selbst ebenso einem stetigen Wandel unterliegt. Was ein Data Engineer wirklich können muss Der Data Scientist als sexiest Job des 21. Data Engineer, der gefragteste Job des 21sten Jahrhunderts?

Data Engineering 52

Data Engineering Data Engineer Engineering Business Intelligence 52

Data Engineering Weekly

OCTOBER 1, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. I wish more data engineers would start sharing their on-call experience soon.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Pipeline 52

Towards Data Science

DECEMBER 1, 2023

Architecture Referencing Joe Reis’s “Fundamentals of Data Engineering,” let’s review our project in alignment with the defined stages of the data lifecycle: Data Engineering Lifecycle [1] Data Generation —  Google Calendar, Fivetran If you’re a Google Calendar user, chances are you’ve accumulated a wealth of data there.

Data Engineering 76

Data Engineering Data Engineer Project Engineering 76

Data Engineering Weekly

MARCH 24, 2024

Dive into Spyne's experience with: - Their search for query acceleration with pre-aggregations and caching - Developing new functionality with Open AI - Optimizing query cost with their data warehouse [link] Suresh Hasuni: Cost Optimization Strategies for Scalable Data Lakehouse Cost is the major concern as the adoption of data lakes increases.

Data Engineering 62

Data Engineering Data Engineer Engineering Metadata 62

U-Next

MARCH 1, 2023

Introduction Data Engineer is responsible for managing the flow of data to be used to make better business decisions. A solid understanding of relational databases and SQL language is a must-have skill, as an ability to manipulate large amounts of data effectively. In 2022, data engineering will hold a share of 29.8%

Data Engineering 75

Data Engineering Data Engineer Non-relational Database Engineering 75

Ascend.io

FEBRUARY 28, 2024

The rise of generative AI is changing more than just technology; it’s reshaping our professional landscapes — and yes, data engineering is directly experiencing the impact. How does AI recalibrate the workload and priorities of data teams? How can data engineers harness the power of AI?

Data Engineering 52

Data Engineering Data Engineer Engineering Metadata 52

ProjectPro

FEBRUARY 16, 2023

The demand for skilled data engineers who can build, maintain, and optimize large data infrastructures does not seem to slow down any sooner. At the heart of these data engineering skills lies SQL that helps data engineers manage and manipulate large amounts of data. use SQL, compared to 61.7%

Data Engineering 52

Data Engineering Data Engineer SQL Engineering 52

Acceldata

FEBRUARY 16, 2023

Why Data Engineers Must Manage Data Pipelines, Not Data Warehouses

Data Warehouse 52

Data Warehouse Data Pipeline Data Engineering Data Engineer 52

Knowledge Hut

NOVEMBER 28, 2023

The contemporary world experiences a huge growth in cloud implementations, consequently leading to a rise in demand for data engineers and IT professionals who are well-equipped with a wide range of application and process expertise. Data Engineer certification will aid in scaling up you knowledge and learning of data engineering.

Data Engineering 52

Data Engineering Data Engineer Engineering Generalist 52

Towards Data Science

MAY 8, 2023

Data Warehouse Dimensional Modeling Design 101 Continue reading on Towards Data Science »

Data Warehouse 72

Data Warehouse Data Engineering Data Engineer Building 72

Data Engineering Weekly

FEBRUARY 26, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Perhaps merge Data LakeHouse & Data Warehouse in the next edition?

Data Engineering 57

Data Engineering Data Engineer Engineering Raw Data 57

Knowledge Hut

JUNE 26, 2023

Welcome to the world of data engineering, where the power of big data unfolds. If you're aspiring to be a data engineer and seeking to showcase your skills or gain hands-on experience, you've landed in the right spot. What are Data Engineering Projects?

Data Engineering 98

Data Engineering Data Engineer Coding Project 98

Data Engineering Weekly

AUGUST 20, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork, and SQL grunt work out of building complete customer profiles, so you can quickly ship actionable, enriched data to every downstream team. See how it works today. 📣 Exciting news!

Data Engineering 52

Data Engineering Data Engineer Engineering Data 52

Data Engineering Weekly

APRIL 16, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make collecting data from every application, website, and SaaS platform easy, then activating it in your warehouse and business tools. 🍒 We'll send exclusive The Data Stack Show swag just for participating!

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Data Engineering Weekly

NOVEMBER 5, 2023

The blog is a classic case study for data engineers who like to build SQL-like abstractions. link] BlaBlaCar: Controlling our Data Platform Costs at BlaBlaCar Cloud data warehouses bring flexibility and user-friendliness with additional infrastructure costs.

Data Engineering 52

Data Engineering Data Engineer Engineering SQL 52

Data Engineering Weekly

DECEMBER 21, 2022

The Data world Before Hadoop Era We must walk through memory lane to understand why functional data engineering is critical. Let’s reference what the data world looked like before the Hadoop era. Garner predicted in 2005 around 50% of Data Warehouse projects would fail. Why is it so?

Data Engineering 57

Data Engineering Data Engineer Engineering Hadoop 57

Knowledge Hut

NOVEMBER 17, 2023

The demand for data-related professions, including data engineering, has indeed been on the rise due to the increasing importance of data-driven decision-making in various industries. Becoming an Azure Data Engineer in this data-centric landscape is a promising career choice.

Data Engineering 52

Data Engineering Data Engineer Engineering Scala 52

Knowledge Hut

JUNE 30, 2023

The demand for experienced data engineers continuously expands in today's data-driven environment. Books on data engineering serve as essential resources to guide you through the vast terrain of data engineering. What is Data Engineering? Who are Data Engineers?

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Data Engineering Weekly

JULY 2, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork, and SQL grunt work out of building complete customer profiles, so you can quickly ship actionable, enriched data to every downstream team. So, let's shape the future of Data Engineering together.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

Data Engineering Weekly

NOVEMBER 26, 2023

It’s the end of the year, and there will be a lot of buzz about what the next five years in data engineering might bring. We saw a fleet of announcements from Data Catalogs tools on how LLM can help to auto-generate documentation [See: How Generative AI Is Making Data Catalogs Smarter ].

Data Engineering 70

Data Engineering Data Engineer Engineering Kafka 70

Maxime Beauchemin

JANUARY 20, 2017

I joined Facebook in 2011 as a business intelligence engineer. By the time I left in 2013, I was a data engineer. We were data engineers! Data Engineering? Data science as a discipline was going through its adolescence of self-affirming and defining itself. We were pioneers.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130

Data Engineering Weekly

APRIL 23, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make collecting data from every application, website, and SaaS platform easy, then activating it in your warehouse and business tools. 🍒 We'll send exclusive The Data Stack Show swag just for participating!

Data Engineering 52

Data Engineering Data Engineer Engineering Data Pipeline 52

Towards Data Science

JANUARY 28, 2024

How I organize everything in my new data warehouse Continue reading on Towards Data Science »

Data Warehouse 70

Data Warehouse Designing Data Science Data 70

Knowledge Hut

OCTOBER 3, 2023

Within the Microsoft Azure ecosystem, the role of an Azure data engineer stands out as one of the most sought-after positions. What Does an Azure Data Engineer Do? Azure Data engineers collaborate with Azure AI services built on top of Azure Cognitive Services APIs to offer end customers a variety of pre-built models.

Data Engineering 52

Data Engineering Data Engineer Engineering Cloud Computing 52