Analytics Vidhya

JUNE 20, 2023

Introduction In today’s data-driven world, organizations across industries are dealing with massive volumes of data, complex pipelines, and the need for efficient data processing.

Data Engineering 217

Data Engineering Data Engineer Data Process Engineering 217

Data Engineering Podcast

SEPTEMBER 17, 2023

Summary A significant amount of time in data engineering is dedicated to building connections and semantic meaning around pieces of information. Linked data technologies provide a means of tightly coupling metadata with raw information. Go to dataengineeringpodcast.com/materialize today to get 2 weeks free!

Building 189

Building Data SQL Python 189

Seattle Data Guy

AUGUST 2, 2023

A few months ago, I uploaded a video where I discussed data warehouses, data lakes, and transactional databases. However, the world of data management is evolving rapidly, especially with the resurgence of AI and machine learning.

Data Lake 130

Data Lake Data Warehouse Data Machine Learning 130

Analytics Vidhya

MARCH 8, 2023

The Biggest Data Science Blogathon is now live! Martin Uzochukwu Ugwu Analytics Vidhya is back with the largest data-sharing knowledge competition- The Data Science Blogathon. Knowledge is power. Sharing knowledge is the key to unlocking that power.”―

Data Science 245

Data Science Data Hadoop Deep Learning 245

Analytics Vidhya

JUNE 25, 2023

In a data-driven world, behind-the-scenes heroes like data engineers play a crucial role in ensuring smooth data flow. A data engineer investigates the issue, identifies a glitch in the e-commerce platform’s data funnel, and swiftly implements seamless data pipelines.

Data Engineering 219

Data Engineering Data Engineer Engineering Data 219

Medium Data Engineering

SEPTEMBER 2, 2023

In its simplest form Data Contract is an agreement between Data Producers and Data Consumers on what the Data being produced should look… Continue reading on Medium »

Data Pipeline 97

Data Pipeline Data Data Architecture Data Engineering 97

Snowflake

SEPTEMBER 18, 2023

As data continues to become more complex, it is critical to have effective ways to present this information. With the explosion of AI/ML, users want to be able to interact with their data and ML models. However, building such data apps has not been easy. Together, they built over 5,000 apps.

Python 117

Python Data Cloud Building 117

Medium Data Engineering

SEPTEMBER 27, 2023

Build a robust GraphQL API service on top of your S3 data lake files with DuckDB and Go. Continue reading on Towards Data Science »

Data Lake 98

Data Lake Data Data Science Accessible 98

Towards Data Science

SEPTEMBER 27, 2023

Data Access API over Data Lake Tables Without the Complexity Build a robust GraphQL API service on top of your S3 data lake files with DuckDB and Go Photo by Joshua Sortino on Unsplash 1. Suppose that we have a data lake table that stores real-time statistics about our customers generated by some Spark application.

Data Lake 81

Data Lake Data Accessible Accessibility 81

Lyft Engineering

OCTOBER 3, 2023

High-quality data is necessary for the success of every data-driven company. It is now the norm for tech companies to have a well-developed data platform. This makes it easy for engineers to generate, transform, store, and analyze data at the petabyte scale. What and Where is Data Quality?

Big Data 57

Big Data Data Data Warehouse Metadata 57

Medium Data Engineering

OCTOBER 2, 2023

If you work as a data engineer, data analytics, or data scientist in an organization that needs you on a project and are using a pretty… Continue reading on Medium »

Data Engineering 52

Data Engineering Data Engineer Data Pipeline Engineering 52

Towards Data Science

AUGUST 29, 2023

Data Quality Chronicles Missing data, missing mechanisms, and missing data profiling Missing Data prevents data scientists to see the entire story the data has to tell. One of them was, unsurprisingly, Missing Data. Photo by Ronan Furuta on Unsplash. Image by Author. Let’s consider an example.

Data 101

Data Datasets Machine Learning Data Science 101

Towards Data Science

MAY 19, 2023

Data Entropy — More Data, More Problems? How to navigate and embrace complexity in a modern data organisation. Business users are unable to find and access data assets critical to their workflows. Data engineers spend countless hours troubleshooting broken pipelines. Every “minor” change upstream results in mayhem.

Data 104

Data Pipeline-centric Data Engineering Data Engineer 104

Medium Data Engineering

SEPTEMBER 21, 2023

Hey there, data friends! As many of you might know, I’ve been a bit quiet lately due to a big transition: moving from one home to another… Continue reading on Medium »

Data 98

Data Data Pipeline Cloud Computing Data Engineering 98

Snowflake

SEPTEMBER 25, 2023

Digital advancements such as smart manufacturing and automation through AI, machine learning (ML), robotics, and IoT require a connected value chain ecosystem with a secure, scalable, and flexible data platform. Easy data sharing and collaboration A key feature of Snowflake is the ability to share and collaborate on data.

Manufacturing 87

Manufacturing Cloud Data Data Storage 87

KDnuggets

SEPTEMBER 10, 2023

Back to Basics: Understanding key data management principles that data scientists should know.

Data Management 118

Data Management Data Science Data Management 118

KDnuggets

SEPTEMBER 26, 2023

This article describes the data maturity pyramid and its various levels, from simple reporting to AI-ready data platforms. It emphasizes the importance of data for business and illustrates how data platforms serve as the driving force behind AI.

Data 85

Data Data Science IT 85

Medium Data Engineering

MAY 20, 2023

In today’s digital age, organizations are grappling with an ever-increasing volume of data generated from various sources. As the need for… Continue reading on Medium »

Data Lake 98

Data Lake Data Warehouse Data Big Data 98

Medium Data Engineering

AUGUST 29, 2023

Missing data, missing mechanisms, and missing data profiling Continue reading on Towards Data Science »

Data 98

Data Data Science Data Analysis Data Engineering 98

Medium Data Engineering

SEPTEMBER 26, 2023

As data becomes the new oil, its smooth flow, availability, and… Continue reading on Medium » The technology landscape has shifted dramatically in the past few decades.

Data Engineering 52

Data Engineering Data Engineer Data Engineering 52

Medium Data Engineering

AUGUST 20, 2023

In today’s data-driven landscape, the role of a Data Engineer is more crucial than ever before. As the architects behind the scenes, Data… Continue reading on Medium »

Data Engineering 98

Data Engineering Data Engineer Engineering Data 98

Medium Data Engineering

OCTOBER 2, 2023

When working with time-series data in business analytics, it’s common to encounter scenarios where the data needs to be transformed for… Continue reading on Medium »

SQL 52

SQL Data Data Analysis Data Engineering 52

Medium Data Engineering

SEPTEMBER 9, 2023

Data engineering is the backbone of modern data-driven businesses. Data engineers design, build, and maintain the infrastructure that… Continue reading on The Resume Whisperer »

Data Engineering 98

Data Engineering Data Engineer Engineering Data 98

DataKitchen

SEPTEMBER 21, 2023

Bridging the Gap: How ‘Data in Place’ and ‘Data in Use’ Define Complete Data Observability In a world where 97% of data engineers report burnout and crisis mode seems to be the default setting for data teams, a Zen-like calm feels like an unattainable dream. What is Data in Place?

Data 64

Data Data Engineering Data Engineer Raw Data 64

Medium Data Engineering

SEPTEMBER 27, 2023

Big data engineers play a vital role in real-time data processing, as they are responsible for designing and building the systems that can… Continue reading on Medium »

Data Engineering 52

Data Engineering Data Engineer Big Data Data Process 52

Databand.ai

JUNE 28, 2023

Data Pipeline Observability: A Model For Data Engineers Eitan Chazbani June 29, 2023 Data pipeline observability is your ability to monitor and understand the state of a data pipeline at any time. We believe the world’s data pipelines need better data observability. What is data pipeline observability?

Data Pipeline 98

Data Pipeline Data Engineering Data Engineer Engineering 98

Medium Data Engineering

OCTOBER 1, 2023

In this article, I will go through use cases and tips for the top python packages used daily by Data Engineers and Data Scientists. Continue reading on Medium »

Data Engineering 52

Data Engineering Data Engineer Python Engineering 52

KDnuggets

SEPTEMBER 29, 2023

Cloud notebooks are game-changers for data science, providing free access to computing, pre-built environments, collaboration features, and third-party integrations - everything you need to enhance your workflow.

Data Science 106

Data Science Cloud Data Accessible 106

The Modern Data Company

JULY 24, 2023

In the modern data-driven landscape, organizations are constantly seeking ways to extract valuable insights from their data assets. While individual data products provide significant value, the true potential lies in harnessing the power of interconnected data products.

Data 85

Data Data Lake Machine Learning Data Integration 85

Medium Data Engineering

SEPTEMBER 4, 2023

Meta-Description: “Unlock the secrets of data storage with our comprehensive guide. Explore the battle of Data Lake vs. Data Warehouse… Continue reading on Medium »

Data Lake 52

Data Lake Data Warehouse Data Storage Data 52

Data Engineering Weekly

SEPTEMBER 24, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. Airflow isn’t meant to process the data. See how it works today.

Data Engineering 69

Data Engineering Data Engineer Engineering Kafka 69

Data Engineering Weekly

OCTOBER 1, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. What is the data behavior? Job Scheduling : how frequently should our data pipeline run?

Data Engineering 52

Data Engineering Data Engineer Engineering Data 52

Medium Data Engineering

SEPTEMBER 5, 2023

Establishing data pipelines towards automated spatial data science Continue reading on Towards Data Science »

Data Engineering 98

Data Engineering Data Engineer Data Engineering 98

Medium Data Engineering

SEPTEMBER 11, 2023

During an SQL coding screen in a data engineer interview, you can typically expect to be presented with real-world scenarios or problems… Continue reading on Medium »

Data Engineering 98

Data Engineering Data Engineer SQL Engineering 98

InData Labs

AUGUST 24, 2023

The world is generating an unprecedented amount of data every second. From online transactions and social media interactions to sensor readings and scientific research, the sheer volume, velocity, and variety of data have given rise to the concept of “Big data.”

Big Data 98

Big Data Data Media Data Engineering 98

Medium Data Engineering

AUGUST 31, 2023

Optimizing queries, improving runtimes and the geospatial data science applications Continue reading on Towards Data Science »

Data Engineering 98

Data Engineering Data Engineer Data Engineering 98

Medium Data Engineering

SEPTEMBER 29, 2023

When you are trying to work with dataset that is beyond small or medium scale then it is advisable to use data warehouse for data storage. Continue reading on Medium »

Data Warehouse 52

Data Warehouse Data Data Storage Datasets 52