Start Data Engineering

MAY 11, 2022

1. Introduction 2. Objective 3. Design 4. Setup 4.1 Prerequisite 4.2 AWS Infrastructure costs 4.3 Data lake structure 5. Code walkthrough 5.1 Loading user purchase data into the data warehouse 5.2 Loading classified movie review data into the data warehouse 5.3 Generating user behavior metric 5.4. Checking results 6. Tear down infra 7. Design considerations 8.

Data Engineering 246

Data Engineering Data Engineer Data Lake Project 246

Simon Späti

APRIL 6, 2022

With burnout and mental stress at every level of our lives, I find my Personal Knowledge Management (PKM) system even more valuable. As a human, I forget lots of things. As a dad, I have more responsibilities with remembering all things related to my kid. As a developer and knowledge worker, I re-use code snippets or create new things. That’s why a PKM system such as a Second Brain to store all of it in a sustainable way is crucial to me.

Management 189

Management Coding Systems IT 189

KDnuggets

DECEMBER 30, 2022

Data science is ever-evolving, so mastering its foundational technical and soft skills will help you be successful in a career as a Data Scientist, as well as pursue advance concepts, such as deep learning and artificial intelligence.

Data Science 160

Data Science Deep Learning Data IT 160

Confessions of a Data Guy

DECEMBER 10, 2022

There once was a day when no one used DataFrames that much. Back before Spark had really gone mainstream, Data Scientists were still plinking around with Pandas a lot. My My, what would your mother say? How things have changed. Now everyone wants a piece of the DataFrame pie. I mean it tastes so good, […] The post Dataframe Showdown – Polars vs Spark vs Pandas vs DataFusion.

Data 147

Data IT Big Data Data Engineering 147

Advertisement

In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate

Data Pipeline

Netflix Tech

DECEMBER 3, 2022

by Jasmine Omeke , Obi-Ike Nwoke , Olek Gorajek Intro This post is for all data practitioners, who are interested in learning about bootstrapping, standardization and automation of batch data pipelines at Netflix. You may remember Dataflow from the post we wrote last year titled Data pipeline asset management with Dataflow. That article was a deep dive into one of the more technical aspects of Dataflow and didn’t properly introduce this tool in the first place.

Data Pipeline 145

Data Pipeline Scala Metadata Food 145

Confluent

DECEMBER 2, 2022

ksqlDB use case: see how apps can use ksqlDB to ingest, filter, enrich, aggregate, and query data directly with Kafka—no complex architectures or data stores needed.

Kafka 144

Kafka Building Architecture Data 144

Cloudera

APRIL 22, 2022

This blog post was written by Dean Bubley , industry analyst, as a guest author for Cloudera. . Communications service providers (CSPs) are rethinking their approach to enterprise services in the era of advanced wireless connectivity and 5G networks, as well as with the continuing maturity of fibre and Software-Defined Wide Area Network (SD-WAN) portfolios. .

Data Solutions 134

Data Solutions Amazon Web Services Data Storage Google Cloud 134

Start Data Engineering

DECEMBER 11, 2022

1. Introduction 2. Source & Sink 2.1. Source Replayability 2.2. Source Ordering 2.3. Sink Overwritability 3. Data pipeline patterns 3.1. Extraction patterns 3.1.1. Time ranged 3.1.2. Full Snapshot 3.1.3. Lookback 3.1.4. Streaming 3.2. Behavioral 3.2.1. Idempotent 3.2.2. Self-healing 3.3. Structural 3.3.1. Multi-hop pipelines 3.3.2. Conditional/ Dynamic pipelines 3.3.3.

Data Pipeline 195

Data Pipeline Designing Data 195

KDnuggets

DECEMBER 30, 2022

It's time again to look at some data science cheatsheets. Here you can find a short selection of such resources which can cater to different existing levels of knowledge and breadth of topics of interest.

Data Science 160

Data Science Data IT 160

KDnuggets

DECEMBER 29, 2022

This article compiles the 38 top Python libraries for data science, data visualization & machine learning, as best determined by KDnuggets staff.

Machine Learning 160

Machine Learning Data Science Python Data 160

Speaker: Tamara Fingerlin, Developer Advocate

Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.

Data Workflow

KDnuggets

DECEMBER 19, 2022

Revise the concepts of machine learning algorithms, frameworks, and methodologies to ace the technical interview round.

Machine Learning 160

Machine Learning Algorithm 160

KDnuggets

DECEMBER 19, 2022

RPA and IA have stunned the business world by availing impressive, intelligent automation capabilities for scales of businesses across industries, which we'll know in this blog.

160

160

KDnuggets

DECEMBER 16, 2022

Many aspiring Data Scientists, especially when self-learning, fail to learn the necessary math foundations. These recommendations for learning approaches along with references to valuable resources can help you overcome a personal sense of not being "the math type" or belief that you "always failed in math.".

Data Science 160

Data Science Data 160

KDnuggets

DECEMBER 9, 2022

As more people are entering the field of Data Science and more companies are hiring for data-centric roles, what type of jobs are currently in highest demand? There is so much data in the world, and it just keeps flooding in, it now looks like companies are targeting those who can engineer that data more than those who can only model the data.

Engineering 160

Engineering Data Engineering Data Engineer Data 160

Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage

There’s a lot of noise surrounding the ability of AI agents to connect to your tools, systems and data. But building an AI application into a reliable, secure workflow agent isn’t as simple as plugging in an API. As an engineering leader, it can be challenging to make sense of this evolving landscape, but agent tooling provides such high value that it’s critical we figure out how to move forward.

Systems

KDnuggets

DECEMBER 1, 2022

In this blog, I shared my story on getting 4 data science job offers including Airbnb, Lyft and Twitter after being laid off. Any data scientist who was laid off due to the pandemic or who is actively looking for a data science position can find something here to which they can relate.

Data Science 160

Data Science Data 160

KDnuggets

NOVEMBER 23, 2022

There exist so many great computational tools available for Data Scientists to perform their work. However, mathematical skills are still essential in data science and machine learning because these tools will only be black-boxes for which you will not be able to ask core analytical questions without a theoretical foundation.

Data Science 160

Data Science Machine Learning Data 160

KDnuggets

NOVEMBER 18, 2022

The Pandas library is core to any Data Science work in Python. This introduction will walk you through the basics of data manipulating, and features many of Pandas important features.

Data Science 160

Data Science Python Data 160

KDnuggets

NOVEMBER 16, 2022

While different ways to learn Data Science for the first time exist, the approach that works for you should be based on how you learn best. One powerful method is to evolve your learning from simple practice into complex foundations, as outlined in this learning path recommended by a physicist who turned into a Data Scientist.

Data Science 160

Data Science IT Data 160

Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives

Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri

Manufacturing

KDnuggets

NOVEMBER 16, 2022

Knowing git is no longer an option for data professionals. Grab this handy reference sheet now and make sure you know how to git the job done.

Data Science 160

Data Science Data Programming 160

KDnuggets

NOVEMBER 16, 2022

Based on the recent discussions with dozens of Fortune 500 data science teams, we can expect to see a continued spotlight on AI model quality in 2023.

Data Science 160

Data Science Data 160

KDnuggets

NOVEMBER 10, 2022

This article is the write-up of a Machine Learning Lighting Talk, intuitively explaining an important data science concept in 3 minutes.

Data Science 160

Data Science Machine Learning Data 160

KDnuggets

NOVEMBER 7, 2022

This article will present the main approaches to text summarization currently employed, as well as discuss some of their characteristics.

Process 160

Process 160

Advertisement

Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.

Cloud

KDnuggets

NOVEMBER 2, 2022

Want to master data visualization? This list of 30 resources and tools will help you get started on your path toward mastering data visualization.

Data 160

Data Data Science 160

KDnuggets

NOVEMBER 2, 2022

A data engineer is a fast-growing profession with amazing challenges and rewards. Which skills do you need to become a data engineer? In this post, we’ll take a look at both hard and soft skills.

Data Engineer 160

Data Engineer Data Engineering Engineering Data 160

KDnuggets

OCTOBER 24, 2022

Preprocessing data for machine learning models is a core general skill for any Data Scientist or Machine Learning Engineer. Follow this guide using Pandas and Scikit-learn to improve your techniques and make sure your data leads to the best possible outcome.

Python 160

Python Machine Learning Data Engineering 160

KDnuggets

OCTOBER 19, 2022

This post is a summary of 2 distinct frameworks for approaching machine learning tasks, followed by a distilled third. Do they differ considerably (or at all) from each other, or from other such processes available?

Machine Learning 160

Machine Learning Process 160

Advertisement

Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?

Business Intelligence

KDnuggets

OCTOBER 14, 2022

Leveraging sparse matrix representations for your data when appropriate can spare you memory storage. Have a look at the reasons why, see how to create sparse matrices in with Python, and compare the memory requirements for standard and sparse representations of the same data.

Python 160

Python Data 160

KDnuggets

OCTOBER 14, 2022

Check out this free ebook covering the fundamentals of mathematics for machine learning, as well as its companion website of exercises and Jupyter notebooks.

Machine Learning 160

Machine Learning IT 160

KDnuggets

OCTOBER 12, 2022

Do you want to learn PyTorch for machine learning and deep learning? Check out this 24 hour long video course with accompanying notes and courseware for free. Did I mention it's free?

Deep Learning 160

Deep Learning Machine Learning IT 160

KDnuggets

OCTOBER 11, 2022

A Data Science Enablement Team consists of people from various departments like marketing, sales, product development, etc. They are responsible for providing the necessary tools and resources to help the data scientists do their job more efficiently.

Data Science 160

Data Science Building Data 160

Speaker: Jay Allardyce, Deepak Vittal, Terrence Sheflin, and Mahyar Ghasemali

As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.

Data