Analytics Vidhya

SEPTEMBER 20, 2023

Data engineering plays a pivotal role in the vast data ecosystem by collecting, transforming, and delivering data essential for analytics, reporting, and machine learning. Aspiring data engineers often seek real-world projects to gain hands-on experience and showcase their expertise. This article presents the top 20 data engineering project ideas with their source code.

Data Engineering 268

Data Engineering Data Engineer Coding Project 268

Marc Lamberti

SEPTEMBER 22, 2023

Wondering how to share data between tasks? What are XCOMs in Apache Airflow? Well, you are at the right place. In this tutorial, you will learn about XComs in Airflow. What they are, how they work, how you can define them, how to get them, and more. If you checked my course “Apache Airflow: The Hands-On Guide”, Aiflow XCom should not sound unfamiliar.

MySQL 246

MySQL Data Pipeline Database Python 246

The Pragmatic Engineer

SEPTEMBER 22, 2023

👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. We cover one out of four topics in yesterday’s subscriber-only The Pulse issue. To get full newsletters twice a week, subscribe here. Two weeks ago, a JavaScript runtime and toolkit called Bun was released and took the Node.js world by storm. Bun was mostly built by Jared Sumner , a former Stripe engineer, and recipient of the Thiel Fellowship (a grant of $100,000 for young people to drop out of s

Programming Language 239

Programming Language Coding Project Engineering 239

KDnuggets

SEPTEMBER 16, 2023

This tutorial offers a comprehensive hands-on walkthrough of machine learning with Scikit-learn. Readers will learn key concepts and techniques including data preprocessing, model training and evaluation, hyperparameter tuning, and compiling ensemble models for enhanced performance.

Machine Learning 144

Machine Learning Data 144

Advertisement

Generative AI is upending the way product developers & end-users alike are interacting with data. Despite the potential of AI, many are left with questions about the future of product development: How will AI impact my business and contribute to its success? What can product managers and developers expect in the future with the widespread adoption of AI?

Data

Analytics Vidhya

SEPTEMBER 20, 2023

Data engineering plays a pivotal role in the vast data ecosystem by collecting, transforming, and delivering data essential for analytics, reporting, and machine learning. Aspiring data engineers often seek real-world projects to gain hands-on experience and showcase their expertise. This article presents the top 20 data engineering project ideas with their source code.

Data Engineering 130

Data Engineering Data Engineer Coding Project 130

Marc Lamberti

SEPTEMBER 22, 2023

What is Apache Airflow? Perhaps your colleagues or YouTube videos have mentioned it. Maybe your job requires you to use it, but you’re unsure what it is. In this article, you will learn everything about what Airflow is, what it isn’t, and its core concepts and components. But, before answering this question, we need a proper understanding of what an “orchestrator” is.

Data Pipeline 130

Data Pipeline Python Metadata Database 130

KDnuggets

SEPTEMBER 20, 2023

KDnuggets' latest cheat sheet covers 10 curated hands-on projects to boost data science workflows with ChatGPT across ML, NLP, and full stack dev, including links to full project details.

Project 143

Project Data Science Data Machine Learning 143

ArcGIS

SEPTEMBER 19, 2023

In collaboration with NOAA, we used the Presence-Only Prediction (Maxent) tool to predict snow crab habitat under changing climate conditions.

Machine Learning 129

Machine Learning 129

Marc Lamberti

SEPTEMBER 21, 2023

Looking to create your first Airflow DAG? Wondering how to process data in Airflow? What are the steps to code your data pipelines? You’ve come to the right place! At the end of this short tutorial, you will have your first Airflow DAG! You might think starting with Apache Airflow is hard, but it is not. The truth is Airflow has so many features that it can be overwhelming.

Data Pipeline 130

Data Pipeline Python Coding Machine Learning 130

databricks

SEPTEMBER 20, 2023

This blog post is in collaboration with Greg Rokita, AVP of Technology at Edmunds. Long envisioned as a key milestone in computing, we've.

Building 118

Building Technology Retail 118

Advertisement

Many organizations today are unlocking the power of their data by using graph databases to feed downstream analytics, enahance visualizations, and more. Yet, when different graph nodes represent the same entity, graphs get messy. Watch this essential video with Senzing CEO Jeff Jonas on how adding entity resolution to a graph database condenses network graphs to improve analytics and save your analysts time.

Database

KDnuggets

SEPTEMBER 20, 2023

This tutorial provides hands-on experience with the key concepts and implementation of K-Means clustering, a popular unsupervised learning algorithm, for customer segmentation and targeted advertising applications.

Algorithm 134

Algorithm Machine Learning 134

ArcGIS

SEPTEMBER 22, 2023

This Climate Week renews focus on nature. Learn more about how ArcGIS supports nature-related assessments to run sustainable organizations.

117

117

Workfall

SEPTEMBER 18, 2023

Reading Time: 9 minutes Imagine your data as pieces of a complex puzzle scattered across different platforms and formats. Making sense of this scattered information often feels like solving a gigantic puzzle blindfolded. This is where the power of data integration comes into play. If you’ve ever wished for a simplified way to seamlessly connect these puzzle pieces, then you’re in for a treat.

Data Pipeline 93

Data Pipeline Raw Data Data Schemas Healthcare 93

databricks

SEPTEMBER 21, 2023

Introduction In this blog post, we'll explore a set of advanced SQL functions available within Apache Spark that leverage the HyperLogLog algorithm, enabling.

Algorithm 105

Algorithm SQL Data Engineering Data Engineer 105

Speaker: Scott Sehlhorst

We know we want to create products which our customers find to be valuable. Whether we label it as customer-centric or product-led depends on how long we've been doing product management. There are three challenges we face when doing this. The obvious challenge is figuring out what our users need; the non-obvious challenges are in creating a shared understanding of those needs and in sensing if what we're doing is meeting those needs.

Certification

KDnuggets

SEPTEMBER 18, 2023

If you're looking for a hands-on experience with a detailed yet beginner-friendly tutorial on implementing Linear Regression using Scikit-learn, you're in for an engaging journey.

Machine Learning 124

Machine Learning 124

LinkedIn Engineering

SEPTEMBER 20, 2023

Since learning frontend and backend skills, Rishika’s passion for engineering has expanded beyond her team at LinkedIn to grow into her own digital community. As she develops as an engineer, giving back has become the most rewarding part of her role. From intern to engineer—life at LinkedIn My career with LinkedIn began with a college internship, where I got to dive into all things engineering.

Engineering 89

Engineering Programming Language Java Media 89

Pinterest Engineering

SEPTEMBER 20, 2023

Pedro Silva | Sr. ML Engineer & Inclusive AI Tech Lead; Bhawna Juneja | Sr. Machine Learning Engineer; Rohan Mahadev | Machine Learning Engineer II; Sujay Khandagale | Machine Learning Engineer II; Abhay Varmaraja | Machine Learning Engineer II Pinterest’s mission as a company is to bring everyone the inspiration to create a life they love. “Everyone” has been the north star for our Inclusive AI and Inclusive Product teams.

Building 84

Building Pipeline-centric Machine Learning Datasets 84

databricks

SEPTEMBER 18, 2023

Costa Rica is known for several things, both culturally and ecologically. Among those are biodiversity, coffee, Pura Vida, and most recently a rapidly.

104

104

Speaker: Timothy Chan, PhD., Head of Data Science

Are you ready to move beyond the basics and take a deep dive into the cutting-edge techniques that are reshaping the landscape of experimentation? 🌐 From Sequential Testing to Multi-Armed Bandits, Switchback Experiments to Stratified Sampling, Timothy Chan, Data Science Lead, is here to unravel the mysteries of these powerful methodologies that are revolutionizing how we approach testing.

Data Science

KDnuggets

SEPTEMBER 20, 2023

On October 11th, 2023 the Feature Store Summit will bring together leading ML companies, such as Uber, WeChat and more, for in-depth discussions about data and AI.

Data 131

Data 131

Snowflake

SEPTEMBER 19, 2023

Why AI has everyone’s attention, what it means for different data roles, and how Alteryx and Snowflake are bringing AI to data use cases There’s a llama on the loose! Well, more specifically, LLaMA (Large Language Model Meta AI), along with other large language models (LLMs) that have suddenly become more open and accessible for everyday applications.

Machine Learning 97

Machine Learning Unstructured Data Data Analytics Government 97

Towards Data Science

SEPTEMBER 20, 2023

A hands-on guide to implementing BigQuery Subscriptions in Pub/Sub for simple message and streaming ingestion Continue reading on Towards Data Science »

Data Science 85

Data Science Google Cloud Data Cloud 85

databricks

SEPTEMBER 18, 2023

We are thrilled to introduce the support of a new SQL feature in Apache Spark and Databricks: Lateral Column Alias (LCA). This feature.

SQL 111

SQL Engineering Data 111

Advertisement

Start-ups & SMBs launching products quickly must bundle dashboards, reports, & self-service analytics into apps. Customers expect rapid value from your product (time-to-value), data security, and access to advanced capabilities. Traditional Business Intelligence (BI) tools can provide valuable data analysis capabilities, but they have a barrier to entry that can stop small and midsize businesses from capitalizing on them.

Data Analysis

KDnuggets

SEPTEMBER 18, 2023

You can now run Python code in Excel to analyze data, build machine learning models, and create visualizations.

Python 154

Python Data Science Machine Learning Data 154

Robinhood

SEPTEMBER 20, 2023

Customers could get up to 3.5% more crypto on Robinhood* Ahead of Mainnet 2023 in New York City, Robinhood announced the results of a study—verified by Radius Insights —showing that Robinhood offers the lowest cost to trade crypto on average. The analysis compares prices quoted from top platforms and exchanges, including Cash App, Coinbase Advanced, Coinbase, Crypto.com, and Kraken, concluding that customers could receive up to 3.5% more crypto on Robinhood.

Insurance 75

Insurance Education Systems Building 75

Snowflake

SEPTEMBER 20, 2023

ADP provides products, services and experiences that simplify work for more than 1 million clients in 140 countries. Large and small organizations across virtually every industry rely on ADP’s cloud-based human capital management (HCM) solutions to streamline HR, payroll, time, tax and benefits administration. Self-service HCM analytics help ADP’s clients understand workforce trends and benchmark their metrics against aggregated, anonymized data from over 30 million employee records.

Management 85

Management Cloud Architecture Data Architecture 85

databricks

SEPTEMBER 19, 2023

“Supply chains compete, not companies” — Martin Christopher No two supply chains are identical - the unique combination of products, industries, and geographic locat.

Manufacturing 98

Manufacturing 98

Speaker: Aarushi Kansal, AI Leader & Author and Tony Karrer, Founder & CTO at Aggregage

Software leaders who are building applications based on Large Language Models (LLMs) often find it a challenge to achieve reliability. It’s no surprise given the non-deterministic nature of LLMs. To effectively create reliable LLM-based (often with RAG) applications, extensive testing and evaluation processes are crucial. This often ends up involving meticulous adjustments to prompts.

Building

KDnuggets

SEPTEMBER 20, 2023

Python in Excel: This Will Change Data Science Forever • KDnuggets Survey: Benchmark With Your Peers On Data Science Spend & Trends 2023 H2 • 5 Best AI Tools For Maximizing Productivity • And much more!

Data Science 120

Data Science Python Data 120

Cloudera

SEPTEMBER 19, 2023

Perhaps your C-suite is already a bit crowded. The typical hierarchy will include a CEO, COO, CFO, CTO, CMO, CIO, and a few more. Adding another position may not be terribly appealing, but there is one C-suite role every company should consider—chief data and analytics officer (CDO or CDAO). The CDO is the point person for your data strategy: the leader who oversees how data is collected, managed, and put to use to improve the organization; the person who ensures that wherever there are opportun

IT 74

IT Data Governance Data Architecture Government 74

Confluent

SEPTEMBER 18, 2023

Learn how Apache Kafka message compression works, why and how to use it, the five types of compression, configurations for the compression type, and how messages are decompressed.

Kafka 70

Kafka IT 70

databricks

SEPTEMBER 20, 2023

For data-driven enterprises, data analysts play a crucial role in extracting insights from data and presenting it in a meaningful way. However, many.

Data Analytics 98

Data Analytics Data IT 98

Speaker: Anne Steiner and David Laribee

As a concept, Developer Experience (DX) has gained significant attention in the tech industry. It emphasizes engineers’ efficiency and satisfaction during the product development process. As product managers, we need to understand how a good DX can contribute not only to the well-being of our development teams but also to the broader objectives of product success and customer satisfaction.

Engineering