Start Data Engineering

SEPTEMBER 18, 2024

Introduction 2. Parts of data engineering 3.1. Requirements 3.1.1. Understand input datasets available 3.1.2. Define what the output dataset will look like 3.1.3. Define SLAs so stakeholders know what to expect 3.1.4. Define checks to ensure the output dataset is usable 3.2. Identify what tool to use to process data 3.3. Data flow architecture 3.

Project 240

Project Building Datasets Architecture 240

Confessions of a Data Guy

MARCH 29, 2024

Have you ever wondered at a high level what it’s like to build production-level data pipelines on Databricks? The post Building Databricks Data Pipelines 101 appeared first on Confessions of a Data Guy. What does it look like, what tools do you use?

Data Pipeline 100

Data Pipeline Building Data IT 100

Confessions of a Data Guy

JUNE 11, 2024

In a show of solidarity to the open-source community, I thought I would publish a PYPI package that will probably be used by 5 people […] The post Building Open-Source Python Packages – SparklePop appeared first on Confessions of a Data Guy.

Python 100

Python Building Software Engineer Software Engineering 100

Analytics Vidhya

FEBRUARY 3, 2023

Imagine scheduling your ML tasks to run automatically without the need for manual […] The post How to Build and Monitor Systems Using Airflow? Airflow can help you manage your workflow and make your life easier with its monitoring and notifications features. appeared first on Analytics Vidhya.

Systems 214

Systems Building Machine Learning Management 214

KDnuggets

JULY 29, 2024

Learn to build the end-to-end data science pipelines from data ingestion to data visualization using Pandas pipe method.

Data Science 117

Data Science Building Data Ingestion Data 117

KDnuggets

SEPTEMBER 5, 2024

A beginner tutorial on building a simple web application for machine learning model inference using FastAPI and Jinja2 templates.

Building 105

Building Machine Learning Python 105

Data Engineering Podcast

FEBRUARY 26, 2023

Jean-Georges Perrin was tasked with designing a new data platform implementation at PayPal and wound up building a data mesh. It's supposed to make building smarter, faster, and more flexible data infrastructures a breeze. We feel your pain. It ends up being anything but that. You can't optimize for everything all at once.

Building 147

Building Metadata Machine Learning Data Integration 147

Snowflake

OCTOBER 7, 2024

When scaling data science and ML workloads, organizations frequently encounter challenges in building large, robust production ML pipelines. Many teams turn to feature stores to create a centralized repository that maintains a consistent and up-to-date set of ML features.

Management 81

Management Building Datasets Government 81

Data Engineering Podcast

OCTOBER 15, 2023

Summary Building streaming applications has gotten substantially easier over the past several years. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. How can you get the best results for your use case?

Process 182

Process Building SQL BI 182

Confluent

MAY 10, 2024

Discover how to build trust in AI by strengthening data and people layers. Learn about risk frameworks, data streaming, and more for effective solutions.

Building 72

Building Data 72

KDnuggets

AUGUST 28, 2024

Learn how to build and deploy an end-to-end data pipeline using Prefect with a few lines of code.

Data Pipeline 76

Data Pipeline Building Data Coding 76

ArcGIS

SEPTEMBER 18, 2024

Learn how to build cross-street data into a geocoding locator using ArcGIS Data Interoperability and its no-code approach.

Building 75

Building Data Coding IT 75

Confessions of a Data Guy

MARCH 25, 2024

Ever wondered how to build and end-to-end project for an Open Source Python Package that gets published to PYPI? link] The post How To Build and Open Source PYPI Python Package appeared first on Confessions of a Data Guy.

Python 100

Python Building Project Data 100

Data Engineering Podcast

MARCH 5, 2023

It's supposed to make building smarter, faster, and more flexible data infrastructures a breeze. By bringing all the layers of the data stack together, TimeXtender helps you build data solutions up to 10 times faster and saves you 70-80% on costs. We feel your pain. It ends up being anything but that. We feel your pain.

Building 147

Building Database Design Machine Learning Metadata 147

Tweag

JULY 5, 2023

Buck2 is a from-scratch rewrite of Buck , a polyglot, monorepo build system that was developed and used at Meta (Facebook), and shares a few similarities with Bazel. As you may know, the Scalable Builds Group at Tweag has a strong interest in such scalable build systems. invoke build buck2 build //starlark-rust/starlark 6.

Systems 141

Systems Building Java Programming Language 141

KDnuggets

JUNE 6, 2024

In this article, you will be impacted by the knowledge you need to start building LLM apps with Python programming language.

Python 116

Python Building Programming Language Programming 116

Snowflake

NOVEMBER 15, 2023

BUILD 2023 is where AI gets real. Join our two-day virtual global conference and learn how to build with the app dev innovations you heard about at Snowflake Summit and Snowday. As always, BUILD is created by builders, for builders, so you’ll find plenty to spark your interest and boost your skill set.

Building 115

Building Accessible Accessibility IT 115

KDnuggets

APRIL 3, 2023

LangChain is a Python library that helps you build GPT-powered applications in minutes. Get started with LangChain by building a simple question-answering app.

Building 157

Building Python Process 157

Data Engineering Podcast

AUGUST 27, 2023

RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. Learn more about Datafold by visiting dataengineeringpodcast.com/datafold You shouldn't have to throw away the database to build with fast-changing data.

Database 130

Database Building PostgreSQL BI 130

KDnuggets

JUNE 13, 2024

Want to build APIs with Python? Learn how to do so using FastAPI with this step-by-step tutorial.

Python 127

Python Building 127

KDnuggets

OCTOBER 4, 2024

Have you ever wondered how you can easily create command-line applications in Python? Gather yourself up because that is what I am going to cover today.

Python 110

Python Building 110

Pinterest Engineering

JULY 15, 2024

David Chang; Staff Software Engineer | Develocity, formerly known as Gradle Enterprise, is a powerful tool that speeds up local and CI build time, helps troubleshoot your builds, and analyzes your data. At Pinterest, we have a dedicated team, Mobile Builds, and we ensure that developers can build fast and often.

Building 64

Building Metadata Software Engineer Software Engineering 64

KDnuggets

OCTOBER 12, 2023

This article talks about several best practices for writing ETLs for building training datasets. It delves into several software engineering techniques and patterns applied to ML.

Building 113

Building Software Engineer Software Engineering Datasets 113

KDnuggets

AUGUST 20, 2024

Learn how to build the recommendation system with advanced technology.

Systems 98

Systems Building Technology Machine Learning 98

Christophe Blefari

MAY 2, 2024

My personal collection of the best resources to bootstrap a data team and get inspired from what others are doing.

Building 130

Building Data 130

databricks

JULY 23, 2024

Communications service providers (CSP) build various solutions. The communications industry is experiencing immense change due to rapid technological advancements and evolving market trends.

Building 87

Building Technology 87

databricks

OCTOBER 1, 2024

Many of our customers are shifting from monolithic prompts with general-purpose models to specialized compound AI systems to achieve the quality needed for.

Systems 114

Systems Building Data Science Engineering 114

Confluent

JUNE 27, 2024

You’re not alone—GenAI apps can be challenging to build. Struggling to get your GenAI app off the ground? Luckily, taking an event-driven approach can make the process more manageable. Learn how.

Building 80

Building Process Management 80

Lyft Engineering

JUNE 28, 2023

On the flip side, there was a substantial appetite to build real-time ML systems from developers at Lyft. Shortly after we built it, it was utilized by another pod within our team to build a Real-time Anomaly Detection product. To meet the needs of our customers, we kicked off the Real-time Machine Learning with Streaming initiative.

Machine Learning 116

Machine Learning Building Metadata Kafka 116

Knowledge Hut

APRIL 16, 2024

Going for CSM certification training and knowing how to build a self-organizing team as a Scrum master will help you get trained well. Most Agile-Scrum organizations emphasize on building the self-organizing team - why? How Do You Build a Self-Organizing Team as a Scrum Master?

Building 98

Building Certification Project Management 98