September, 2024

article thumbnail

Paying down tech debt: further learnings

The Pragmatic Engineer

This is a follow-up to the article Paying down tech debt , written by industry veteran Lou Franco. Lou has been in the software business for over 30 years as an engineer, EM, and executive. He’s also worked at four startups and the companies that later acquired them; most recently Atlassian as a Principal Engineer on the Trello iOS app. Later this year, he’s publishing a book on tech debt.

article thumbnail

How to build a data project with step-by-step instructions

Start Data Engineering

1. Introduction 2. Setup 3. Parts of data engineering 3.1. Requirements 3.1.1. Understand input datasets available 3.1.2. Define what the output dataset will look like 3.1.3. Define SLAs so stakeholders know what to expect 3.1.4. Define checks to ensure the output dataset is usable 3.2. Identify what tool to use to process data 3.3. Data flow architecture 3.

Project 240
Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Trending Sources

article thumbnail

Data Teams Survey 2020-2024 Analysis

Jesse Anderson

Survey Changes Over Time Between 2020 and 2024 (see 2020, 2023, and 2024 for each year’s information), I’ve been conducting a data teams survey. I wanted to dedicate an entire post to examining the change in data teams over time. Total Value Creation The most important question I ask each year concerns data team value creation. I break the question into two parts: “How successful would the business say your projects are?

article thumbnail

5 Quirky Data Science Projects to Impress

KDnuggets

Develop unique yet standing-out data science projects to improve your data portfolio.

article thumbnail

Apache Airflow® 101 Essential Tips for Beginners

Apache Airflow® is the open-source standard to manage workflows as code. It is a versatile tool used in companies across the world from agile startups to tech giants to flagship enterprises across all industries. Due to its widespread adoption, Airflow knowledge is paramount to success in the field of data engineering.

article thumbnail

Setup Mage AI with Postgres to Build and Manage Your Data Pipeline

Analytics Vidhya

Introduction Imagine yourself as a data professional tasked with creating an efficient data pipeline to streamline processes and generate real-time information. Sounds challenging, right? That’s where Mage AI comes in to ensure that the lenders operating online gain a competitive edge. Picture this: thus, unlike many other extensions that require deep setup and constant coding, […] The post Setup Mage AI with Postgres to Build and Manage Your Data Pipeline appeared first on Analytics Vidhy

More Trending

article thumbnail

Fine-tuning Llama 3.1 with Long Sequences

databricks

Mosaic AI Model Training now supports fine-tuning up to 131K context length for Llama 3.1 models. More efficient training at long sequence lengths is made possible by several optimizations highlighted in this post.

137
137
article thumbnail

How to decide on a data project for your portfolio

Start Data Engineering

1. Introduction 2. Steps to decide on a data project to build 2.1. Objective 2.2. Research 2.2.1. Job description 2.2.2. Potential referral/hiring manager research 2.2.3. Company research 2.3. Data 2.3.1. Dataset Search 2.3.2. Generate fake data 2.4. Outcome 2.4.1. Visualization 2.5. Presentation 3. Conclusion 4. Read these 1.

Portfolio 130
article thumbnail

How To Modernize Your Data Strategy And Infrastructure For 2025

Seattle Data Guy

We are still in the early days of data and the value it can add to companies. You’ll read plenty of statistics about how much value data can drive and how far behind companies that aren’t using data are. And as a data consultant, I have helped companies find that value in their data. It… Read more The post How To Modernize Your Data Strategy And Infrastructure For 2025 appeared first on Seattle Data Guy.

article thumbnail

7 Steps to Mastering Coding for Data Science

KDnuggets

Are you an aspiring data scientist or early in your data science career? If so, you know that you should use your programming, statistics, and machine learning skills—coupled with domain expertise—to use data to answer business questions. To succeed as a data scientist, therefore, becoming proficient in coding is essential. Especially for handling and analyzing.

article thumbnail

Apache Airflow® Best Practices: DAG Writing

Speaker: Tamara Fingerlin, Developer Advocate

In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!

article thumbnail

Simulator-based reinforcement learning for data center cooling optimization

Engineering at Meta

We’re sharing more about the role that reinforcement learning plays in helping us optimize our data centers’ environmental controls. Our reinforcement learning-based approach has helped us reduce energy consumption and water usage across various weather conditions. Meta is revamping its new data center design to optimize for artificial intelligence and the same methodology will be applicable for future data center optimizations as well.

Data 130
article thumbnail

How Producers Work: Kafka Producer and Consumer Internals, Part 1

Confluent

Dive into Kafka internals with a four-part series examining client requests and brokers. Part 1 covers what a producer does to prepare raw event data for the broker.

Kafka 139
article thumbnail

Introducing Meta Llama 3.2 on Databricks: faster language models and powerful multi-modal models

databricks

We are excited to partner with Meta to launch the latest models in the Llama 3 series on the Databricks Data Intelligence Platform.

Data 135
article thumbnail

What are the Key Parts of Data Engineering?

Start Data Engineering

1. Introduction 2. Key parts of data systems: 2.1. Requirements 2.2. Data flow design 2.3. Orchestrator and scheduler 2.4. Data processing design 2.5. Code organization 2.6. Data storage design 2.7. Monitoring & Alerting 2.9. Infrastructure 3. Conclusion 1. Introduction If you are trying to break into (or land a new) data engineering job, you will inevitably encounter a slew of data engineering tools.

article thumbnail

Optimizing The Modern Developer Experience with Coder

Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.

article thumbnail

Real-time Analytics Vs Stream Processing – What Is The Difference?

Seattle Data Guy

One of the holy grails that many data teams seem to chase is real-time data analytics. After all, if you can have real-time analytics, you can make better decisions faster. However, there often is a conflation between real-time data analytics and stream processing. These are two different concepts that are crucial to understanding how to… Read more The post Real-time Analytics Vs Stream Processing – What Is The Difference?

Process 130
article thumbnail

10 Built-In Python Modules Every Data Engineer Should Know

KDnuggets

Interested in data engineering? Check out this round-up of built-in Python modules that'll come in handy for data engineering tasks.

Python 151
article thumbnail

How does Data Interoperability relate to FME?

ArcGIS

Learn the difference between ArcGIS Data Interoperability and FME technology and how they relate to one another.

Data 127
article thumbnail

Inside Bento: Jupyter Notebooks at Meta

Engineering at Meta

This episode of the Meta Tech Podcast is all about Bento , Meta’s internal distribution of Jupyter Notebooks, an open-source web-based computing platform. Bento allows our engineers to mix code, text, and multimedia in a single document and serves a wide range of use cases at Meta from prototyping to complex machine learning workflows. Pascal Hartig ( @passy ) is joined by Steve, whose team has built several features on top of Jupyter, including scheduled notebooks , sharing with colleagues, and

article thumbnail

15 Modern Use Cases for Enterprise Business Intelligence

Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?

article thumbnail

Unleash Your Innovation: Announcing the Databricks Generative AI Startup Challenge with Over $1 Million in Credits, Prizes, and Potential Venture Funding

databricks

The Databricks Generative AI Startup Challenge offers $1M+ in prizes for innovative startups building Generative AI use cases on Databricks. Apply by November 1, 2024!

Building 134
article thumbnail

9 Mainframe Statistics That May Surprise You

Precisely

Are mainframes still relevant today? You bet! The following ten statistics paint a picture that shows mainframes are still going strong, with no signs of slowing. 1. The Mainframe Turns 60: A Milestone in Computing History. 60 years can really fly by! On April 7, 2024 , the Mainframe turned 60. At this milestone, we should all reflect on what the mainframe has done to the computing industry.

Banking 116
article thumbnail

AI (LLMs) and Software Engineering (Writing Code)

Confessions of a Data Guy

I recently wrote on my Substack (Data Engineering Central) about how I used the new OpenAI o1 model to do some basic Data Engineering tasks surrounding PostgreSQL. It did ok. I’ve also been using CoPilot and ChatGPT for over a year now to assist me with my daily code that I have to write for […] The post AI (LLMs) and Software Engineering (Writing Code) appeared first on Confessions of a Data Guy.

article thumbnail

10 GitHub Repositories to Master Computer Vision

KDnuggets

The GitHub repository includes up-to-date learning resources, research papers, guides, popular tools, tutorials, projects, and datasets.

Datasets 149
article thumbnail

Prepare Now: 2025s Must-Know Trends For Product And Data Leaders

Speaker: Jay Allardyce, Deepak Vittal, Terrence Sheflin, and Mahyar Ghasemali

As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.

article thumbnail

Transform 2D building footprint polygons into 3D buildings using 3D Object Feature Layer

ArcGIS

Interested in 3D GIS but not sure where to start? Learn the proper method to transform pre-existing 2D footprint polygons into a 3D buildings.

Building 113
article thumbnail

Read Meta’s 2024 Sustainability Report

Engineering at Meta

We are working in partnership with others to scale inclusive solutions that support the transition to a zero-carbon economy and help create a healthier planet for all.

article thumbnail

Databricks announces significant improvements to the built-in LLM judges in Agent Evaluation

databricks

An improved answer-correctness judge in Agent Evaluation Agent Evaluation enables Databricks customers to define, measure, and understand how to improve the quality of.

article thumbnail

Handling the Producer Request: Kafka Producer and Consumer Internals, Part 2

Confluent

Learn how your data goes from a producing client all the way to disk on a broker—along the way traversing buffers, threads, queues and more.

Kafka 111
article thumbnail

How to Drive Cost Savings, Efficiency Gains, and Sustainability Wins with MES

Speaker: Nikhil Joshi, Founder & President of Snic Solutions

Is your manufacturing operation reaching its efficiency potential? A Manufacturing Execution System (MES) could be the game-changer, helping you reduce waste, cut costs, and lower your carbon footprint. Join Nikhil Joshi, Founder & President of Snic Solutions, in this value-packed webinar as he breaks down how MES can drive operational excellence and sustainability.

article thumbnail

Data Modeling in the Brave New Lakehouse World

Confessions of a Data Guy

It is a Brave New World out there these days. The new tools and features come out faster than your mom on Sunday morning getting you ready for church. The same goes for the context and advice being produced on a myriad of platforms, the ole’ Like and Subscribe, and all that bit. It does […] The post Data Modeling in the Brave New Lakehouse World appeared first on Confessions of a Data Guy.

Data 114
article thumbnail

Free Courses That Are Actually Free: Data Analytics Edition

KDnuggets

Kickstart your data analyst career with all these free courses.

article thumbnail

How to publish customized views of the same source data

ArcGIS

To publish different views of the same source data, alter map layer settings before you publish each web feature layer.

Data 109