The Pragmatic Engineer

AUGUST 8, 2024

The below article was originally published in The Pragmatic Engineer , on 29 February 2024. I am re-publishing it 6 months later as a free-to-read article. This is because the below case is a good example on hype versus reality with GenAI. To get timely analysis like this in your inbox, subscribe to The Pragmatic Engineer. Klarna launched its AI chatbot, built in collaboration with OpenAI, which the company wants to use to eliminate 2/3rds of customer support positions.

IT 253

IT Software Engineer Software Engineering Systems 253

Analytics Vidhya

AUGUST 4, 2024

Introduction Managing complicated, interrelated information is more important than ever in today’s data-driven society. Traditional databases, while still valuable, often falter when it comes to handling highly connected data. Enter the unsung heroes of the data world: graph databases. These powerful tools are designed to manage and query intricate data relationships effortlessly.

Database 219

Database Data Engineering Data Engineer Engineering 219

Start Data Engineering

AUGUST 13, 2024

1. Introduction 2. Data structures and algorithms to know 2.1. List 2.2. Dictionary 2.3. Queue 2.4. Stack 2.5. Set 2.6. Counter (from collections module) 2.7. Heap 2.8. Graph search 2.8.1 Depth First Search (DFS) 2.8.2. Breadth First Search BFS 2.9. Binary Search 3. Common DSA questions asked during DE interviews 3.1. Intervals 3.

Algorithm 200

Algorithm Data Engineering Data Engineer Engineering 200

KDnuggets

AUGUST 9, 2024

Optimize LLM performance and scalability using techniques like prompt engineering, retrieval augmentation, fine-tuning, model pruning, quantization, distillation, load balancing, sharding, and caching.

Engineering 157

Engineering 157

Speaker: Alex Salazar, CEO & Co-Founder @ Arcade | Nate Barbettini, Founding Engineer @ Arcade | Tony Karrer, Founder & CTO @ Aggregage

If AI agents are going to deliver ROI, they need to move beyond chat and actually do things. But, turning a model into a reliable, secure workflow agent isn’t as simple as plugging in an API. In this new webinar, Alex Salazar and Nate Barbettini will break down the emerging AI architecture that makes action possible, and how it differs from traditional integration approaches.

Systems

Confessions of a Data Guy

AUGUST 29, 2024

I still remember the good ole days when Apache Spark was fresh and hot, hardly anyone was using it, except a few poor AWS Glue and EMR users … Lord have mercy on their ragged souls. It’s funny how that GOAT of a tool went from being used by a few companies for extremely large […] The post Apache Spark’s Most Annoying Use Case appeared first on Confessions of a Data Guy.

AWS 147

AWS IT Data Big Data 147

Jesse Anderson

AUGUST 28, 2024

In the spring of 2024, I ran a new survey to gather more data for my Data Teams book and update my 2023 and 2020 surveys. In total, we had 81 respondents. This survey was designed to get information about how management uses data teams, the value they’re creating, and how they’re creating it. The survey asked about the best and worst practices that teams are using or experiencing.

Consulting 147

Consulting Data Big Data Data Engineering 147

Analytics Vidhya

AUGUST 7, 2024

Introduction Apache Airflow is a crucial component in data orchestration and is known for its capability to handle intricate workflows and automate data pipelines. Many organizations have chosen it due to its flexibility and strong scheduling capabilities. Yet, as data requirements change, Airflow’s lack of scalability, real-time processing capabilities, and setup complexity may lead to […] The post Airflow Alternatives for Data Orchestration appeared first on Analytics Vidhya.

Data Pipeline 192

Data Pipeline Data Process Data Workflow 192

ArcGIS

AUGUST 23, 2024

With a new GIS mapping tool you can map the most visited national parks (and much more!) to explore your spatial data even further.

Designing 145

Designing Data 145

KDnuggets

AUGUST 12, 2024

Want to take the next step in your journey to becoming a data scientist? Check out these Python libraries for data science that you can't do without.

Python 152

Python Data Science Data 152

The Pragmatic Engineer

AUGUST 14, 2024

I (Gergely) sometimes get reachouts to do talks at events in Amsterdam (where I am based,) the Netherlands, or somewhere in Europe. Unfortunately, rarely do talks – I do one conference per year. However, I asked around in the community about tech professionals who do paid talks that software engineers find interesting, engaging, and educational.

Software Engineer 138

Software Engineer Software Engineering Education Architecture 138

Speaker: Andrew Skoog, Founder of MachinistX & President of Hexis Representatives

Manufacturing is evolving, and the right technology can empower—not replace—your workforce. Smart automation and AI-driven software are revolutionizing decision-making, optimizing processes, and improving efficiency. But how do you implement these tools with confidence and ensure they complement human expertise rather than override it? Join industry expert Andrew Skoog as he explores how manufacturers can leverage automation to enhance operations, streamline workflows, and make smarter, data-dri

Manufacturing

Engineering at Meta

AUGUST 5, 2024

AI networks play an important role in interconnecting tens of thousands of GPUs together, forming the foundational infrastructure for training, enabling large models with hundreds of billions of parameters such as LLAMA 3.1 405B. This week at ACM SIGCOMM 2024 in Sydney, Australia, we are sharing details on the network we have built at Meta over the past few years to support our large-scale distributed AI training workload.

Transportation 131

Transportation Designing Architecture Data Ingestion 131

databricks

AUGUST 6, 2024

Fueled by the exponential growth in external data and AI for innovation, organizations across all industries are looking for effective ways to collaborate.

Data 140

Data 140

Waitingforcode

AUGUST 22, 2024

Code organization and assertions flow are both important but even them, they can't guarantee your colleagues' adherence to the unit tests. There are other user-facing attributes to consider as well.

Coding 130

Coding 130

ArcGIS

AUGUST 21, 2024

Does your map need a north arrow? It depends.

IT 136

IT Designing 136

Advertisement

With Airflow being the open-source standard for workflow orchestration, knowing how to write Airflow DAGs has become an essential skill for every data engineer. This eBook provides a comprehensive overview of DAG writing features with plenty of example code. You’ll learn how to: Understand the building blocks DAGs, combine them in complex pipelines, and schedule your DAG to run exactly when you want it to Write DAGs that adapt to your data at runtime and set up alerts and notifications Scale you

Data Engineering

KDnuggets

AUGUST 15, 2024

The AI and ML complexity results in a growing number and diversity of jobs that require AI & ML expertise. We’ll give you a rundown of these jobs regarding the technical skills they need and the tools they employ.

Machine Learning 148

Machine Learning 148

The Pragmatic Engineer

AUGUST 14, 2024

I (Gergely) sometimes get reachouts to do talks at events in Amsterdam (where I am based,) the Netherlands, or somewhere in Europe. Unfortunately, rarely do talks – I do one conference per year. However, I asked around in the community about tech professionals who do paid talks that software engineers find interesting, engaging, and educational.

Software Engineer 130

Software Engineer Software Engineering Education Architecture 130

Seattle Data Guy

AUGUST 8, 2024

If you work in data, then AI is everywhere at this point. But whether AI is hype or reality doesn’t change the fact that data engineers will play a major role in ensuring that the data sets that are utilized for the growing use cases are usable both by machines and humans. Whether that data… Read more The post Essential Skills for Data Engineers in the Age of AI appeared first on Seattle Data Guy.

Data Engineering 130

Data Engineering Data Engineer Engineering Utilities 130

databricks

AUGUST 1, 2024

Today, we are excited to announce that Lakehouse Federation in Unity Catalog is now Generally Available (GA) across AWS, Azure, and GCP! Lakehouse.

AWS 139

AWS 139

Speaker: Tamara Fingerlin, Developer Advocate

In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!

Data

Waitingforcode

AUGUST 9, 2024

Testing batch jobs is not the same as testing streaming ones. Although the transformation (the WHAT from the previous article) is similar in both cases, more complete validation tests on the job logic are not. After all, streaming jobs often iteratively build the final outcome while the batch ones generate it in a single pass.

Building 130

Building IT 130

ArcGIS

AUGUST 14, 2024

Different thematic map types are better at supporting some questions than others. Here are a range of alternative approaches.

Designing 136

Designing 136

KDnuggets

AUGUST 23, 2024

Learn large language models with these free resources from Deeplearning.AI, Microsoft, AWS, and more.

AWS 147

AWS 147

Data Engineering Weekly

AUGUST 6, 2024

TL;DR Aswin and I are thrilled to announce the release of the first version of our comprehensive guide for evaluating Change Data Capture. CDC Evaluation Guide Google Sheet Link: [link] CDC Evaluation Guide Github Link: [link] Change Data Capture (CDC) is a powerful technology in data engineering that allows for continuously capturing changes (inserts, updates, and deletes) made to source systems.

Data Lake 126

Data Lake Data Warehouse Database Data Architecture 126

Advertisement

Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.

Cloud

Seattle Data Guy

AUGUST 1, 2024

What would you do if you became the head or director of data for a 1,000-person company? Yesterday, you were plugging along as an analyst, and now, suddenly, you have all these new responsibilities. Figuring out where to start is part of the job. You’d probably feel a strong temptation to freak out. Who wouldn’t?… Read more The post How To Run A Data Team As A New Head Of Data appeared first on Seattle Data Guy.

Data 130

Data Consulting Big Data Data Analytics 130

databricks

AUGUST 2, 2024

Training a high-quality machine learning model requires careful data and feature preparation. To fully utilize raw data stored as tables in Databricks, running.

Engineering 135

Engineering Raw Data Machine Learning Utilities 135

Waitingforcode

AUGUST 1, 2024

Welcome to the second blog post dedicated to the previous Data+AI Summit. This time I'm going to share with you a summary of Apache Spark talks.

Data 130

Data 130

ArcGIS

AUGUST 16, 2024

How to combine Living Atlas layers into a plausibly 1890s style ArcGIS Pro basemap. And thoughts on time travel.

132

132

Advertisement

Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?

Business Intelligence

KDnuggets

AUGUST 6, 2024

Learn statistics through interactive books, code examples, cheat sheets, guides, and tools documentation.

Coding 145

Coding Data Science Data 145

Snowflake

AUGUST 1, 2024

Since introducing Snowpark Container Services, we’ve seen overwhelming adoption across industries from customers and partners, including Landing.AI , Relational.AI , H20.AI , SailPoint , AIR MILES , Spark NZ , and Eutelsat OneWeb. These organizations and many more are using Snowpark Container Services capabilities to easily and securely deploy everything from custom front-ends and large-scale ML training and inference to open source and homegrown models, all securely within Snowflake.

Deep Learning 126

Deep Learning Government AWS Architecture 126

Engineering at Meta

AUGUST 27, 2024

At Meta, we’ve been diligently working to incorporate privacy into different systems of our software stack over the past few years. Today, we’re excited to share some cutting-edge technologies that are part of our Privacy Aware Infrastructure (PAI) initiative. These innovations mark a major milestone in our ongoing commitment to honoring user privacy.

Programming Language 118

Programming Language Coding Data Warehouse Systems 118

databricks

AUGUST 26, 2024

We're excited to announce the general availability of hybrid search in Mosaic AI Vector Search. Hybrid search is a powerful feature that combines.

Data Science 134

Data Science Data 134

Speaker: Ben Epstein, Stealth Founder & CTO | Tony Karrer, Founder & CTO, Aggregage

When tasked with building a fundamentally new product line with deeper insights than previously achievable for a high-value client, Ben Epstein and his team faced a significant challenge: how to harness LLMs to produce consistent, high-accuracy outputs at scale. In this new session, Ben will share how he and his team engineered a system (based on proven software engineering approaches) that employs reproducible test variations (via temperature 0 and fixed seeds), and enables non-LLM evaluation m

Software Engineer