Scala - Data Engineering Digest

Scala as a Junior Developer

Rock the JVM

SEPTEMBER 17, 2023

Lucas’ story is shared by lots of beginner Scala developers, which is why I wanted to post it here on the blog. I’ve watched thousands of developers learn Scala from scratch, and, like Lucas, they love it! If you want to learn Scala well and fast, take a look at my Scala Essentials course at Rock the JVM. sum > 8 ).

Scala

Scala Coding Programming Java

A Functional Load Balancer with Scala, Http4s and Cats Effect

Rock the JVM

OCTOBER 29, 2023

Project Structure We will use Scala 3.3.0, values :+ Url ( "url1" )) assertEquals ( obtained , expected ) } } Let’s run tests in SBT: sbt:loadbalancer> test [ info] compiling 1 Scala source to ~/loadbalancer/target/scala-3.3.0/test-classes. info] compiling 2 Scala sources to ~/loadbalancer/target/scala-3.3.0/test-classes.

Scala

Scala Coding Algorithm Bytes

Useful classes for data engineers - Scala & Java

Waitingforcode

FEBRUARY 3, 2023

In this blog post I'll share with you a list of Java and Scala classes I use almost every time in data engineering projects. We all have our habits and as programmers, libraries and frameworks are definitely a part of the group. The part for Python will follow next week!

Scala

Scala Java Data Engineering Data Engineer

Webinars

LLMs in Production: Tooling, Process, and Team Structure

MORE WEBINARS

Scala?—?Pattern Matching

Medium Data Engineering

NOVEMBER 2, 2023

Pattern matching is a powerful feature in Scala that allows you to destructure and match values against patterns, making it a concise and… Continue reading on Medium »

Scala

Scala Data Engineering Data Engineer Data Science

Scala-Collections Quick Reference

Medium Data Engineering

OCTOBER 15, 2023

Scala provides a rich set of collections to work with data. These collections are divided into two main categories: mutable and immutable. Continue reading on Medium »

Scala

Scala Data Big Data Data Engineering

Understanding the Difference Between lit and typedLit in Scala Spark

Medium Data Engineering

NOVEMBER 24, 2023

When working with Spark in Scala, the lit and typedLit functions play a crucial role in manipulating and transforming data within… Continue reading on Medium »

Scala

Scala Data Datasets Data Engineering

Spark?—?Scala vs Python

Medium Data Engineering

OCTOBER 1, 2023

The choice between Python and Scala for Apache Spark depends on your specific use cases, project requirements, and your team’s familiarity… Continue reading on Medium »

Scala

Scala Python Project Big Data

Programming in Scala [Chapter 16]?—?Working with Lists

Medium Data Engineering

NOVEMBER 26, 2023

In-depth explanation on working with Lists in Scala Continue reading on Medium »

Scala

Scala Programming Java Data Engineering

Scala?—?Case Classes

Medium Data Engineering

OCTOBER 22, 2023

Scala case classes are a special type of class that is primarily used for modeling immutable data. They have several useful features and… Continue reading on Medium »

Scala

Scala Data Big Data Data Engineering

Demystifying Monads in Scala: A Beginner’s Guide with Examples

Medium Data Engineering

OCTOBER 31, 2023

Monads are an essential concept in functional programming and are particularly prevalent in Scala. They provide a structured way to handle… Continue reading on Medium »

Scala

Scala Programming Data Engineering Data Engineer

Scala Fold Functions

Medium Data Engineering

MAY 28, 2023

In Scala, foldLeft and foldRight are higher-order functions commonly used to perform iterative operations on collections. Both functions… Continue reading on Medium »

Scala

Scala Data Engineering Data Engineer Programming

Why Scala Dominates Data Engineering

Medium Data Engineering

SEPTEMBER 3, 2023

Discover why Scala outperforms other languages in data engineering with Apache Spark, the ultimate big data processing engine. Continue reading on Medium »

Scala

Scala Data Engineering Data Engineer Engineering

Programming in Scala [Chapter 15]?—?Case Classes and Pattern Matching

Medium Data Engineering

NOVEMBER 26, 2023

How to use case classes and pattern matching in Scala Continue reading on Medium »

Scala

Scala Programming Java Data Engineering

gRPC in Scala with Fs2 and Scalapb

Rock the JVM

OCTOBER 1, 2023

Scala is not officially supported at the moment however the ScalaPB library provides a good wrapper around the official gRPC Java implementation, it provides an easy API that enables translation of Protocol Buffers to Scala case classes with support for Scala3, Scala.js, and Java Interoperability. Setting Up. in ( file ( "protobuf" )).

Scala

Scala Metadata Coding Java

Decoding Spark: Navigating the Realm of Scala with Clarity?—?Unraveling the Enigma Between lit and…

Medium Data Engineering

NOVEMBER 26, 2023

Once upon a time in the bustling kingdom of DataLandia, there lived two siblings, Lit and TypedLit, in the vast realm of Scala Spark. The… Continue reading on Medium »

Scala

Scala Data Analysis Data Engineering Data Engineer

A Comprehensive Guide to Choosing the Best Scala Course

Rock the JVM

MAY 22, 2023

This article is all about choosing the right Scala course for your journey. How should I get started with Scala? Do you have any tips to learn Scala quickly? How to Learn Scala as a Beginner Scala is not necessarily aimed at first-time programmers. Which course should I take?

Scala

Scala Java Data Engineering Data Engineer

Programming in Scala [Chapter 9]?—?Control Abstraction

Medium Data Engineering

SEPTEMBER 27, 2023

Shows how to augment Scala’s basic control structures by using control abstractions Continue reading on Medium »

Scala

Scala Programming Programming Language Java

Programming in Scala [Chapter 12]?—?Traits

Medium Data Engineering

NOVEMBER 10, 2023

Using traits in Scala Continue reading on Medium »

Scala

Scala Programming Java Data Engineering

Programming in Scala [Chapter 10]?—?Composition and Inheritance

Medium Data Engineering

OCTOBER 1, 2023

This is part of my personal reading summary of the book — Programming in Scala. Take into account, we use Scala 2 since it is in industry and has vastly more resources available than Scala 3…

Scala

Scala Programming Programming Language Java

Programming in Scala [Chapter 13]?—?Packages and Imports

Medium Data Engineering

NOVEMBER 12, 2023

How to use packages and imports in Scala Continue reading on Medium »

Scala

Scala Programming Java Data Engineering

Programming in Scala [Chapter 14]?—?Assertions and Unit Testing

Medium Data Engineering

NOVEMBER 13, 2023

Learn how to execute proper tests in Scala Continue reading on Medium »

Scala

Scala Programming Java Data Engineering

Scala CLI Tutorial: Creating a CLI Sudoku Solver

Rock the JVM

JANUARY 8, 2023

Antonio is an alumnus of Rock the JVM, now a senior Scala developer with his own contributions to Scala libraries and junior devs under his mentorship. Which brings us to this article: Antonio originally started from my Sudoku backtracking article and built a Scala CLI tutorial for the juniors he’s mentoring.

Scala

Scala Java Coding Project

OAuth Authentication in Scala with Http4s

Rock the JVM

JULY 26, 2023

SNAPSHOT" , scalaVersion := scala3Version , libraryDependencies ++= Seq ( emberServer , emberClient , http4sDsl , http4sCirce , ciris , cirisCirce ) ++ circeLibs ) We use Scala 3 in this tutorial, if you need to use Scala 2.13, you can do so with minimal code changes. val Http4sVersion = "0.23.18" val CirisVersion = "3.2.0" pure [ F ].

Scala

Scala Coding Accessible Accessibility

Programming in Scala Chapter 1?—?A Scalable Language Summary

Medium Data Engineering

SEPTEMBER 23, 2023

An overview of Scala’s design, reasoning, and history Continue reading on Medium »

Scala

Scala Programming Designing Programming Language

Two-Factor Authentication in Scala with Http4s

Rock the JVM

JULY 26, 2023

If you want to master the Typelevel Scala libraries (including Http4s) with real-life practice, check out the Typelevel Rite of Passage course, a full-stack project-based course. HOTP scala implementation HOTP generation is quite tedious, therefore for simplicity, we will use a java library, otp-java by Bastiaan Jansen.

Scala

Scala Java Coding Algorithm

Best Practices to Follow for writing Spark jobs in Scala

Medium Data Engineering

MARCH 15, 2023

When writing Spark jobs in scala, there are some things that should be followed to ensure optimal performance and avoid potential issues. Continue reading on Medium »

Scala

Scala Data Engineering Data Engineer Python

Arbitrary stateful processing in PySpark with applyInPandasWithState

Waitingforcode

SEPTEMBER 27, 2023

It's always a huge pleasure to see the PySpark API covering more and more Scala API features. But since the API is a little bit different than the one available on the Scala side, I wanted to take a deeper look. Starting from Apache Spark 3.4.0 you can even write arbitrary stateful processing jobs!

Scala

Scala Process IT

REST APIs Using Play Framework and Scala: A Comprehensive Guide

Rock the JVM

SEPTEMBER 3, 2023

Play Framework “makes it easy to build web applications with Java & Scala”, as it is stated on their site, and it’s true. In this article we will try to develop a basic skeleton for a REST API using Play and Scala. PlayScala plugin defines default settings for Scala-based applications. import Keys._

Scala

Scala Database Project Data

HTTP Authentication with Scala and Http4s

Rock the JVM

JUNE 5, 2023

Http4s is one of the most powerful libraries in the Scala ecosystem, and it’s part of the Typelevel stack. If you want to master the Typelevel Scala libraries with real-life practice, check out the Typelevel Rite of Passage course, a full-stack project-based course. by Herbert Kateu Hey, it’s Daniel here. val scala3Version = "3.2.2"

Scala

Scala Coding Database Accessible

HTTP Authentication with Scala and Http4s

Rock the JVM

JUNE 5, 2023

Http4s is one of the most powerful libraries in the Scala ecosystem, and it’s part of the Typelevel stack. If you want to master the Typelevel Scala libraries with real-life practice, check out the Typelevel Rite of Passage course, a full-stack project-based course. by Herbert Kateu Hey, it’s Daniel here. val scala3Version = "3.2.2"

Scala

Scala Coding Database Accessible

Data Engineering?—?Scala vs Python for Spark

Medium Data Engineering

NOVEMBER 3, 2023

If you are a Data Engineer, you will most likely need to know python anyways. This really depends on what you want to do within data… Continue reading on Medium »

Data Engineering

Data Engineering Data Engineer Python Scala

A Backtracking Sudoku Solver in Scala

Rock the JVM

OCTOBER 30, 2022

This article is for Scala beginners. After you learn the language, the next big thing you need to master is how to write essential “algorithms” in Scala. This tends to make algorithms in Scala quite difficult. This article works identically for Scala 2 and Scala 3. map ( row => row. mkString ( " " )).

Scala

Scala Algorithm Java Python

Top 10 Skills (Mostly Mental Models) to Learn to Be a Scala Developer

Rock the JVM

NOVEMBER 6, 2022

This article is for aspiring Scala developers. As the Scala ecosystem matures and evolves, this is the best time to become a Scala developer, and in this piece you will learn the essential tools that you should master to be a good Scala software engineer. Read this article to understand what you need to work with Scala.

Scala

Scala Java Python Coding

Finagle Tutorial: Twitter’s RPC Library for Scala

Rock the JVM

NOVEMBER 27, 2022

This article is for Scala developers of all levels - you don’t need any fancy Scala knowledge to make the best out of this piece. For those who don’t know yet, almost the entire Twitter backend runs on Scala, and the Finagle library is at the core of almost all Twitter services. Finagle is an RPC library for distributed systems.

Scala

Scala Systems Algorithm Data

Making applyInPandasWithState less painful

Waitingforcode

OCTOBER 4, 2023

However, due to Python duck typing, some operations are more difficult and more risky to express in the code than in the strongly typed Scala API. Having applyInPandasWithState in the PySpark API is huge!

Scala

Scala Python Coding

Simple project to learn Spark Streaming from Kafka in Scala?—?Part 1

Medium Data Engineering

MARCH 26, 2023

Hands on Spark Streaming, Kafka, Scala !! Continue reading on Dev Genius »

Scala

Scala Kafka Project Data Engineering

Demystifying Spark Scala: Exploring the Differences Between explode and posexplode

Medium Data Engineering

NOVEMBER 29, 2023

Apache Spark provides powerful tools for processing and transforming data, and two functions that are often used in the context of working… Continue reading on Medium »

Scala

Scala Data Process Data Engineering

A Detailed Guide of Interview Questions on Apache Kafka

Analytics Vidhya

APRIL 28, 2023

It is a famous Scala-coded data processing tool that offers low latency, extensive throughput, and a unified platform to handle the data in real-time. Introduction Apache Kafka is an open-source publish-subscribe messaging application initially developed by LinkedIn in early 2011.

Kafka

Kafka Scala Data Process Coding

A Comprehensive Guide to ETL with Spark and Scala

Medium Data Engineering

NOVEMBER 29, 2023

Introduction: In the ever-expanding realm of big data, organizations face the challenge of efficiently processing and analyzing massive… Continue reading on Medium »

Scala

Scala Big Data Data Process

Scala For Big Data Engineering – Why should you care?

Advancing Analytics: Data Engineering

APRIL 23, 2020

The thought of learning Scala fills many with fear, its very name often causes feelings of terror. The truth is Scala can be used for many things; from a simple web application to complex ML (Machine Learning). The name Scala stands for “scalable language.” So what companies are actually using Scala?

Scala

Scala Data Engineering Data Engineer Big Data

Design Patterns in Python, Scala, and Java: A Data Engineering Perspective!

Medium Data Engineering

NOVEMBER 18, 2023

Throughout my 13+ years of journey in the IT industry, I have had the unique opportunity to work on all 3 programming languages. When I… Continue reading on Medium »

Scala

Scala Data Engineering Data Engineer Java

The Dog Days of PySpark

Confessions of a Data Guy

APRIL 15, 2023

PySpark is the abstraction that lets a bazillion Data Engineers forget about that blight Scala and cuddle their wonderfully soft and ever-kind Python code, while choking down gobs of data like some Harkonnen glutton. One of those things to hate and love, well … kinda hard not to love.

Scala

Scala Python Data Engineering Data Engineer

PySpark and vectorized User-Defined Functions

Waitingforcode

FEBRUARY 3, 2023

The Scala API of Apache Spark SQL has various ways of transforming the data, from the native and User-Defined Function column-based functions, to more custom and row-level map functions. PySpark doesn't have this mapping feature but does have the User-Defined Functions with an optimized version called vectorized UDF!

Scala

Scala SQL Data

Getting Started with Scala and Apache Kafka

Confluent

DECEMBER 8, 2020

If you’re getting started with Apache Kafka® and event streaming applications, you’ll be pleased to see the variety of languages available to start interacting with the event streaming platform. It […].

Kafka

Kafka Scala IT

Scala as a Junior Developer

A Functional Load Balancer with Scala, Http4s and Cats Effect

Webinars

Trending Sources

Useful classes for data engineers - Scala & Java

Webinars

Scala?—?Pattern Matching

Scala-Collections Quick Reference

Understanding the Difference Between lit and typedLit in Scala Spark

Spark?—?Scala vs Python

Programming in Scala [Chapter 16]?—?Working with Lists

Scala?—?Case Classes

Demystifying Monads in Scala: A Beginner’s Guide with Examples

Scala Fold Functions

Why Scala Dominates Data Engineering

Programming in Scala [Chapter 15]?—?Case Classes and Pattern Matching

gRPC in Scala with Fs2 and Scalapb

Decoding Spark: Navigating the Realm of Scala with Clarity?—?Unraveling the Enigma Between lit and…

A Comprehensive Guide to Choosing the Best Scala Course

Programming in Scala [Chapter 9]?—?Control Abstraction

Programming in Scala [Chapter 12]?—?Traits

Programming in Scala [Chapter 10]?—?Composition and Inheritance

Programming in Scala [Chapter 13]?—?Packages and Imports

Programming in Scala [Chapter 14]?—?Assertions and Unit Testing

Scala CLI Tutorial: Creating a CLI Sudoku Solver

OAuth Authentication in Scala with Http4s

Programming in Scala Chapter 1?—?A Scalable Language Summary

Two-Factor Authentication in Scala with Http4s

Best Practices to Follow for writing Spark jobs in Scala

Arbitrary stateful processing in PySpark with applyInPandasWithState

REST APIs Using Play Framework and Scala: A Comprehensive Guide

HTTP Authentication with Scala and Http4s

HTTP Authentication with Scala and Http4s

Data Engineering?—?Scala vs Python for Spark

A Backtracking Sudoku Solver in Scala

Top 10 Skills (Mostly Mental Models) to Learn to Be a Scala Developer

Finagle Tutorial: Twitter’s RPC Library for Scala

Making applyInPandasWithState less painful

Simple project to learn Spark Streaming from Kafka in Scala?—?Part 1

Demystifying Spark Scala: Exploring the Differences Between explode and posexplode

A Detailed Guide of Interview Questions on Apache Kafka

A Comprehensive Guide to ETL with Spark and Scala

Scala For Big Data Engineering – Why should you care?

Design Patterns in Python, Scala, and Java: A Data Engineering Perspective!

The Dog Days of PySpark

PySpark and vectorized User-Defined Functions

Getting Started with Scala and Apache Kafka

Stay Connected