Arbitrary stateful processing in PySpark with applyInPandasWithState
Waitingforcode
SEPTEMBER 27, 2023
It's always a huge pleasure to see the PySpark API covering more and more Scala API features. Starting from Apache Spark 3.4.0 you can even write arbitrary stateful processing jobs! But since the API is a little bit different than the one available on the Scala side, I wanted to take a deeper look.
Let's personalize your content