databricks

APRIL 24, 2024

Unlock the power of Apache Spark™ with Unity Catalog Lakeguard on Databricks Data Intelligence Platform. Run SQL, Python & Scala workloads with full data governance & cost-efficient multi-user compute.

Data Governance 120

Data Governance Government Scala SQL 120

Waitingforcode

FEBRUARY 3, 2023

I've written my first Kubernetes on Apache Spark blog post in 2018 with a try to answer the question, what Kubernetes can bring to Apache Spark? Four years later this resource manager is a mature Spark component, but a new question has arisen in my head. Should I stay on YARN or switch to Kubernetes?

Management 130

Management 130

Waitingforcode

JULY 6, 2023

That's often a dilemma, whether we should put multiple sinks working on the same data source in the same or in different Apache Spark Structured Streaming applications? Both solutions may be valid depending on your use case but let's focus here on the former one including multiple sinks together.

Data 130

Data 130

Waitingforcode

JUNE 23, 2023

Shuffle is a permanent point in the What's new in Apache Spark series. It's often one the most time consuming part of the jobs and knowing the improvement simply helps writing better pipelines.

IT 130

IT 130

Confessions of a Data Guy

JULY 7, 2023

It seems like there is someone at the base of a titering rock with a crowbar, picking and […] The post Ballista (Rust) vs Apache Spark. With the rise of Rust, new tools like DuckDB, Polars, and whatever else, things do seem to shifting at a fundamental level. A Tale of Woe. appeared first on Confessions of a Data Guy.

Data Engineering 130

Data Engineering Data Engineer Engineering Data 130

Waitingforcode

FEBRUARY 3, 2023

Now, when I'm a data engineer, I feel the same for Apache Spark. It's difficult to see all the use cases of a framework. Back in time, when I was a backend engineer, I never succeeded to see all applications of Spring framework. Fortunately, the community is there to show me some outstanding features!

IT 130

IT Data Engineering Data Engineer Engineering 130

databricks

JUNE 29, 2023

Introduction We are thrilled to unveil the English SDK for Apache Spark, a transformative tool designed to enrich your Spark experience. Apache Spark™.

Programming Language 145

Programming Language Programming Designing 145

databricks

FEBRUARY 27, 2024

Introduction Apache Spark™ Structured Streaming is a popular open-source stream processing platform that provides scalability and fault tolerance, built on top of the S.

Process 105

Process Data Engineering Data Engineer Engineering 105

Waitingforcode

JULY 21, 2023

Starting from Apache Spark 3.2.0 is now possible to load an initial state of the arbitrary stateful pipelines. Even though the feature is easy to implement, it hides some interesting implementation details!

IT 130

IT 130

databricks

SEPTEMBER 15, 2023

Today, we are happy to announce the availability of Apache Spark™ 3.5 on Databricks as part of Databricks Runtime 14.0. We extend our s.

Engineering 103

Engineering 103

Waitingforcode

MAY 24, 2023

Apache Spark is infamous for its correctness issue for chained stateful operations. Fortunately things get improved in each release. The most recent one, the 3.4.0, also got some important changes on that field!

IT 130

IT 130

Waitingforcode

MAY 17, 2023

Finally, the time has come to start the analysis of the new features in Apache Spark. The first of them that grabbed my attention was the Async progress tracking from Structured Streaming.

130

130

databricks

SEPTEMBER 21, 2023

Introduction In this blog post, we'll explore a set of advanced SQL functions available within Apache Spark that leverage the HyperLogLog algorithm, enabling.

Algorithm 99

Algorithm SQL Data Engineering Data Engineer 99

Waitingforcode

FEBRUARY 3, 2023

It's time for the last "What's new in Apache Spark 3.3.0." before a break. Today we'll see what changed in PySpark. Spoiler alert: Pandas users should find one feature very exciting!

IT 100

IT 100

databricks

NOVEMBER 6, 2023

In Apache Spark™, Python User-Defined Functions (UDFs) are among the most popular features. They empower users to craft custom code tailored to their u.

Python 102

Python Coding Data Engineering Data Engineer 102

databricks

APRIL 18, 2023

Last year Spark Connect was introduced at the Data and AI Summit. As part of the recently released Apache SparkTM 3.4, Spark Connect.

Data 79

Data Engineering 79

databricks

MAY 15, 2023

Apache Spark Structured Streaming is the leading open source stream processing platform. It is also the core technology that powers streaming on the.

Technology 99

Technology Process IT Engineering 99

Start Data Engineering

JUNE 19, 2020

Intro A lot of tutorials show how to write spark code with just the API and code samples, but they do not explain how to write “efficient Apache Spark” code.

Coding 130

Coding IT Data 130

databricks

APRIL 14, 2023

Today, we are happy to announce the availability of Apache Spark™ 3.4 on Databricks as part of Databricks Runtime 13.0. We extend our s.

Data Engineering 105

Data Engineering Data Engineer Engineering Data 105

Cloudera

AUGUST 3, 2022

Fine grained access control (FGAC) with Spark. One approach is to use 3rd party tools (such as Privacera ) that integrate with Spark. This makes CDP the only platform where customers can use Spark with fine grained access control automatically, without requiring any additional tools or integrations. Starting with CDP 7.1.7

Datasets 84

Datasets BI Machine Learning Accessible 84

Data Science Blog: Data Engineering

JULY 4, 2022

Apache Spark is a Big Data tool that aims to handle large datasets in a parallel and distributed manner. Apache Spark began as a research project at UC Berkeley’s AMPLab, a student, researcher, and faculty collaboration centered on data-intensive application domains, in 2009. 5 best practices of Apache Spark 1.

Hadoop 52

Hadoop Big Data Datasets Scala 52

Cloudera

JUNE 10, 2021

Apache Spark is a very popular analytics engine used for large-scale data processing. CDP Operational Database Experience Experience (COD) is a CDP Public Cloud service that lets you create and manage operational database instances and it is powered by Apache HBase and Apache Phoenix. . Prerequisites .

Database 87

Database Data Engineering Data Engineer Big Data 87

ProjectPro

FEBRUARY 21, 2023

With around 35k stars and over 26k forks on Github, Apache Spark is one of the most popular big data frameworks used by 22,760 companies worldwide. Apache Spark is the most efficient, scalable, and widely used in-memory data computation tool capable of performing batch-mode, real-time, and analytics operations.

Scala 52

Scala Programming Language Java Hadoop 52

ProjectPro

FEBRUARY 16, 2023

Apache Spark is an open-source, distributed computing system for big data processing and analytics. Today, the Apache Spark project has over 1,000 contributors from over 250 companies worldwide. Thus, there is a tremendous demand for spark professionals in the coming years. Indeed recently posted nearly 2.4k

Big Data 52

Big Data Scala Machine Learning Hadoop 52

Waitingforcode

OCTOBER 17, 2023

I've already written about watermarks in a few places in the blog but despite that, I still find things to refresh. One of them is the watermark used to filter out the late data, which will be the topic of this blog post.

Data 130

Data 130

Waitingforcode

AUGUST 10, 2023

There are probably not that many people working today on the flat files with Structured Streaming than 5 years ago thanks to the table file formats. However, if you are in this group and are still generating CSVs or JSONs with the streaming sink, brace yourself, the memory problems are coming if you don't take action!

130

130

Data Engineering Podcast

DECEMBER 9, 2018

Summary Apache Spark is a popular and widely used tool for a variety of data oriented projects. Jean George Perrin has been so impressed by the versatility of Spark that he is writing a book for data engineers to hit the ground running. Can you start by explaining what Spark is? Who uses Spark?

Scala 100

Scala MySQL Kafka Hadoop 100

Waitingforcode

AUGUST 2, 2023

When you wrote your first arbitrary stateful processing pipelines, the state expiration is maybe the first tricky point you had to deal with. Why is that? After all, it's just about setting the timeout, doesn't it? Most of the time, yes, but there is an exception.

Process 130

Process IT 130

Rockset

NOVEMBER 4, 2021

Apache Spark is an open-source project that was started at UC Berkeley AMPLab. Even though Spark is written in Scala, you can interact with Spark with multiple languages like Spark, Python, and Java. Rockset actually complements Apache Spark for real-time analytics.

Scala 52

Scala Java AWS Hadoop 52

databricks

JANUARY 9, 2024

Ray is an open-source unified compute framework that simplifies scaling AI and Python workloads in a distributed environment. Since we introduced support for.

Python 99

Python Engineering 99

Waitingforcode

FEBRUARY 20, 2024

You can limit the max throughput for Apache Spark Structured Streaming jobs for popular data sources such as Apache Kafka, Delta Lake, or raw files. Have you known that you can also control the lower limit, at least for Apache Kafka? I bet you know it already.

Kafka 130

Kafka IT Data 130

Knowledge Hut

APRIL 25, 2024

If you pursue the MSc big data technologies course, you will be able to specialize in topics such as Big Data Analytics, Business Analytics, Machine Learning, Hadoop and Spark technologies, Cloud Systems etc. There are a variety of big data processing technologies available, including Apache Hadoop, Apache Spark, and MongoDB.

Big Data 98

Big Data Technology NoSQL Hadoop 98

Analytics Vidhya

FEBRUARY 28, 2023

Introduction Azure Databricks is a fast, easy, and collaborative Apache Spark-based analytics platform that is built on top of the Microsoft Azure cloud. A collaborative and interactive workspace allows users to perform big data processing and machine learning tasks easily.

Big Data 310

Big Data Machine Learning Cloud Data Process 310

Waitingforcode

MARCH 22, 2024

Apache Spark leverages the observer design pattern for the framework-to-code communication. One of the consumers' implementations is StreamingQueryListener.

Coding 130

Coding Designing 130

databricks

FEBRUARY 27, 2023

Ray is a prominent compute framework for running scalable AI and Python workloads, offering a variety of distributed machine learning tools, large-scale hyperparameter.

Machine Learning 117

Machine Learning Python Engineering 117

databricks

JUNE 28, 2023

In this blog post, we will review the advancements in Spark Structured Streaming since we announced Project Lightspeed a year ago, from performance.

Project 95

Project Engineering 95

Waitingforcode

FEBRUARY 27, 2024

Apache Spark Structured Streaming relies on the micro-batch pattern which evaluates the same query in each execution. That's only a high level vision, though. Under-the-hood, there are many other interesting things that happen.

130

130