Research Survey: Productivity benefits from Databricks Assistant

In the fast-paced landscape of data science and engineering, integrating Artificial Intelligence (AI) has become integral for enhancing productivity. We’ve seen many tools emerge and transform the lives of data practitioners, making complex tasks easier and encouraging innovation. When we launched Databricks Assistant in Public Preview in July of 2023, we designed it exclusively for streamlining efficiency amongst data scientists, analysts, and engineers. To better understand how well we’re achieving this goal, we decided to survey some of our top users across multiple organizations, varying in experience. 

Purpose of the Survey

To better understand Databricks Assistant’s impact on data professionals, we meticulously designed this survey to capture a broad spectrum of user experiences. Our goal in sending out this survey was to not only better understand the Assistant's impact on the everyday lives of our users but also to understand better who’s using the Assistant the most, how often the Assistant is being summoned, and users’ perceptions of the response quality.

We recognize that productivity and satisfaction are often hard to measure strictly quantitatively, so we purposefully designed the survey around a mix of both qualitative and quantitative questions. Quantitatively, we captured data around how frequently users engaged with the Assistant, what their main use cases were, and utilized Likert scales to gauge satisfaction and productivity. Qualitatively, we asked participants to delve deeper into the common issues they’ve experienced in interacting with the Assistant, what their favorite features and interactions have been and how these elements have altered their workflow. 

The 70 responses we received came primarily from engineers (31.9%), though we also received responses from data scientists (18.8%%), SQL analysts (23.2%), and other professionals (26.1%). These respondents spanned a broad range of experience levels, from 0 to over 20 years in their respective fields. 

The survey was distributed to multiple organizations who’ve been eagerly taking advantage of our public preview. We made sure to emphasize the importance of candid feedback, in order to paint a comprehensive picture of Databricks Assistant’s current standing among active users and constructive feedback for future improvements.

Key Findings

The following findings highlight the three main takeaways from our investigation.

Finding 1: The Assistant has proven to be great at seamlessly integrating into users’ working environments and offering fast responses.

Our survey aimed to uncover the aspects that Databricks Assistant users liked the most. Two interrelated reasons emerged as the top highlights to 93.6% of respondents:

• Seamless workflow integration: Users claimed to enjoy how the Assistant is integrated directly into their existing Databricks environments. 
• Efficient and immediate assistance: Developers appreciated the Assistant’s quick and accurate responses, from generating Python and SQL to looking up Spark functionality. Databricks Assistant is seen as a major time-saver, eliminating the previous pain of having to consult external sources for answers.

“Databricks Assistant introduces an integrated approach to development, seamlessly incorporating AI throughout the process, from initial stages to execution." - Alaeddin Khader, Director of Data + AI, Core42 / G42

Finding 2: Developers go to the Assistant most often for writing code and debugging.

In the survey, we saw that software engineers, data scientists, and SQL analysts all primarily used the Assistant for two main reasons:

• Fixing errors/Troubleshooting: Most respondents (88.4%) reported they primarily interact with the Assistant due to its effective bug-fixing capabilities.
• Help writing code: About half (49.3%) of users stated they interacted with the Assistant primarily to help them write code. Databricks Assistant not only suggests code and improves speed but also the quality of solutions. 

“I was able to code 200+ lines of robust code this week in a language I've never coded before...leveraging Databrick's AI Assistant." - Josue A. Bogran, Solutions Architect Manager, Kythera Labs. 

Finding 3: The Assistant has made a substantial impact on time management.

The Assistant not only streamlines daily workflows but also significantly boosts time efficiency, as demonstrated by our findings:

• Quantified time saved: Over 72% of users claimed that the Assistant saved them at least 30% of their time on any given task.
• Enhanced focus: Respondents highlighted that the Assistant effectively frees up users' time, allowing them to concentrate on more strategic and high-value tasks.

“Databricks became even more powerful with the Databricks Assistant! This cutting-edge AI companion has revolutionized my data analysis journey, simplifying complex tasks and accelerating productivity." - Byron Exaporriton, Advanced Analytics Consultant, ABN AMRO Bank N.V.

With all of those findings in mind, when asked about the productivity boost provided by the Assistant on a scale from 1 (much less productive) to 5 (much more productive), a significant 55.5% of developers rated their experience with a 4 or 5. This feedback underscores the effectiveness of Databricks Assistant at streamlining workflows and correcting and writing code. 

Areas of Investment

While our survey revealed many of our strengths, it also highlighted some key areas for improvement. 

Investment 1: While many users praised the Assistant for its quick responses, a few noted areas for performance improvement.

There are several things we are working on to make Databricks Assistant faster and more efficient. We’ve transitioned to asynchronous content filtering, not only speeding up stream time but also focussing on delivering faster, better formatting. Additionally, we want to ensure consistent performance irrespective of conversation history.

Investment 2: Respondents noted that while the Assistant generally provides relevant information, there are occasional instances of outdated data.

We recognize Databricks Assistant can occasionally provide incorrect information, and are dedicated to building trust and improving the accuracy of our replies. First and foremost, we plan on ensuring Databricks-specific suggestions are up-to-date and continually modernizing our retrieval areas. Furthermore, we plan to incorporate more detailed feedback mechanisms on specific responses that we can use to self-evaluate and improve.  

Conclusion

We are committed to supporting data practitioners in enhancing efficiency and satisfaction in their everyday work. Our research found that in just the short timeline of our Public Preview, a significant portion of users prompted Databricks Assistant on a day-to-day basis (48.6%). We are continually learning how we can make our Assistant even better. As the field continues to evolve, we are optimistic that we will not only refine our Assistant to be even better but also are eager to see the innovations the broader research community will uncover.

Databricks Assistant is available now.

Follow the instructions here to enable the assistant. If you don't have an account, you can start with Databricks with a free trial.