Analytics Vidhya

JUNE 25, 2023

In a data-driven world, behind-the-scenes heroes like data engineers play a crucial role in ensuring smooth data flow. Imagine being an online shopper who suddenly receives irrelevant recommendations. A data engineer investigates the issue, identifies a glitch in the e-commerce platform’s data funnel, and swiftly implements seamless data pipelines.

Data Engineering 220

Data Engineering Data Engineer Engineering Data Pipeline 220

Start Data Engineering

JUNE 30, 2023

1. Introduction 2. What is self-serve? 2.1. Components of a self-serve platform 3. Building a self-serve data platform 3.1. Creating dataset(s) 3.1.1. Gather requirements 3.1.2. Get data foundations right 3.2. Accessing data 3.3. Identify and remove dependencies 4. Conclusion 5. Further reading 6. References 1. Introduction Most companies want to build a self-serve data platform.

Building 130

Building Datasets Data Accessible 130

Waitingforcode

JUNE 30, 2023

Surprised? You shouldn't. I've always been eager to learn, including 5 years ago when for the first time, I left my Apache Spark comfort zone to explore Apache Beam. Since then I had a chance to write some Dataflow streaming pipelines to fully appreciate this technology and work on AWS, GCP, and Azure. But there is some excitement for learning-from scratch I miss.

AWS 130

AWS Technology 130

Confessions of a Data Guy

JUNE 28, 2023

I’ve been a dog licking my wounds for some time now. Over on my Substack newsletter, I’ve been doing a small series on DSA (Data Structures and Algorithms). I tackled some of the easier stuff first, like Linked Lists, Binary Search, and the like. What’s more, I actually did most of it in Rust, since […] The post Exploring Graphs in Rust.

Algorithm 130

Algorithm Data IT Big Data 130

Advertisement

Generative AI is upending the way product developers & end-users alike are interacting with data. Despite the potential of AI, many are left with questions about the future of product development: How will AI impact my business and contribute to its success? What can product managers and developers expect in the future with the widespread adoption of AI?

Data

Analytics Vidhya

JUNE 24, 2023

Introduction In the era of data-driven decision-making, having accurate data modeling tools is essential for businesses aiming to stay competitive. As a new developer, a robust data modeling foundation is crucial for effectively working with databases. Properly configured data structures ensure a smoother workflow and prevent data loss or misplacement.

Database 211

Database Utilities Data Data Science 211

databricks

JUNE 29, 2023

Introduction We are thrilled to unveil the English SDK for Apache Spark, a transformative tool designed to enrich your Spark experience. Apache Spark™.

Programming Language 145

Programming Language Programming Designing 145

Lyft Engineering

JUNE 28, 2023

Written by Konstantin Gizdarski and Martin Liu at Lyft. In early 2022, Lyft already had a comprehensive Machine Learning Platform called LyftLearn composed of model serving , training , CI/CD, feature serving , and model monitoring systems. On the real-time front, LyftLearn supported real-time inference and input feature validation. However, streaming data was not supported as a first-class citizen across many of the platform’s systems — such as training, complex monitoring, and others.

Machine Learning 115

Machine Learning Building Metadata Kafka 115

Analytics Vidhya

JUNE 24, 2023

Introduction We had an amazing opportunity to learn from Mr. Pavan. He is an experienced data engineer with a passion for problem-solving and a drive for continuous growth. Throughout the conversation, Mr. Pavan shares his journey, inspirations, challenges, and accomplishments. Thus, providing valuable insights into the field of data engineering. As we explore Mr.

Data Engineering 204

Data Engineering Data Engineer Engineering Data 204

Confluent

JUNE 28, 2023

Get an introduction into the world of events and event-driven architecture in Apache Kafka. Learn what events are and the role they play in event design, event streaming, and event-driven design.

Kafka 109

Kafka Architecture Designing 109

databricks

JUNE 27, 2023

Generative AI will have a transformative impact on every business. Databricks has been pioneering AI innovations for a decade, actively collaborating with thousands.

Building 117

Building Data 117

Advertisement

Many organizations today are unlocking the power of their data by using graph databases to feed downstream analytics, enahance visualizations, and more. Yet, when different graph nodes represent the same entity, graphs get messy. Watch this essential video with Senzing CEO Jeff Jonas on how adding entity resolution to a graph database condenses network graphs to improve analytics and save your analysts time.

Database

KDnuggets

JUNE 29, 2023

Large language models are getting released left right and center, and if you want to understand them better you need to know about NLP. Here are 5 Free books to help you.

Process 103

Process 103

Databand.ai

JUNE 28, 2023

Data Pipeline Observability: A Model For Data Engineers Eitan Chazbani June 29, 2023 Data pipeline observability is your ability to monitor and understand the state of a data pipeline at any time. Specifically, observability provides insights into the pipeline’s internal states and how they interact with the system’s outputs. We believe the world’s data pipelines need better data observability.

Data Pipeline 98

Data Pipeline Data Engineering Data Engineer Engineering 98

Towards Data Science

JUNE 27, 2023

The Top 5 Features for Efficient Data Manipulation This April, pandas 2.0.0 was officially launched , making huge waves across the data science community. Photo by Yancy Min on Unsplash. Due to its extensive functionality and versatility, pandas has secured a place in every data scientist’s heart. From data input/output to data cleaning and transformation, it’s nearly impossible to think about data manipulation without import pandas as pd, right ?

Pipeline-centric 98

Pipeline-centric Data Science Datasets Machine Learning 98

Knowledge Hut

JUNE 26, 2023

Having knowledge of real-world software applications or projects are very essential for any projects for backend developers aspiring software engineers or developers. The portfolio projects showcase their talents and skills whenever they try to look for new opportunities and jobs. This article is mainly focused on explaining different backend projects for beginners or students, intermediate learners, or those who have mid enough software development experience building large scalable projects.

Portfolio 98

Portfolio Project Food Media 98

Speaker: Scott Sehlhorst

We know we want to create products which our customers find to be valuable. Whether we label it as customer-centric or product-led depends on how long we've been doing product management. There are three challenges we face when doing this. The obvious challenge is figuring out what our users need; the non-obvious challenges are in creating a shared understanding of those needs and in sensing if what we're doing is meeting those needs.

Certification

KDnuggets

JUNE 28, 2023

KDnuggets' latest cheat sheet presents you with an impressive array of advanced tools and resources designed to support your data science game. They cover a wide range of applications, from understanding complex scientific literature to writing high-quality manuscripts and more.

Data Science 100

Data Science Data Designing Machine Learning 100

databricks

JUNE 27, 2023

Today, we are thrilled to announce LakehouseIQ, a knowledge engine that learns the unique nuances of your business and data to power natural.

Engineering 110

Engineering Data 110

Uber Engineering

JUNE 29, 2023

Experience the power of row-level secondary indexing in Apache Parquet, enabling 3-20X faster upserts and unlocking new possibilities for efficient table ACID operations in today’s Lakehouse architecture.

Architecture 97

Architecture Data 97

Knowledge Hut

JUNE 26, 2023

Welcome to the world of data engineering, where the power of big data unfolds. If you're aspiring to be a data engineer and seeking to showcase your skills or gain hands-on experience, you've landed in the right spot. Get ready to delve into fascinating data engineering project concepts and explore a world of exciting data engineering projects in this article.

Data Engineering 98

Data Engineering Data Engineer Coding Project 98

Speaker: Timothy Chan, PhD., Head of Data Science

Are you ready to move beyond the basics and take a deep dive into the cutting-edge techniques that are reshaping the landscape of experimentation? 🌐 From Sequential Testing to Multi-Armed Bandits, Switchback Experiments to Stratified Sampling, Timothy Chan, Data Science Lead, is here to unravel the mysteries of these powerful methodologies that are revolutionizing how we approach testing.

Data Science

KDnuggets

JUNE 30, 2023

Every job is at risk. Here’s how you can AI-proof your career.

Data 154

Data Data Science 154

databricks

JUNE 27, 2023

Data teams face many challenges to quickly access the right data primarily due to data fragmentation, time and cost involved in consolidating data.

Accessible 104

Accessible Accessibility Data 104

Snowflake

JUNE 30, 2023

Since launching this year’s contest in October, receiving hundreds of submissions, and completing three rounds of judging, the wait is over: Maxa is the 2023 Snowflake Startup Challenge grand prize winner! Maxa’s goal is to automate financial and operations ERP insights extremely fast and without requiring special skills. To make that happen, it leverages the breadth of the Snowflake platform to transform raw data from multiple financial and operational systems into a common data model that user

Unstructured Data 94

Unstructured Data Raw Data Python SQL 94

ArcGIS

JUNE 29, 2023

This third blog in a series provides a set of sample tools to migrate a personal geodatabase from ArcMap, to a file or mobile geodatabase in ArcGIS Pro.

Data 97

Data Data Management Management 97

Advertisement

Start-ups & SMBs launching products quickly must bundle dashboards, reports, & self-service analytics into apps. Customers expect rapid value from your product (time-to-value), data security, and access to advanced capabilities. Traditional Business Intelligence (BI) tools can provide valuable data analysis capabilities, but they have a barrier to entry that can stop small and midsize businesses from capitalizing on them.

Data Analysis

KDnuggets

JUNE 28, 2023

10 ChatGPT Plugins for Data Science Cheat Sheet • Noteable Plugin: The ChatGPT Plugin That Automates Data Analysis • 3 Ways to Access Claude AI for Free • What are Vector Databases and Why Are They Important for LLMs?

Data Analysis 93

Data Analysis Data Science Database Data 93

databricks

JUNE 28, 2023

In this blog post, we will review the advancements in Spark Structured Streaming since we announced Project Lightspeed a year ago, from performance.

Project 97

Project Engineering 97

ThoughtSpot

JUNE 30, 2023

Pride is more than just a month-long celebration; it is a powerful movement that reminds us of the importance of equality, acceptance, and love. It is that special time of year for the global queer community to come together to celebrate, commemorate, and continue to push for progress. It’s no different here at ThoughtSpot. We believe in creating an inclusive environment where everyone feels seen, heard, and valued.

Education 82

Education Recruitment Programming Building 82

ArcGIS

JUNE 26, 2023

Tax Parcel Data Management helps to inventory tax parcels from record information and share this info with internal and external stakeholders.

Data Management 97

Data Management Management Data Government 97

Speaker: Aarushi Kansal, AI Leader & Author and Tony Karrer, Founder & CTO at Aggregage

Software leaders who are building applications based on Large Language Models (LLMs) often find it a challenge to achieve reliability. It’s no surprise given the non-deterministic nature of LLMs. To effectively create reliable LLM-based (often with RAG) applications, extensive testing and evaluation processes are crucial. This often ends up involving meticulous adjustments to prompts.

Building

KDnuggets

JUNE 26, 2023

This list of the most commonly used machine learning algorithms in Python and R is intended to help novice engineers and enthusiasts get familiar with the most commonly used algorithms.

Machine Learning 93

Machine Learning Algorithm Python Engineering 93

databricks

JUNE 27, 2023

The fundamental principles of governance – accountability, compliance, quality, and transparency – that are essential for data management have now become equally imperative for.

Government 87

Government Data Data Management Management 87

ThoughtSpot

JUNE 27, 2023

Active listening is an admired and sought after skill in both the professional and personal sphere. After all, who doesn’t love to be heard? But what happens when we apply that mindset to the way our organizations solicit feedback and interact with our customers? We don’t have to make any assumptions to answer this question, because we have the data.

Programming 82

Programming Data Science Data Analytics Process 82

ArcGIS

JUNE 26, 2023

Explore some high-level mapping and visualization features in ArcGIS Pro and learn how to find out more at the User Conference.

Data 97

Data 97

Speaker: Anne Steiner and David Laribee

As a concept, Developer Experience (DX) has gained significant attention in the tech industry. It emphasizes engineers’ efficiency and satisfaction during the product development process. As product managers, we need to understand how a good DX can contribute not only to the well-being of our development teams but also to the broader objectives of product success and customer satisfaction.

Engineering