Analytics Vidhya

FEBRUARY 7, 2023

Introduction Data engineering is the field of study that deals with the design, construction, deployment, and maintenance of data processing systems. The goal of this domain is to collect, store, and process data efficiently and efficiently so that it can be used to support business decisions and power data-driven applications. This includes designing and implementing […] The post Most Essential 2023 Interview Questions on Data Engineering appeared first on Analytics Vidhya.

Data Engineering 240

Data Engineering Data Engineer Engineering Data 240

Confessions of a Data Guy

FEBRUARY 10, 2023

Hmm … data types. We all know they are important, but we don’t take them very seriously. I mean we know the difference between boolean, string, and integers, those are easy to get right. But we all get sloppy, sometimes we got the string and varchar route because we don’t spend enough time on the […] The post Data Types in Delta Lake + Spark.

Data 238

Data Big Data Data Engineering Data Engineer 238

Waitingforcode

FEBRUARY 9, 2023

Compaction is also a feature present in Apache Iceberg. However, it works a little bit differently than for Delta Lake presented last time. Why? Let's see in this new blog post!

IT 130

IT 130

Engineering at Meta

FEBRUARY 6, 2023

Facebook for iOS (FBiOS) is the oldest mobile codebase at Meta. Since the app was rewritten in 2012 , it has been worked on by thousands of engineers and shipped to billions of users, and it can support hundreds of engineers iterating on it at a time. After years of iteration , the Facebook codebase does not resemble a typical iOS codebase: It’s full of C++, Objective-C(++), and Swift.

Architecture 141

Architecture Coding Engineering Systems 141

Speaker: Timothy Chan, PhD., Head of Data Science

Are you ready to move beyond the basics and take a deep dive into the cutting-edge techniques that are reshaping the landscape of experimentation? 🌐 From Sequential Testing to Multi-Armed Bandits, Switchback Experiments to Stratified Sampling, Timothy Chan, Data Science Lead, is here to unravel the mysteries of these powerful methodologies that are revolutionizing how we approach testing.

Data Science

Analytics Vidhya

FEBRUARY 6, 2023

Introduction A design pattern is simply a repeatable solution for problems that keep on reoccurring. The pattern is not an actual code but a template that can be used to solve problems in different situations. Especially while working with databases, it is often considered a good practice to follow a design pattern. This ensures easy […] The post What are Data Access Object and Data Transfer Object in Python?

Accessible 234

Accessible Accessibility Python Database 234

Confessions of a Data Guy

FEBRUARY 7, 2023

As I started to use Rust on and off, more out of curiosity than anything, I discovered some specs of gold buried down in the depths. Some of the things I’m going to talk about, well … all of it, is probably fairly obvious to most Rust folk, but it’s enjoyable to learn what new […] The post Ownership and Borrowing in Rust – Data Engineering Gold Mine. appeared first on Confessions of a Data Guy.

Data Engineering 221

Data Engineering Data Engineer Engineering Data 221

Engineering at Meta

FEBRUARY 7, 2023

A lot has changed since the initial launch of our basemap in late 2020. We’re Meta now, but our mission remains the same: Giving people the power to build community and bring the world closer together. Across Meta, our family of applications (Facebook, Instagram, WhatsApp, among others) are using our basemap to connect people through functions like status updates, location sharing, and location-based searching.

Entertainment 133

Entertainment Transportation Data Schemas AWS 133

Analytics Vidhya

FEBRUARY 8, 2023

Introduction Amazon Redshift is a fully managed, petabyte-scale data warehousing Amazon Web Services (AWS). It allows users to easily set up, operate, and scale a data warehouse in the cloud. Redshift uses columnar storage techniques to store data efficiently and supports data warehousing workloads intelligence, reporting, and analytics. It allows users to perform complex queries […] The post Top 6 Amazon Redshift Interview Questions appeared first on Analytics Vidhya.

Amazon Web Services 221

Amazon Web Services AWS Data Warehouse Cloud 221

Confluent

FEBRUARY 8, 2023

Learn what windowing is, the difference between the four types of windows (hopping and tumbling, or session and sliding), and how to create them.

Kafka 140

Kafka 140

Teradata

FEBRUARY 9, 2023

In the aftermath of the 2008 financial crash, service providers have been subject to increasing rules & requirements. To what extent has this climate held back advances in data analytics?

Data Analytics 116

Data Analytics Data 116

Speaker: Anne Steiner and David Laribee

As a concept, Developer Experience (DX) has gained significant attention in the tech industry. It emphasizes engineers’ efficiency and satisfaction during the product development process. As product managers, we need to understand how a good DX can contribute not only to the well-being of our development teams but also to the broader objectives of product success and customer satisfaction.

Engineering

KDnuggets

FEBRUARY 6, 2023

KDnuggets and its partners have just released a Spend & Trends survey to provide you the opportunity to benchmark with your peers on how folks are spending and the mindsets around current trends.

IT 110

IT 110

Analytics Vidhya

FEBRUARY 6, 2023

Introduction While working with multiple projects, there are chances of issues with versions of packages in python; for example, a project needs a new version of a package, and another requires a different version. Sometimes the python version itself changes from project to project. Managing these different python versions and different versions of packages is […] The post Isolated Python Environments using Docker appeared first on Analytics Vidhya.

Python 218

Python Project Management Data Engineering 218

Picnic Engineering

FEBRUARY 8, 2023

Delivering the right events at low latency and with a high volume is critical to Picnic’s system architecture. In our previous blog, Dima Kalashnikov explained how we configure our Internal services pipeline in the Analytics Platform. In this post, we will explain how our team automates the creation of new data pipeline deployments. The step towards automation was an important improvement for us, as the previous setup was manual, slow, and error-prone.

Data Pipeline 105

Data Pipeline Kafka Data Architecture 105

Tweag

FEBRUARY 8, 2023

In this post I want to explore the current issues with developing parts of NixOS on macOS and how we can make this task easier. Why would I want to run a NixOS virtual machine on macOS? My colleague at Tweag, Dominic Steinitz, asked me this question after I shared my first minor achievement in this area, and it struck me that I have never described why exactly I run virtual machines (VMs) on my laptop and why I want to make it easier for myself (and others).

Systems 99

Systems Building Cloud IT 99

Speaker: Aarushi Kansal, AI Leader & Author and Tony Karrer, Founder & CTO at Aggregage

Software leaders who are building applications based on Large Language Models (LLMs) often find it a challenge to achieve reliability. It’s no surprise given the non-deterministic nature of LLMs. To effectively create reliable LLM-based (often with RAG) applications, extensive testing and evaluation processes are crucial. This often ends up involving meticulous adjustments to prompts.

Building

KDnuggets

FEBRUARY 7, 2023

Let's learn how ChatGPT can help us learn about Python. or really anything at all.

Python 160

Python 160

Analytics Vidhya

FEBRUARY 6, 2023

Introduction The demand for data to feed machine learning models, data science research, and time-sensitive insights is higher than ever thus, processing the data becomes complex. To make these processes efficient, data pipelines are necessary. Data engineers specialize in building and maintaining these data pipelines that underpin the analytics ecosystem.

Amazon Web Services 218

Amazon Web Services Data Pipeline Machine Learning Data Science 218

Confluent

FEBRUARY 7, 2023

Migrate Kafka clusters from ZooKeeper to KRaft with no downtime (early access), get improvements for Kafka Streams and Kafka Connect, and more.

Kafka 104

Kafka Accessible Accessibility 104

databricks

FEBRUARY 6, 2023

Advances in Natural Language Processing (NLP) have unlocked unprecedented opportunities for businesses to get value out of their text data. Natural Language Processing.

Process 92

Process Data Data Science Engineering 92

Advertisement

Are you trying to decide which entity resolution capabilities you need? It can be confusing to determine which features are most important for your project. And sometimes key features are overlooked. Get the Entity Resolution Evaluation Checklist to make sure you’ve thought of everything to make your project a success! The list was created by Senzing’s team of leading entity resolution experts, based on their real-world experience.

Project

KDnuggets

FEBRUARY 8, 2023

This post provides an overview of topics in linear programming, history, and recent advances, software packages, common problem specifications, and a case study using Toronto shelters data and the PuLP software package.

Programming 108

Programming Data 108

Analytics Vidhya

FEBRUARY 8, 2023

source: svitla.com Introduction Before jumping to the data warehouse interview questions, let’s first understand the overview of a data warehouse. A data warehouse is a system used for collecting and managing large amounts of data from various sources, such as transactional systems, log files, and external data sources. The data is then organized and structured […] The post Data Warehouse Interview Questions appeared first on Analytics Vidhya.

Data Warehouse 214

Data Warehouse Data Systems Management 214

Knowledge Hut

FEBRUARY 7, 2023

In this post, the Binary Search Algorithm will be covered. Finding a certain element in the list is the process of searching. The method is deemed successful and returns the element's location if the element is present in the list. If not, the search is deemed fruitless. will be covered. Finding a certain element in the list is the process of searching.

Algorithm 89

Algorithm Coding Java Programming 89

Towards Data Science

FEBRUARY 9, 2023

An experiment on BigQuery If you are processing a couple of MB or GB with your dbt model, this is not a post for you; you are doing just fine! This post is for those poor souls that need to scan terabytes of data in BigQuery to calculate some counts, sums, or rolling totals over huge event data on a daily or even at a higher frequency basis. In this post, I will go over a technique for enabling a cheap data injestion and cheap data consumption for “big data”.

Big Data 87

Big Data Raw Data Aggregated Data Data 87

Speaker: Margaret-Ann Seger, Head of Product, Statsig

Experimentation is often seen as an aspirational practice, especially at smaller, fast-moving companies who are strapped for time and resources. So, how can you get your team making decisions in a more data-driven way while continuing to remain lean and maintaining ship velocity? In this webinar, Margaret-Ann Seger, Head of Product at Statsig, will teach you how to build an experimentation culture from the ground-up, graduating from just getting started with data-driven development to operating

Building

KDnuggets

FEBRUARY 6, 2023

Collecting and labeling data in the real world can be time-consuming and expensive. This data can also come with quality, diversity, and quantity issues. Fortunately, problems like these can be helped with synthetic data.

Data 108

Data 108

Analytics Vidhya

FEBRUARY 5, 2023

Introduction In this technical era, Big Data is proven as revolutionary as it is growing unexpectedly. According to the survey reports, around 90% of the present data was generated only in the past two years. Big data is nothing but the vast volume of datasets measured in terabytes or petabytes or even more. Big data […] The post A Beginner’s Guide to the Basics of Big Data and Hadoop appeared first on Analytics Vidhya.

Big Data 205

Big Data Hadoop Datasets Data 205

ThoughtSpot

FEBRUARY 9, 2023

Two years ago, we announced our Databricks partnership —including the launch of ThoughtSpot for Databricks, which gives joint customers the ability to run ThoughtSpot search queries directly on the Databricks Lakehouse without the need to move any data. Since then, we’ve empowered teams at companies like Johnson & Johnson, NASDAQ, and Flyr to safely self-serve business-critical insights on governed and reliable data.

Government 82

Government SQL Machine Learning Cloud 82

Towards Data Science

FEBRUARY 6, 2023

How to setup a simple ETL pipeline with AWS Lambda that can be triggered via an API Endpoint or Schedule and write the results to an S3… Continue reading on Towards Data Science »

AWS 85

AWS Data Science Data Data Engineering 85

Speaker: David Bard, Principal at VP Product Coaching

In the fast-paced world of digital innovation, success is often accompanied by a multitude of challenges - like the pitfalls lurking at every turn, threatening to derail the most promising projects. But fret not, this webinar is your key to effective product development! Join us for an enlightening session to empower you to lead your team to greater heights.

Certification

KDnuggets

FEBRUARY 7, 2023

Kickstart your Data Engineering career with these curated GitHub repositories.

Data Engineering 160

Data Engineering Data Engineer Engineering Data 160

Analytics Vidhya

FEBRUARY 8, 2023

Introduction The February installment of the webinar series is now open! It’s a farewell time to your quest for finding the ideal data science learning platform, as Analytics Vidhya has arrived. Explore your ultimate data science destination where the emphasis is on supporting the community and fostering professional development. Attend expert-led DataHour sessions to boost […] The post February DataHour: Enhance Your Skills with Expert Sessions appeared first on Analytics Vidhya.

Data Science 200

Data Science Data Machine Learning Data Engineering 200

Cloudera

FEBRUARY 9, 2023

Special co-author credits: Adam Andras Toth, S oftware Engineer Intern With enterprises’ needs for data analytics and processing getting more complex by the day, Cloudera aims to keep up with these needs, offering constantly evolving, cutting-edge solutions to all your data related problems. Cloudera Stream Processing aims to take real-time data analytics to the next level.

SQL 79

SQL Kafka Aggregated Data Architecture 79

databricks

FEBRUARY 7, 2023

Today, we're excited to announce that Databricks has expanded Brickbuilder Solutions by collaborating with key partners in Europe, the Middle East, and Africa.

83

83

Advertisement

“Reimagined: Building Products with Generative AI” is an extensive guide for integrating generative AI into product strategy and careers featuring over 150 real-world examples, 30 case studies, and 20+ frameworks, and endorsed by over 20 leading AI and product executives, inventors, entrepreneurs, and researchers.

Building