Confessions of a Data Guy

JANUARY 19, 2023

I remember those days, oh so long ago, it seems like another lifetime. I haven’t used Pandas in many a year, decades, or whatever. We’ve all been there, done that. Pandas I mean. I would dare say it’s a rite of passage for most data folk. For those using Python, it’s probably one of the […] The post Replacing Pandas with Polars.

Python 361

Python Data Big Data Data Engineering 361

Analytics Vidhya

JANUARY 31, 2023

Introduction Big data is revolutionizing the healthcare industry and changing how we think about patient care. In this case, big data refers to the vast amounts of data generated by healthcare systems and patients, including electronic health records, claims data, and patient-generated data. With the ability to collect, manage, and analyze vast amounts of data, […] The post The Impact of Big Data on Healthcare Decision Making appeared first on Analytics Vidhya.

Big Data 252

Big Data Healthcare Data Electronics 252

Seattle Data Guy

JANUARY 17, 2023

With all the recent data events I have put together I inevitably run into new data engineers who are either finishing up college or looking to transition into a data engineer or data scientist position. In fact I have talked to several newly graduated engineers who are struggling to find work. A few told me… Read more The post How To Hire Junior Data Engineers appeared first on Seattle Data Guy.

Data Engineering 223

Data Engineering Data Engineer Engineering Data 223

Simon Späti

JANUARY 3, 2023

I used VS Code, Sublime, Notepad++, TextMate, and others, but the shortcut with cmd(+shift)+end, jumping with option+arrow-keys from word to word, needed to be faster at some point. I was hitting my limits. Everything I was doing I did decently fast, but I didn’t get any faster. Vim is the only editor you get faster with time. Vim is based solely on shortcuts.

Data Engineering 195

Data Engineering Data Engineer Engineering Data 195

Speaker: Dr. Greg Loughnane and Chris Alexiuk

Technology professionals developing generative AI applications are finding that there are big leaps from POCs and MVPs to production-ready applications. They're often developing using prompting, Retrieval Augmented Generation (RAG), and fine-tuning (up to and including Reinforcement Learning with Human Feedback (RLHF)), typically in that order. However, during development – and even more so once deployed to production – best practices for operating and improving generative AI applications are le

Process

The Pragmatic Engineer

JANUARY 12, 2023

👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. We cover one and a half out of eight topics in today’s subscriber-only issue, Inside Pollen's Transparent Compensation Data. If you’re not yet a subscriber, you also missed this week’s deep-dive on Becoming a Fractional CTO. To get this newsletter every week, subscribe here.

Software Engineer 187

Software Engineer Software Engineering Engineering Data Engineer 187

Confessions of a Data Guy

JANUARY 12, 2023

Anyone who’s been roaming around the forest of Data Engineering has probably run into many of the newish tools that have been growing rapidly around the concepts of Data Warehouses, Data Lakes, and Lake Houses … the merging of the old relational database functionality with TB and PB level cloud-based file storage systems. Tools like […] The post Simplify Delta Lake Complexity with mack. appeared first on Confessions of a Data Guy.

Data Lake 162

Data Lake Data Warehouse Relational Database Data Engineering 162

Analytics Vidhya

JANUARY 30, 2023

Introduction Azure Functions is a serverless computing service provided by Azure that provides users a platform to write code without having to provision or manage infrastructure in response to a variety of events. Whether we are analyzing IoT data streams, managing scheduled events, processing document uploads, responding to database changes, etc. Azure functions allow developers […] The post How to Develop Serverless Code Using Azure Functions?

Coding 239

Coding Database Data Management 239

Seattle Data Guy

JANUARY 17, 2023

2022 is coming to an end. What is the state of data infra? Are Snowflake and Databricks still fighting over total cost of ownership? Is everyone switching to DuckDB? Are data engineers all learning Rust? Let’s try to answer these questions. Our team is putting together an all day event focused on helping answer some… Read more The post What Is The State Of Data Engineering And Infrastructure In 2023 appeared first on Seattle Data Guy.

Data Engineering 211

Data Engineering Data Engineer Engineering Data 211

Jesse Anderson

JANUARY 9, 2023

If you haven’t heard, Confluent announced they’re buying Immerok. This purchase represents a significant shift in strategy for Confluent. I started a Twitter thread with some of my initial thoughts, but I want to write a post giving more analysis and opinions. In short, I still echo the sentiment from my original tweet “This was always the way it should have been.

Kafka 147

Kafka SQL Coding Data 147

Advertisement

Are you thinking of adding enhanced data matching and relationship detection to your product or service? Do you need to know more about what to look for when assessing your options? The Senzing Entity Resolution Buyer’s Guide gives you step-by-step details about everything you should consider when evaluating entity resolution technologies. You’ll learn about use cases, technology and deployment options, top ten evaluation criteria and more.

Technology

The Pragmatic Engineer

JANUARY 26, 2023

Comments

330

330

Data Engineering Podcast

JANUARY 29, 2023

Summary Business intelligence has gone through many generational shifts, but each generation has largely maintained the same workflow. Data analysts create reports that are used by the business to understand and direct the business, but the process is very labor and time intensive. The team at Omni have taken a new approach by automatically building models based on the queries that are executed.

Business Intelligence 130

Business Intelligence PostgreSQL SQL Building 130

Confessions of a Data Guy

JANUARY 13, 2023

Rust has been on my mind a lot lately, probably because of Data Engineering boredom, watching Spark clusters chug along like some medieval farm worker endlessly trudging through the muck and mire of life. Maybe Rust has breathed some life back into my stagnant soul, reminding me there is a big world out there, […] The post Using Rust to write a Data Pipeline.

Data Pipeline 147

Data Pipeline Data Data Engineer Data Engineering 147

Analytics Vidhya

JANUARY 30, 2023

Introduction We are all aware of the Internet’s explosive expansion as a primary source of information and a platform for opinion expression. It has now become essential to gather and analyze the ever-expanding data that follows. While in the past, manual analysis of data has been possible and even served us well, the same cannot […] The post Top 10 Applications of Sentiment Analysis in Business appeared first on Analytics Vidhya.

Data 236

Data Data Engineering Data Engineer Python 236

Christophe Blefari

JANUARY 27, 2023

My view from the train window ( credits ) Dear Data News readers it's a joy every week to write this newsletter, we are slowly approaching the second birthday of this newsletter. In order to celebrate this together I'd love to receive your stories about data —can be short or long, anonymous or not. This is an open box, just write me with what you have on the mind and I'll bundle an edition with it.

Data 130

Data Data Pipeline SQL Coding 130

Seattle Data Guy

JANUARY 24, 2023

Modern data stack consultant plays an important role in companies looking to become data-driven. They help companies design and deploy centralized data sets that are easy to use and reliable. They do so by using cloud based solutions that help automate data pipelines and processes with less code than in the past. But in order… Read more The post Do You Need A Modern Data Stack Consultant appeared first on Seattle Data Guy.

Consulting 130

Consulting Data Data Pipeline Cloud 130

The Pragmatic Engineer

JANUARY 19, 2023

👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. We cover one out of five topics in today’s subscriber-only The Scoop issue. To get the full issues, twice a week: subscribe here. Update on 20 January: less than a day after publishing this article, Google announced historic layoffs that will impact ~12,000 positions.

Cloud 142

Cloud AWS Banking Data 142

KDnuggets

JANUARY 20, 2023

Is ChatGPT useful for Python programmers, specifically those of us who use Python for data processing, data cleaning, and building machine learning models? Let's give it a try and find out.

Python 145

Python Programming Machine Learning Data Process 145

Confluent

JANUARY 6, 2023

Introducing fully managed Apache Kafka® + Flink for the most robust, cloud-native data streaming platform with stream processing, integration, and streaming analytics in one.

Kafka 145

Kafka Cloud Data Process 145

Analytics Vidhya

JANUARY 31, 2023

Introduction In today’s world, machine learning and artificial intelligence are widely used in almost every sector to improve performance and results. But are they still useful without the data? The answer is No. The machine learning algorithms heavily rely on data that we feed to them. The quality of data we feed to the algorithms […] The post Practicing Machine Learning with Imbalanced Dataset appeared first on Analytics Vidhya.

Machine Learning 235

Machine Learning Datasets Algorithm Data 235

Engineering at Meta

JANUARY 27, 2023

Managing network solutions amidst a growing scale inherently brings challenges around performance, deployment, and operational complexities. At Meta, we’ve found that these challenges broadly fall into three themes: 1.) Data center networking: Over the past decade, on the physical front, we have seen a rise in vendor-specific hardware that comes with heterogeneous feature and architecture sets (e.g., non-blocking architecture).

Engineering 117

Engineering Software Engineer Software Engineering Architecture 117

Netflix Tech

JANUARY 25, 2023

Scalable Annotation Service — Marken by Varun Sekhri , Meenakshi Jindal Introduction At Netflix, we have hundreds of micro services each with its own data models or entities. For example, we have a service that stores a movie entity’s metadata or a service that stores metadata about images. All of these services at a later point want to annotate their objects or entities.

Algorithm 116

Algorithm Metadata Data Ingestion Data Pipeline 116

DoorDash Engineering

JANUARY 31, 2023

App startup time is a critical metric for users, as it’s their first interaction with the app, and even minor improvements can have significant benefits for the user experience. First impressions are a big driver in consumer conversion, and startup times often indicate the app’s overall quality. Furthermore, other companies found that an increase in latency equals a decrease in sales.

Architecture 113

Architecture Engineering Process Coding 113

KDnuggets

JANUARY 9, 2023

Data-driven organizations require complex collaboration between data teams and business stakeholders. Here are 4 proactive tips for reducing information asymmetries and achieving better collaboration.

Data 145

Data IT Data Engineering Data Engineer 145

Confluent

JANUARY 10, 2023

CDC is a software design pattern that identifies and captures changes made to data in a database. Learn how CDC works, the best solutions, and how to get started with various implementations.

Data 116

Data Database Designing Process 116

Analytics Vidhya

JANUARY 31, 2023

Introduction YARN stands for Yet Another Resource Negotiator. It is a powerful resource management system for a horizontal server environment. It is designed to be more flexible and generic than the original Hadoop MapReduce system, making it an attractive choice for companies looking to implement Hadoop. It allows companies to process data types and run […] The post YARN for Large Scale Computing: Beginner’s Edition appeared first on Analytics Vidhya.

Hadoop 230

Hadoop Systems Data Designing 230

databricks

JANUARY 26, 2023

This is a collaborative post from Databricks and wisecube.ai. We thank Vishnu Vettrivel, Founder, and Alex Thomas, Principal Data Scientist, for their contributions.

Data Lake 112

Data Lake Data Building 112

Snowflake

JANUARY 19, 2023

The Chief Data Officer is arguably one of the most important roles at a company, particularly those that aspire to be data-driven. CDO appointments and the elevation of data leaders have accelerated in recent years, and the role has morphed as perceptions of data have evolved. Responsibilities span strategy and execution, people and processes, and the technology needed to deliver on the promise of data.

Data 107

Data Data Architecture Data Analytics Data Governance 107

Uber Engineering

JANUARY 16, 2023

In this blog, we share how we improved the daily edit-build-run developer experience using DevPods, Uber’s remote development environment. We cover the challenges, pain points, our architecture, and lastly the future of remote development at Uber.

Architecture 105

Architecture Building Engineering 105

KDnuggets

JANUARY 3, 2023

Matplotlib is the most famous and commonly used plotting library in Python. It allows you to create clear and interactive visualizations that make your data easier to understand and your results more concrete.

Python 144

Python Data Data Science IT 144

Netflix Tech

JANUARY 30, 2023

By Grace Tang , Aneesh Vartakavi , Julija Bagdonaite , Cristina Segalin , and Vi Iyengar When members are shown a title on Netflix, the displayed artwork, trailers, and synopses are personalized. That means members see the assets that are most likely to help them make an informed choice. These assets are a critical source of information for the member to make a decision to watch, or not watch, a title.

Algorithm 104

Algorithm Machine Learning Data Datasets 104

Precisely

JANUARY 27, 2023

In today’s hypercompetitive business environment, companies must deliver a standout experience for their target audience. Companies that excel at customer experience (CX) are better at building brand loyalty, increasing total customer lifetime value, and turning occasional customers into brand evangelists. This compelling drive for outstanding CX coincides with an intensive shift toward digitization, personalization, and omnichannel alignment.

High Quality Data 104

High Quality Data Data Governance Government Data 104

Engineering at Meta

JANUARY 31, 2023

We’ve made architecture changes to Meta’s event driven asynchronous computing platform that have enabled easy integration with multiple event-sources. We’re sharing our learnings from handling various workloads and how to tackle trade offs made with certain design choices in building the platform. Asynchronous computing is a paradigm where the user does not expect a workload to be executed immediately; instead, it gets scheduled for execution sometime in the near future without blocking the la

Architecture 102

Architecture Transportation Data Warehouse Data 102