The Pragmatic Engineer

OCTOBER 20, 2022

👋 Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. We cover one out of five topics in today’s subscriber-only The Scoop issue. To get this newsletter every week, subscribe. Pollen, the events festival tech startup, went bankrupt in August after raising more than $200M in venture funding. In an exclusive investigative article , I covered the events and details leading up this bankruptcy.

Banking 130

Banking Recruitment Consulting Media 130

Simon Späti

OCTOBER 19, 2022

Will Rust kill Python for Data Engineers? If you only came here to know this, my answer is no. Betteridge’s Law strikes again! But then again, you have to ask: was Python made for Data Engineering in the first place? Rust may not replace Python outright, but it has consumed more and more of JavaScript tooling and there are increasingly many projects trying to do the same with Python/Data Engineering.

Data Engineering 130

Data Engineering Data Engineer Engineering Programming Language 130

Jesse Anderson

OCTOBER 19, 2022

I have a calendar reminder that tells me when I founded Big Data Institute. It just told me I founded the company eight years ago. The reminder is called “Independent Anniversary.” It’s the day I split off and executed my vision for an independent, big data consulting company. Independence has all sorts of manifestations. For you, it’s an independent look at technology and vendors from someone who’s worked at a vendor (Cloudera) and worked in distributed systems for even longer.

Consulting 130

Consulting Big Data Technology Designing 130

KDnuggets

OCTOBER 19, 2022

This post is a summary of 2 distinct frameworks for approaching machine learning tasks, followed by a distilled third. Do they differ considerably (or at all) from each other, or from other such processes available?

Machine Learning 160

Machine Learning Process 160

Speaker: Timothy Chan, PhD., Head of Data Science

Are you ready to move beyond the basics and take a deep dive into the cutting-edge techniques that are reshaping the landscape of experimentation? 🌐 From Sequential Testing to Multi-Armed Bandits, Switchback Experiments to Stratified Sampling, Timothy Chan, Data Science Lead, is here to unravel the mysteries of these powerful methodologies that are revolutionizing how we approach testing.

Data Science

Data Engineering Podcast

OCTOBER 16, 2022

Summary The "data lakehouse" architecture balances the scalability and flexibility of data lakes with the ease of use and transaction support of data warehouses. Dremio is one of the companies leading the development of products and services that support the open lakehouse. In this episode Jason Hughes explains what it means for a lakehouse to be "open" and describes the different components that the Dremio team build and contribute to.

Data Lake 100

Data Lake Food MongoDB Scala 100

Simon Späti

OCTOBER 19, 2022

Will Rust kill Python for Data Engineers? If you only came here to know this, my answer is no. Betteridge’s Law strikes again! But then again, you have to ask: was Python made for Data Engineering in the first place? Rust may not replace Python outright, but it has consumed more and more of JavaScript tooling and there are increasingly many projects trying to do the same with Python/Data Engineering.

Data Engineering 130

Data Engineering Data Engineer Engineering Programming Language 130

KDnuggets

OCTOBER 17, 2022

Sparse features can cause problems like overfitting and suboptimal results in learning models, and understanding why this happens is crucial when developing models. Multiple methods, including dimensionality reduction, are available to overcome issues due to sparse features.

Machine Learning 157

Machine Learning 157

Data Engineering Podcast

OCTOBER 16, 2022

Summary Logistics and supply chains are under increased stress and scrutiny in recent years. In order to stay ahead of customer demands, businesses need to be able to react quickly and intelligently to changes, which requires fast and accurate insights into their operations. Pathway is a streaming database engine that embeds artificial intelligence into the storage, with functionality designed to support the spatiotemporal data that is crucial for shipping and logistics.

Database 100

Database Metadata MongoDB MySQL 100

Rockset

OCTOBER 20, 2022

Introduction Making use of real-time data for analytics is a deeply collaborative project. We’ve helped data engineers, data architects, engineering leaders, ML teams, and product managers connect the dots between various systems to deliver on Rockset’s promise of fast queries on fresh data. Not only are we collaborating with customers on analytics projects, we use our own product daily and collaborate across teams internally.

SQL 52

SQL Data Architect Datasets Python 52

Cloudera

OCTOBER 19, 2022

The telecommunications industry continues to develop hybrid data architectures to support data workload virtualization and cloud migration. However, while the promise of the cloud remains essential — not just for data workloads but also for network virtualisation and B2B offerings — the sheer volume and scale of data in the industry require careful management of the “journey to the cloud.”.

Cloud 78

Cloud Telecommunication Data Governance Government 78

Speaker: Anne Steiner and David Laribee

As a concept, Developer Experience (DX) has gained significant attention in the tech industry. It emphasizes engineers’ efficiency and satisfaction during the product development process. As product managers, we need to understand how a good DX can contribute not only to the well-being of our development teams but also to the broader objectives of product success and customer satisfaction.

Engineering

KDnuggets

OCTOBER 17, 2022

Outshine others and increase your odds of getting hired by maintaining a data science portfolio with projects, resumes, blogs, and reports.

Portfolio 159

Portfolio Data Science Building Data 159

Teradata

OCTOBER 21, 2022

The complexity of modern vehicles means that spotting root-causes that prevent them from working is difficult. Mechanics, operators & OEMs must step into a new era of digital data-based diagnostics.

Data 52

Data 52

Rockset

OCTOBER 21, 2022

Real-time customer 360 applications are essential in allowing departments within a company to have reliable and consistent data on how a customer has engaged with the product and services. Ideally, when someone from a department has engaged with a customer, you want up-to-date information so the customer doesn’t get frustrated and repeat the same information multiple times to different people.

Kafka 52

Kafka Building SQL Database 52

Cloudera

OCTOBER 20, 2022

Information technology has been at the heart of governments around the world, enabling them to deliver vital citizen services, such as healthcare, transportation, employment, and national security. All of these functions rest on technology and share a valuable commodity: data. . Data is produced and consumed in ever-increasing amounts and therefore must be protected.

Big Data 75

Big Data Transportation Healthcare Government 75

Speaker: Aarushi Kansal, AI Leader & Author and Tony Karrer, Founder & CTO at Aggregage

Software leaders who are building applications based on Large Language Models (LLMs) often find it a challenge to achieve reliability. It’s no surprise given the non-deterministic nature of LLMs. To effectively create reliable LLM-based (often with RAG) applications, extensive testing and evaluation processes are crucial. This often ends up involving meticulous adjustments to prompts.

Building

KDnuggets

OCTOBER 21, 2022

Check out this article on crafting a data science portfolio that will get you that job. And learn 4 resume mistakes to avoid at any cost.

Portfolio 150

Portfolio Data Science Data 150

U-Next

OCTOBER 20, 2022

Introduction . The primary goal of data collection is to gather high-quality information that aims to provide responses to all of the open-ended questions. Businesses and management can obtain high-quality information by collecting data that is necessary for making educated decisions. . It is necessary to gather data to draw conclusions and decide what is factual to increase the quality of the information. .

Data Collection 52

Data Collection Big Data Data Medical 52

Trio

OCTOBER 19, 2022

React enables much of the modern web you’re familiar with: fluid, responsive, and animation-rich websites. It’s no wonder that React.js is the most used JavsScript framework for web development, according to the 2021 State of JavaScript survey.

Engineering 52

Engineering 52

know.bi

OCTOBER 19, 2022

The Apache Hop team just released version 2.1.0. This new release is the result of four and a half months of work on over 200 tickets and comes packed with new functionality, bug fixes and improvements.

Data Engineering 52

Data Engineering Data Engineer Engineering Data 52

Advertisement

Are you trying to decide which entity resolution capabilities you need? It can be confusing to determine which features are most important for your project. And sometimes key features are overlooked. Get the Entity Resolution Evaluation Checklist to make sure you’ve thought of everything to make your project a success! The list was created by Senzing’s team of leading entity resolution experts, based on their real-world experience.

Project

KDnuggets

OCTOBER 18, 2022

Calculus is one of the foundational pillars of understanding the mathematics behind machine learning algorithms. The post shares five free courses to help you master calculus and learn its real-world applications.

Algorithm 112

Algorithm Machine Learning IT Data Science 112

AltexSoft

OCTOBER 18, 2022

Everyday the global healthcare system generates tons of medical data that — at least, theoretically — could be used for machine learning purposes. Regardless of industry, data is considered a valuable resource that helps companies outperform their rivals, and healthcare is not an exception. In this post, we’ll briefly discuss challenges you face when working with medical data and make an overview of publucly available healthcare datasets, along with practical tasks they help solve.

Medical 52

Medical Datasets Machine Learning Hospitality 52

Lyft Engineering

OCTOBER 17, 2022

Feature flags are everywhere in modern software development: They’re a great tool for running A/B experiments, slowly rolling out changes to users, and even turning off problematic codepaths during incidents. When an engineer implements a new feature, it’s practically second-nature to gate it behind a feature flag. While this practice is largely beneficial for the most part, incidents are occasionally caused when a feature flag enables a buggy codepath and causes a crash or an otherwise degraded

Engineering 52

Engineering Utilities Building Accessible 52

Confluent

OCTOBER 20, 2022

With ksqlDB 0.28.2 comes an easier getting started experience with auto-topic import,, new query options, and more support for data pipelines.

Cloud 52

Cloud Data Pipeline Data Process 52

Speaker: Margaret-Ann Seger, Head of Product, Statsig

Experimentation is often seen as an aspirational practice, especially at smaller, fast-moving companies who are strapped for time and resources. So, how can you get your team making decisions in a more data-driven way while continuing to remain lean and maintaining ship velocity? In this webinar, Margaret-Ann Seger, Head of Product at Statsig, will teach you how to build an experimentation culture from the ground-up, graduating from just getting started with data-driven development to operating

Building

KDnuggets

OCTOBER 20, 2022

Free Python for Data Science Course • 7 Machine Learning Portfolio Projects to Boost the Resume • Free Algorithms in Python Course • How to Select Rows and Columns in Pandas • 5 Data Science Skills That Pay & 5 That Don't • Everything You’ve Ever Wanted to Know About Machine Learning • Free SQL and Database Course • 7 Data Analytics Interview Questions & Answers.

Data Science 110

Data Science Python Portfolio Machine Learning 110

U-Next

OCTOBER 21, 2022

Introduction . Customer behavior is very difficult to interpret. When you think you have finally understood them completely, some new factor jumps in and proves you wrong. Here is where customer journey mapping comes into the picture. Consider the time when the customer first interacts with your business at the beginning of the journey and when the customers make a purchase from your business at the end of the journey.

IT 40

IT Designing Process Programming 40

Propel Data

OCTOBER 18, 2022

Learn how to move data from Amazon RDS into Snowflake so that it can be used for analytics.

Database 52

Database Data Analytics AWS Data 52

Lyft Engineering

OCTOBER 18, 2022

When the Roe v. Wade decision was announced, a lot of us felt disheartened.

Engineering 52

Engineering 52

Speaker: David Bard, Principal at VP Product Coaching

In the fast-paced world of digital innovation, success is often accompanied by a multitude of challenges - like the pitfalls lurking at every turn, threatening to derail the most promising projects. But fret not, this webinar is your key to effective product development! Join us for an enlightening session to empower you to lead your team to greater heights.

Certification

KDnuggets

OCTOBER 17, 2022

It is called Adaptive Boosting due to the fact that the weights are re-assigned to each instance, with higher weights being assigned to instances that are not correctly classified - therefore it ‘adapts’.

IT 110

IT Machine Learning 110

U-Next

OCTOBER 20, 2022

Introduction . The success of the company is largely dependent on the sales department of any organization. The one-of-a-kind and crucial function of sales is to fill the gap between potential customers’ wants and the items or services that the company provides. Sales greatly impact how customers become loyal and how much faith they have in brands.

Management 40

Management Retail Utilities Media 40

Confluent

OCTOBER 19, 2022

Designing Events and Event Streams: Introduction and Best Practices.

Designing 52

Designing Kafka 52

dbt Developer Hub

OCTOBER 17, 2022

For years working in data and analytics engineering roles, I treasured the daily camaraderie sharing a small office space with talented folks using a range of tools - from analysts using SQL and Excel to data scientists working in Python. I always sensed that there was so much we could work on in collaboration with each other - but siloed data and tooling made this much more difficult.

SQL 52

SQL Python Programming Language Data Analysis 52

Advertisement

“Reimagined: Building Products with Generative AI” is an extensive guide for integrating generative AI into product strategy and careers featuring over 150 real-world examples, 30 case studies, and 20+ frameworks, and endorsed by over 20 leading AI and product executives, inventors, entrepreneurs, and researchers.

Building