Start Data Engineering

FEBRUARY 22, 2022

1. Introduction 2. CI 3. Sample project: Data testing with Github Actions 3.1. Prerequisites 3.2. Project overview 3.3. Automating data tests with Github Actions 4. Conclusion 5. Further reading 1. Introduction Automated testing is crucial for ensuring that your code is bug-free and avoiding regressions. If you are wondering How can data tests be integrated into a CI (Continuous Integration) pipeline?

Data 130

Data Coding Project Systems 130

KDnuggets

FEBRUARY 22, 2022

Machine Learning as a technology, ensures that our current gadgets and their software get smarter by the day. Here are the algorithms that you ought to know about to understand Machine Learning’s varied and extensive functionalities and their effectiveness.

Machine Learning 157

Machine Learning Algorithm Technology 157

Data Engineering Podcast

FEBRUARY 20, 2022

Summary The life sciences as an industry has seen incredible growth in scale and sophistication, along with the advances in data technology that make it possible to analyze massive amounts of genomic information. In this episode Guy Yachdav, director of software engineering for ImmunAI, shares the complexities that are inherent to managing data workflows for bioinformatics.

Systems 100

Systems Software Engineer Software Engineering Data Warehouse 100

Confluent

FEBRUARY 24, 2022

A few years ago I helped build an event-driven system for gym bookings. The pitch was that we were building a better experience for both the gym members booking different […].

Systems 122

Systems Building Data 122

Speaker: Timothy Chan, PhD., Head of Data Science

Are you ready to move beyond the basics and take a deep dive into the cutting-edge techniques that are reshaping the landscape of experimentation? 🌐 From Sequential Testing to Multi-Armed Bandits, Switchback Experiments to Stratified Sampling, Timothy Chan, Data Science Lead, is here to unravel the mysteries of these powerful methodologies that are revolutionizing how we approach testing.

Data Science

Cloudera

FEBRUARY 23, 2022

The telecommunications industry has been doing well since the pandemic started (not that many would notice). Revenues have remained relatively stable, while consumption has gone up, as virtual engagement has become the primary mode of operations for many businesses (and families!) In the mean-time, digital transformation has been accelerating both as a means to respond to the pandemic, and as a mechanism to drive costs down further, allowing for margin growth.

Telecommunication 91

Telecommunication Cloud Media Government 91

KDnuggets

FEBRUARY 21, 2022

A collection of cheat sheets that will help you prepare for a technical interview on Data Structures & Algorithms, Machine learning, Deep Learning, Natural Language Processing, Data Engineering, Web Frameworks.

Data Science 148

Data Science Deep Learning Algorithm Machine Learning 148

Confluent

FEBRUARY 23, 2022

In many ways, Storyblocks’ technical journey has mirrored that of most other startups and disruptors: Start small and as simple as possible (i.e., with a PHP monolith) Watch the company […].

Cloud 52

Cloud 52

Cloudera

FEBRUARY 24, 2022

In the latest installment of the EMEA Influential Women in Data webinar series, we welcomed Shirley Collie, Chief Health Analytics Actuary at Discovery Health to discuss everything from how the pandemic has impacted working, to the opportunities within data, and the importance of intentionality. A data-driven organization. Shirley knows better than most about the impact that COVID 19 has had on the world.

Healthcare 82

Healthcare Algorithm IT Management 82

KDnuggets

FEBRUARY 24, 2022

Object-relational mapping, or ORM, is a technique that allows you to interact with databases using the object-oriented paradigm of the programming language of your choosing. How is that different from structured query language, though, and when do you use them?

SQL 147

SQL Programming Language Database Programming 147

U-Next

FEBRUARY 21, 2022

Artificial Intelligence (AI) is not just making our lives convenient. It is empowering us with information and insights that have the potential to change the world for the better. With its application across diverse industries, market segments and real-world concerns, the role of AI is becoming increasingly inevitable by the day. This is to the extent that we see AI as a savior to some of the most plaguing concerns of humankind.

IT 52

IT Education Machine Learning 52

Speaker: Anne Steiner and David Laribee

As a concept, Developer Experience (DX) has gained significant attention in the tech industry. It emphasizes engineers’ efficiency and satisfaction during the product development process. As product managers, we need to understand how a good DX can contribute not only to the well-being of our development teams but also to the broader objectives of product success and customer satisfaction.

Engineering

Preset

FEBRUARY 21, 2022

CrateDB is a distributed SQL database that excels at IoT and Time Series data workflows. In this post, we'll showcase how CrateDB and Superset can be used together.

Data Workflow 52

Data Workflow SQL Database Data 52

Cloudera

FEBRUARY 22, 2022

Over the past decade, the successful deployment of large scale data platforms at our customers has acted as a big data flywheel driving demand to bring in even more data, apply more sophisticated analytics, and on-board many new data practitioners from business analysts to data scientists. This unprecedented level of big data workloads hasn’t come without its fair share of challenges.

Metadata 105

Metadata Datasets BI SQL 105

KDnuggets

FEBRUARY 25, 2022

Learn data analytics by taking the best YouTube courses. These courses will cover data analysis with Python, R, SQL, PowerBI, Tableau, Excel, and SPSS.

Data Analytics 144

Data Analytics Data Analysis SQL Python 144

U-Next

FEBRUARY 21, 2022

Cryptocurrencies are game-changing. NFTs are revolutionary. Blockchain is super airtight. Agreed. However, amidst all the news on people becoming millionaires through NFTs and cryptocurrencies rewriting conventions, there are also news that are quite alarming – crypto scams. Yes, with the world gradually adapting blockchain applications and concepts and several countries revisiting their policies on cryptocurrencies, this comes at the wrong time.

Media 52

Media 52

Speaker: Aarushi Kansal, AI Leader & Author and Tony Karrer, Founder & CTO at Aggregage

Software leaders who are building applications based on Large Language Models (LLMs) often find it a challenge to achieve reliability. It’s no surprise given the non-deterministic nature of LLMs. To effectively create reliable LLM-based (often with RAG) applications, extensive testing and evaluation processes are crucial. This often ends up involving meticulous adjustments to prompts.

Building

Preset

FEBRUARY 23, 2022

This article walks you through a potential approach to monitor your Superset usage directly within Superset leveraging the internal metadata database.

Metadata 52

Metadata Database 52

Marc Lamberti

FEBRUARY 21, 2022

Airflow dynamic DAGs can save you a ton of time. As you know, Apache Airflow is written in Python, and DAGs are created via Python scripts. That makes it very flexible and powerful (even complex sometimes). By leveraging Python, you can create DAGs dynamically based on variables, connections, a typical pattern, etc. This very nice way of generating DAGs comes at the price of higher complexity and subtle tricky things that you must know.

Python 130

Python Metadata Coding Process 130

KDnuggets

FEBRUARY 25, 2022

This blog post aims to describe the vanishing gradient problem and explain how use of the sigmoid function resulted in it.

IT 159

IT Machine Learning 159

ProjectPro

FEBRUARY 25, 2022

When the world was under lockdown and movement was restricted to an absolute emergency- millions were introduced to the world of online shopping. The convenience of online shopping helped e-commerce platforms record historic sales. While that happened, it is no surprise that the rate of online financial fraud also increased incredibly. Online fraud cases using credit and debit cards saw a historic upsurge of 225 percent during the COVID-19 pandemic in 2020 as compared to 2019.

Machine Learning 52

Machine Learning Project Algorithm Datasets 52

Advertisement

Are you trying to decide which entity resolution capabilities you need? It can be confusing to determine which features are most important for your project. And sometimes key features are overlooked. Get the Entity Resolution Evaluation Checklist to make sure you’ve thought of everything to make your project a success! The list was created by Senzing’s team of leading entity resolution experts, based on their real-world experience.

Project

Data Engineering Podcast

FEBRUARY 20, 2022

Summary Python has grown to be one of the top languages used for all aspects of data, from collection and cleaning, to analysis and machine learning. Along with that growth has come an explosion of tools and engines that help power these workflows, which introduces a great deal of complexity when scaling from single machines and exploratory development to massively parallel distributed computation.

Python 100

Python Data Process IT Building 100

Rockset

FEBRUARY 24, 2022

Event-based architectures have been gaining popularity for some time. With increased adoption has come a flood of options for aggregating and analyzing events. Which databases are optimized for ingesting streaming events and analyzing them in real time? The answer is complex, nuanced and heavily dependent on the precise problem being solved. This post is intended to help anyone seeking to make a selection from a difficult to understand landscape.

AWS 52

AWS Amazon Web Services Kafka Business Intelligence 52

KDnuggets

FEBRUARY 23, 2022

This article outlines some of the most common design patterns encountered when creating successful Machine Learning solutions.

Machine Learning 154

Machine Learning Designing 154

ProjectPro

FEBRUARY 22, 2022

This article will teach you exciting SQL project ideas to develop data analysis skills. You will explore challenging problems that you can quickly solve with this simple query language. It doesn’t matter if you are a beginner or a professional at using SQL; our list of SQL database projects has one for you. Data, data, everywhere! Where’s the way to manage it?

Data Analysis 52

Data Analysis SQL Project Banking 52

Speaker: Margaret-Ann Seger, Head of Product, Statsig

Experimentation is often seen as an aspirational practice, especially at smaller, fast-moving companies who are strapped for time and resources. So, how can you get your team making decisions in a more data-driven way while continuing to remain lean and maintaining ship velocity? In this webinar, Margaret-Ann Seger, Head of Product at Statsig, will teach you how to build an experimentation culture from the ground-up, graduating from just getting started with data-driven development to operating

Building

Hepta Analytics

FEBRUARY 24, 2022

Week 3 was about data warehousing, working on the data that was ingested in the week 2. We will take the already ingested data and create an external table from it and optimize the performance of queries through partitioning and clustering. Then automate the whole process using airflow. There are two systems types when dealing with data: Online Transaction Processing (OLTP) and Online Analytical Processing (OLAP).

Data Warehouse 52

Data Warehouse Data Engineering Data Engineer Engineering 52

AltexSoft

FEBRUARY 21, 2022

There has been a lot of buzz around data science, machine learning (ML), and artificial intelligence (AI) lately. As you may already know, to train a machine learning model, you need data. Lots of data, to be more precise. Lots of quality data, to be even more precise. To save you time, watch our 14-minute video on how data is prepared for machine learning.

Machine Learning 52

Machine Learning Healthcare Algorithm Medical 52

KDnuggets

FEBRUARY 24, 2022

Machine Learning with PyTorch and Scikit-learn is the PyTorch book from the widely acclaimed and bestselling Python Machine Learning series, fully updated and expanded to cover PyTorch, transformers, graph neural networks, and best practices.

Machine Learning 130

Machine Learning Python 130

ProjectPro

FEBRUARY 24, 2022

Working with audio data has been a relatively less widespread and explored problem in machine learning. In most cases, benchmarks for the latest seminal work in deep learning are measured on text and image data performances. Moreover, the most significant advances in deep learning are found in models that work with text and images. Amidst this, speech and audio, an equally important type of data, often gets overlooked.

Deep Learning 52

Deep Learning Project Datasets Machine Learning 52

Speaker: David Bard, Principal at VP Product Coaching

In the fast-paced world of digital innovation, success is often accompanied by a multitude of challenges - like the pitfalls lurking at every turn, threatening to derail the most promising projects. But fret not, this webinar is your key to effective product development! Join us for an enlightening session to empower you to lead your team to greater heights.

Certification

Pipeline Data Engineering

FEBRUARY 22, 2022

Data engineering salon. News and interesting reads about the world of data. We’ve only scratched the surface of the full potential for the data warehouse Mikkel Dengsøe, Head of Data Science, Operations & Financial Crime, Monzo Bank Why I think the data warehouse will become the control centre for modern companies Git, SQL, CLI Vicki Boykis, Machine Learning Engineer, Automattic I’ve narrowed it down to three basic tools.

Data Warehouse 52

Data Warehouse Banking Metadata Data Science 52

KDnuggets

FEBRUARY 23, 2022

The rush towards cloud storage means that the cloud has to offer a valuable proposition to businesses. Let’s explore why businesses regardless of their size should consider moving to the cloud.

Cloud Storage 112

Cloud Storage Cloud Data Science Data 112

KDnuggets

FEBRUARY 21, 2022

According to a Stack Overflow survey, 13.2% of the data scientists are looking for a new job, as they are not satisfied in their current role. So why is this happening? What are the challenges Data Scientists are facing?

Data 110

Data 110

KDnuggets

FEBRUARY 24, 2022

While a vast majority of research efforts today are preoccupied solely with ML models and algorithms, the data itself tends to be secondary and is treated as fixed. This claim is potentially detrimental.

Algorithm 108

Algorithm Data 108

Advertisement

“Reimagined: Building Products with Generative AI” is an extensive guide for integrating generative AI into product strategy and careers featuring over 150 real-world examples, 30 case studies, and 20+ frameworks, and endorsed by over 20 leading AI and product executives, inventors, entrepreneurs, and researchers.

Building