Start Data Engineering

FEBRUARY 22, 2022

1. Introduction 2. CI 3. Sample project: Data testing with Github Actions 3.1. Prerequisites 3.2. Project overview 3.3. Automating data tests with Github Actions 4. Conclusion 5. Further reading 1. Introduction Automated testing is crucial for ensuring that your code is bug-free and avoiding regressions. If you are wondering How can data tests be integrated into a CI (Continuous Integration) pipeline?

Data 130

Data Coding Project Systems 130

KDnuggets

FEBRUARY 14, 2022

Calculus is the key to fully understanding how neural networks function. Go beyond a surface understanding of this mathematics discipline with these free course materials from MIT.

Deep Learning 160

Deep Learning Machine Learning 160

Data Engineering Podcast

FEBRUARY 20, 2022

Summary The life sciences as an industry has seen incredible growth in scale and sophistication, along with the advances in data technology that make it possible to analyze massive amounts of genomic information. In this episode Guy Yachdav, director of software engineering for ImmunAI, shares the complexities that are inherent to managing data workflows for bioinformatics.

Systems 100

Systems Software Engineer Software Engineering Data Warehouse 100

Confluent

FEBRUARY 24, 2022

A few years ago I helped build an event-driven system for gym bookings. The pitch was that we were building a better experience for both the gym members booking different […].

Systems 122

Systems Building Data 122

Speaker: Timothy Chan, PhD., Head of Data Science

Are you ready to move beyond the basics and take a deep dive into the cutting-edge techniques that are reshaping the landscape of experimentation? 🌐 From Sequential Testing to Multi-Armed Bandits, Switchback Experiments to Stratified Sampling, Timothy Chan, Data Science Lead, is here to unravel the mysteries of these powerful methodologies that are revolutionizing how we approach testing.

Data Science

Cloudera

FEBRUARY 9, 2022

As we celebrate Black History Month, for this Employee Spotlight I sat down with Marque Blackman, co-lead of the Cloudera Black Employee Network (CBEN). We discussed his experience at Cloudera, his career transitions, and what he learned along the way. We also discussed his work with CBEN and his perspective on Black History Month. Meet Marque Blackman, Director of Global Workplace .

Entertainment 119

Entertainment Media Programming Designing 119

DataKitchen

FEBRUARY 8, 2022

The post New Data Horizons: Data Prep, Data Visualization, and Data Catalogs Are Ready for Prime Time first appeared on DataKitchen.

Data 98

Data 98

KDnuggets

FEBRUARY 17, 2022

There's no free lunch in machine learning. So, determining which algorithm to use depends on many factors from the type of problem at hand to the type of output you are looking for. This guide offers several considerations to review when exploring the right ML approach for your dataset.

Machine Learning 160

Machine Learning Algorithm Datasets 160

Data Engineering Podcast

FEBRUARY 13, 2022

Summary Collecting, integrating, and activating data are all challenging activities. When that data pertains to your customers it can become even more complex. To simplify the work of managing the full flow of your customer data and keep you in full control the team at Rudderstack created their eponymous open source platform that allows you to work with first and third party data, as well as build and manage reverse ETL workflows.

Building 100

Building Hadoop Data Pipeline Metadata 100

Confluent

FEBRUARY 18, 2022

As data flows in and out of your Confluent Cloud clusters, it’s imperative to monitor their behavior. Bring Your Own Monitoring (BYOM) means you can configure an application performance monitoring […].

Cloud 117

Cloud Data 117

Cloudera

FEBRUARY 17, 2022

CDP Private Cloud Base is an on-premises version of Cloudera Data Platform (CDP). This new product combines the best of Cloudera Enterprise Data Hub and Hortonworks Data Platform Enterprise along with new features and enhancements across the stack. This unified distribution is a scalable and customizable platform where you can securely run many types of workloads.

Cloud 98

Cloud Data Professional Services Metadata 98

Speaker: Anne Steiner and David Laribee

As a concept, Developer Experience (DX) has gained significant attention in the tech industry. It emphasizes engineers’ efficiency and satisfaction during the product development process. As product managers, we need to understand how a good DX can contribute not only to the well-being of our development teams but also to the broader objectives of product success and customer satisfaction.

Engineering

ProjectPro

FEBRUARY 28, 2022

Facial Expression Recognition (FER) based technologies are an integral part of the emotion recognition market, which is anticipated to reach $56 billion by 2024—detecting Emotions? Using AI? Can we really do that? The answer is YES! One can easily build a facial emotion recognition project in Python. Continue reading to find the answer to how you can do that.

Coding 52

Coding Project Deep Learning Datasets 52

U-Next

FEBRUARY 21, 2022

Artificial Intelligence (AI) is not just making our lives convenient. It is empowering us with information and insights that have the potential to change the world for the better. With its application across diverse industries, market segments and real-world concerns, the role of AI is becoming increasingly inevitable by the day. This is to the extent that we see AI as a savior to some of the most plaguing concerns of humankind.

IT 52

IT Education Machine Learning 52

KDnuggets

FEBRUARY 11, 2022

Here are a few approaches that I have settled on for managing my own reusable Python code as a data scientist, presented from most to least general code use, and aimed at beginners.

Python 160

Python Coding Management Data 160

Data Engineering Podcast

FEBRUARY 6, 2022

Summary There are many dimensions to the work of protecting the privacy of users in our data. When you need to share a data set with other teams, departments, or businesses then it is of utmost importance that you eliminate or obfuscate personal information. In this episode Will Thompson explores the many ways that sensitive data can be leaked, re-identified, or otherwise be at risk, as well as the different strategies that can be employed to mitigate those attack vectors.

Data Pipeline 100

Data Pipeline Data Metadata Software Engineer 100

Speaker: Aarushi Kansal, AI Leader & Author and Tony Karrer, Founder & CTO at Aggregage

Software leaders who are building applications based on Large Language Models (LLMs) often find it a challenge to achieve reliability. It’s no surprise given the non-deterministic nature of LLMs. To effectively create reliable LLM-based (often with RAG) applications, extensive testing and evaluation processes are crucial. This often ends up involving meticulous adjustments to prompts.

Building

Confluent

FEBRUARY 3, 2022

A common challenge organizations face is how to extract, transform, and load (ETL) Salesforce data into a data warehouse, so that the business can use the data. Salesforce (SFDC) is […].

Data Warehouse 104

Data Warehouse Data Cloud 104

Cloudera

FEBRUARY 11, 2022

In recent years, Ethical AI has become an area of increased importance to organisations. Advances in the development and application of Machine Learning (ML) and Deep Learning (DL) algorithms, require greater care to ensure that the ethics embedded in previous rule-based systems are not lost. This has led to Ethical AI being an increasingly popular search term and the subject of many industry analyst reports and papers.

Machine Learning 94

Machine Learning Algorithm Data Science Government 94

ProjectPro

FEBRUARY 25, 2022

What is a Machine Learning Pipeline? A machine learning pipeline helps automate machine learning workflows by processing and integrating data sets into a model, which can then be evaluated and delivered. A well-built pipeline helps in the flexibility of the model implementation. A pipeline in machine learning is a technical infrastructure that allows an organization to organize and automate machine learning operations.

Machine Learning 52

Machine Learning Building Amazon Web Services AWS 52

U-Next

FEBRUARY 21, 2022

Cryptocurrencies are game-changing. NFTs are revolutionary. Blockchain is super airtight. Agreed. However, amidst all the news on people becoming millionaires through NFTs and cryptocurrencies rewriting conventions, there are also news that are quite alarming – crypto scams. Yes, with the world gradually adapting blockchain applications and concepts and several countries revisiting their policies on cryptocurrencies, this comes at the wrong time.

Media 52

Media 52

Advertisement

Are you trying to decide which entity resolution capabilities you need? It can be confusing to determine which features are most important for your project. And sometimes key features are overlooked. Get the Entity Resolution Evaluation Checklist to make sure you’ve thought of everything to make your project a success! The list was created by Senzing’s team of leading entity resolution experts, based on their real-world experience.

Project

KDnuggets

FEBRUARY 1, 2022

Are you trying to teach yourself machine learning from scratch, but aren’t sure where to start? I will attempt to condense all the resources I’ve used over the years into 7 steps that you can follow to teach yourself machine learning.

Machine Learning 159

Machine Learning Python 159

Pipeline Data Engineering

FEBRUARY 18, 2022

A bit more than a year ago I’ve compiled an annotated list of the best free courses and learning resources that could help anyone to become a data engineer on a shoestring. We’ve received an overwhelming amount of positive feedback on it, so after a full year of running the bootcamp I sat down again and collected an other bunch of resources we’ve bumped into during the cohorts.

Data Engineering 52

Data Engineering Data Engineer Engineering Python 52

Confluent

FEBRUARY 23, 2022

In many ways, Storyblocks’ technical journey has mirrored that of most other startups and disruptors: Start small and as simple as possible (i.e., with a PHP monolith) Watch the company […].

Cloud 52

Cloud 52

Cloudera

FEBRUARY 23, 2022

The telecommunications industry has been doing well since the pandemic started (not that many would notice). Revenues have remained relatively stable, while consumption has gone up, as virtual engagement has become the primary mode of operations for many businesses (and families!) In the mean-time, digital transformation has been accelerating both as a means to respond to the pandemic, and as a mechanism to drive costs down further, allowing for margin growth.

Telecommunication 92

Telecommunication Cloud Media Government 92

Advertisement

Outdated or absent analytics won’t cut it in today’s data-driven applications – not for your end users, your development team, or your business. That’s what drove the five companies in this e-book to change their approach to analytics. Download this e-book to learn about the unique problems each company faced and how they achieved huge returns beyond expectation by embedding analytics into applications.

Building

DataKitchen

FEBRUARY 18, 2022

DataOps is a discipline focused on the delivery of data faster, better, and cheaper to derive business value quickly. It closely follows the best practices of DevOps although the implementation of DataOps to data is nothing like DevOps to code. This paper will focus on providing a prescriptive approach in implementing a data pipeline using a DataOps discipline for data practitioners.

High Quality Data 52

High Quality Data Business Intelligence Data Pipeline Government 52

ProjectPro

FEBRUARY 17, 2022

Feature selection techniques are fundamental to predictive modeling tasks; one can not create predictive models without selecting the features correctly. What are these feature selection methods, and how are they used in building efficient predictive models? You will find out all the answers in this article. If you have ever baked a cake in your life or perhaps witnessed someone following a recipe to bake it, you must have noticed how crucial it is to precisely measure each ingredient's quantity

Machine Learning 52

Machine Learning Algorithm Datasets Banking 52

KDnuggets

FEBRUARY 22, 2022

Machine Learning as a technology, ensures that our current gadgets and their software get smarter by the day. Here are the algorithms that you ought to know about to understand Machine Learning’s varied and extensive functionalities and their effectiveness.

Machine Learning 157

Machine Learning Algorithm Technology 157

Pipeline Data Engineering

FEBRUARY 18, 2022

Data engineers have to wear many different hats at the same time: they are architects, designers, builders, maintainers, procurement and quality assurance — to just name a few. If you’d like to break into this profession, you need to prove that you can do all of the above, and more. One of the key assets you can use to do that is a data product that you’ve built with your own hands.

Building 52

Building Portfolio Kafka Machine Learning 52

Speaker: David Bard, Principal at VP Product Coaching

In the fast-paced world of digital innovation, success is often accompanied by a multitude of challenges - like the pitfalls lurking at every turn, threatening to derail the most promising projects. But fret not, this webinar is your key to effective product development! Join us for an enlightening session to empower you to lead your team to greater heights.

Certification

Hepta Analytics

FEBRUARY 14, 2022

DE Zoomcamp 2.2.1 – Introduction to Workflow Orchestration Following last weeks blog , we move to data ingestion. We already had a script that downloaded a csv file, processed the data and pushed the data to postgres database. This was used to test our setup. This week, we got to think about our data ingestion design. We looked at the following: How do we ingest – ETL vs ELT Where do we store the data – Data lake vs data warehouse Which tool to we use to ingest – cronjob

Data Ingestion 52

Data Ingestion Data Engineering Data Engineer Engineering 52

Cloudera

FEBRUARY 16, 2022

While it is a little dated, one amusing example that has been the source of countless internet memes is the famous, “is this a chihuahua or a muffin?” classification problem. Figure 01: Is this a chihuahua or a muffin? In this example, the Machine Learning (ML) model struggles to differentiate between a chihuahua and a muffin. The eyes and nose of a chihuahua, combined with the shape of its head and colour of its fur do look surprising like a muffin if we squint at the images in figure 01 above.

Machine Learning 83

Machine Learning Algorithm Government Metadata 83

Preset

FEBRUARY 13, 2022

Apache Superset is the most popular open-source data exploration and visualization platform in the world. YugabyteDB is a distributed SQL database that works seamlessly using the standards PostgreSQL connector.

PostgreSQL 52

PostgreSQL SQL Database Data 52

Grouparoo

FEBRUARY 9, 2022

Today, organizations are adopting modern ETL tools and approaches to gain as many insights as possible from their data. However, to ensure the accuracy and reliability of such insights, effective ETL testing needs to be performed. So what is an ETL tester’s responsibility? In this ETL testing tutorial, we’ll look at what ETL testing involves, the different types of ETL tests, and some challenges of ETL testing.

Process 52

Process ETL System Data Warehouse Metadata 52

Speaker: Jon Harmer, Product Manager for Google Cloud

Move from feature factory to customer outcomes and drive impact in your business! This session will provide you with a comprehensive set of tools to help you develop impactful products by shifting from output-based thinking to outcome-based thinking. You will deepen your understanding of your customers and their needs as well as identifying and de-risking the different kinds of hypotheses built into your roadmap.

Certification