Simon Späti

JUNE 20, 2022

Data consumers, such as data analysts, and business users, care mostly about the production of data assets. On the other hand, data engineers have historically focused on modeling the dependencies between tasks (instead of data assets) with an orchestrator tool. How can we reconcile both worlds? This article reviews open-source data orchestration tools (Airflow, Prefect, Dagster) and discusses how data orchestration tools introduce data assets as first-class objects.

Data Pipeline 147

Data Pipeline Data Data Engineering Data Engineer 147

Start Data Engineering

JUNE 24, 2022

1. Introduction 2. Steps 2.1. Choosing companies to work for 2.2. Optimizing your Linkedin & resume 2.3. Landing interviews 2.4. Preparing for interviews 2.5. Offers & Negotiation 3. Conclusion 4. Further reading 5. Reference 1. Introduction The data industry is booming! & data engineering salaries are skyrocketing. But landing a new job is not an easy task.

Data Engineering 130

Data Engineering Data Engineer Engineering Data 130

Azure Data Engineering

JUNE 19, 2022

While we have discussed various ways for running custom SQL code in Azure Data Factory in a previous post , recently, a new activity has been added to Azure Data Factory called Script Activity , which provides a more flexible way of running custom SQL scripts. Azure Data Factory: Script Activity As shown in the screenshot above, this activity supports execution of custom Data Query Language (DQL) as well as Data Definition Language (DDL) and Data Manipulation Language (DML).

SQL 130

SQL Datasets Data Database 130

Data Engineering Podcast

JUNE 19, 2022

Summary Data analysis is a valuable exercise that is often out of reach of non-technical users as a result of the complexity of data systems. In order to lower the barrier to entry Ryan Buick created the Canvas application with a spreadsheet oriented workflow that is understandable to a wide audience. In this episode Ryan explains how he and his team have designed their platform to bring everyone onto a level playing field and the benefits that it provides to the organization.

Unstructured Data 130

Unstructured Data Metadata MongoDB MySQL 130

Speaker: Timothy Chan, PhD., Head of Data Science

Are you ready to move beyond the basics and take a deep dive into the cutting-edge techniques that are reshaping the landscape of experimentation? 🌐 From Sequential Testing to Multi-Armed Bandits, Switchback Experiments to Stratified Sampling, Timothy Chan, Data Science Lead, is here to unravel the mysteries of these powerful methodologies that are revolutionizing how we approach testing.

Data Science

KDnuggets

JUNE 23, 2022

Essential Linux commands to improve the data science workflow. It will give you the power to automate tasks, build pipelines, access file systems, and enhance development operations.

Data Science 155

Data Science Data Accessible Accessibility 155

Cloudera

JUNE 18, 2022

Cloudera customers run some of the biggest data lakes on earth. These lakes power mission critical large scale data analytics, business intelligence (BI), and machine learning use cases, including enterprise data warehouses. In recent years, the term “data lakehouse” was coined to describe this architectural pattern of tabular analytics over data in the data lake.

Data Lake 93

Data Lake Data Warehouse BI SQL 93

know.bi

JUNE 22, 2022

This guide will teach you the process of exporting data from a relational database (MySQL) and importing it into a graph database (Neo4j). You will learn how to take data from the relational system and to the graph by translating the schema and using Apache Hop as import tools. This Tutorial uses a specific data set, but the principles in this tutorial can be applied and reused with any data domain.

MySQL 52

MySQL Relational Database Metadata Database 52

KDnuggets

JUNE 21, 2022

Collect validated user behavior data that’s ready to model on without prepwork. Take models built on one dataset and deploy & run them on another.

Datasets 160

Datasets Data 160

Cloudera

JUNE 22, 2022

Across the globe, cloud concentration risk is coming under greater scrutiny. The UK HM Treasury department recently issued a policy paper “ Critical Third Parties to the Finance Sector.” The paper is a proposal to enable oversight of third parties providing critical services to the UK financial system. The proposal would grant authority to classify a third party as “critical” to the financial stability and welfare of the UK financial system, and then provide governance in order to minimize the p

Cloud 83

Cloud Insurance Banking Government 83

Pipeline Data Engineering

JUNE 22, 2022

It’s time to share some important news with you: we’re taking time off to focus on our health and families, the launch of new data engineering cohorts is on hold until further notice. Health and family Running a bootstrapped company in times of repeated economic crises and data industry vibe shifts is a gift and a curse at the same time. No surprises here: it can be highly rewarding and joyful, but it can be exhausting and stressful as well.

Data Architect 52

Data Architect Education Data Architecture Data Engineering 52

Speaker: Anne Steiner and David Laribee

As a concept, Developer Experience (DX) has gained significant attention in the tech industry. It emphasizes engineers’ efficiency and satisfaction during the product development process. As product managers, we need to understand how a good DX can contribute not only to the well-being of our development teams but also to the broader objectives of product success and customer satisfaction.

Engineering

Zalando Engineering

JUNE 22, 2022

What were the biggest learnings in your career so far? And what advice would you give your younger self today? How do you get ahead in your career? We’re celebrating International Women in Engineering Day by talking to three senior Zalando Women in Tech: Mahak Swami , Engineering Manager; Floriane Gramlich , Director of Product Payments; and Ana Peleteiro Ramallo , Head of Applied Science.

Engineering 52

Engineering Designing Building IT 52

KDnuggets

JUNE 20, 2022

Check out Super Study Guide: Algorithms and Data Structures, a free ebook covering foundations, data structures, graphs, and trees, sorting and searching.

Algorithm 136

Algorithm Data Data Science 136

Yelp Engineering

JUNE 21, 2022

Our community of users will always come first, which is why Yelp takes significant measures to protect sensitive user information. In this spirit, the Database Reliability Engineering team implemented a data sanitization process long ago to prevent any sensitive information from leaving the production environment. The data sanitization process still enables developers to test new features and asynchronous jobs against a complete, real time dataset without complicated data imports.

MySQL 52

MySQL Datasets Data Database 52

U-Next

JUNE 22, 2022

Scrum is a framework for developing complicated products under the Agile product development umbrella. The term scrum is also used during a sprint to describe the daily standup sessions. A sprint is one iteration of a continuous development cycle that is timed. During a Sprint, the team must complete a set amount of work and prepare it for review. Sprints are the smallest and most reliable time intervals used by scrum teams.

Process 52

Process IT Management 52

Speaker: Aarushi Kansal, AI Leader & Author and Tony Karrer, Founder & CTO at Aggregage

Software leaders who are building applications based on Large Language Models (LLMs) often find it a challenge to achieve reliability. It’s no surprise given the non-deterministic nature of LLMs. To effectively create reliable LLM-based (often with RAG) applications, extensive testing and evaluation processes are crucial. This often ends up involving meticulous adjustments to prompts.

Building

Emeritus

JUNE 20, 2022

Data is the new oil. In a crude, unrefined form, it is of no real use. But once it is cleaned and processed, its value shoots up. From understanding customer behavior to sales performance, everything makes more sense when data is analyzed the right way. The ability to take existing data, and process it with… The post 10 Best Online Data Science Courses Hand-Picked for You appeared first on Emeritus Online Courses.

Data Science 52

Data Science Data Process IT 52

KDnuggets

JUNE 20, 2022

Tech visionaries to address accelerating machine learning, unifying AI platforms and taking intelligence to the edge, at the fifth annual AI Hardware Summit & Edge AI Summit, Santa Clara.

Machine Learning 126

Machine Learning 126

Teradata

JUNE 21, 2022

Companies have started to explore deployment of 5G networks across their value chains. This post will look at the impact of 5G on manufacturing value chain activities.

Manufacturing 52

Manufacturing 52

U-Next

JUNE 21, 2022

The distinction between hashing and encryption is that hashing refers to converting permanent data into message digests, but encryption operates in two ways: decoding and encoding the data. Hashing serves to maintain the information’s integrity, while md5 encryption and decryption are used to keep data out of the hands of third parties. Encryption and Hashing difference appears to be indistinguishable, yet they are not.

Algorithm 52

Algorithm Banking Utilities Data Security 52

Advertisement

Are you trying to decide which entity resolution capabilities you need? It can be confusing to determine which features are most important for your project. And sometimes key features are overlooked. Get the Entity Resolution Evaluation Checklist to make sure you’ve thought of everything to make your project a success! The list was created by Senzing’s team of leading entity resolution experts, based on their real-world experience.

Project

Confluent

JUNE 22, 2022

How to use data pipelines, unlock the benefits of real-time data flow, and achieve seamless data streaming and analytics at scale with Confluent.

Data Pipeline 52

Data Pipeline Kafka Data 52

KDnuggets

JUNE 24, 2022

In this article we introduce a few tools and techniques for studying relationships between the stock market and the news. We explore time series processing, anomaly detection, and an event-based view of the news. We also generate intuitive charts to demonstrate some of these concepts, and share the code behind all of this in a notebook.

Coding 120

Coding Process Data 120

Rock the JVM

JUNE 20, 2022

This article is brought to you by Yadu Krishnan , a new contributor to Rock the JVM. He’s a senior developer and constantly shares his passion for new languages, libraries and technologies. He also loves writing Scala articles, especially for newcomers. This is a beginner-friendly article to get started with Slick, a popular database library in Scala.

Scala 40

Scala PostgreSQL Database SQL 40

U-Next

JUNE 21, 2022

Introduction. People naturally spend a substantial portion of their day online now that digital media has become an essential part of their lives. As a result, digital platforms have become a very familiar location for individuals worldwide, and people have begun to trust the information provided on digital platforms. The term refers to any electronic information on our computers or cell phones.

Digital Media 52

Digital Media Insurance Electronics Media 52

Speaker: Margaret-Ann Seger, Head of Product, Statsig

Experimentation is often seen as an aspirational practice, especially at smaller, fast-moving companies who are strapped for time and resources. So, how can you get your team making decisions in a more data-driven way while continuing to remain lean and maintaining ship velocity? In this webinar, Margaret-Ann Seger, Head of Product at Statsig, will teach you how to build an experimentation culture from the ground-up, graduating from just getting started with data-driven development to operating

Building

Marc Lamberti

JUNE 19, 2022

Dynamic Task Mapping is a new feature of Apache Airflow 2.3 that puts your DAGs to a new level. Now, you can create tasks dynamically without knowing in advance how many tasks you need. This feature is for you if you want to process various files, evaluate multiple machine learning models, or process a varied number of data based on a SQL request. Excited?

SQL 130

SQL Coding Machine Learning Python 130

KDnuggets

JUNE 20, 2022

Also: Decision Tree Algorithm, Explained; 15 Python Coding Interview Questions You Must Know For Data Science; Naïve Bayes Algorithm: Everything You Need to Know; Primary Supervised Learning Algorithms Used in Machine Learning.

Algorithm 120

Algorithm Machine Learning Data Science Python 120

Data Engineering Podcast

JUNE 19, 2022

Summary Metadata is the lifeblood of your data platform, providing information about what is happening in your systems. A variety of platforms have been developed to capture and analyze that information to great effect, but they are inherently limited in their utility due to their nature as storage systems. In order to level up their value a new trend of active metadata is being implemented, allowing use cases like keeping BI reports up to date, auto-scaling your warehouses, and automated data g

Metadata 130

Metadata MongoDB Scala MySQL 130

U-Next

JUNE 21, 2022

Principal component analysis in machine learning. Principal component analysis in machine learning is a statistical procedure that employs an immaterial transformation to convert a set of correlated variables into uncorrelated variables. PCA in machine learning is the most widely used tool in exploratory data analysis and predictive modeling in machine learning.

Machine Learning 52

Machine Learning Datasets Data Science Data Analysis 52

Speaker: David Bard, Principal at VP Product Coaching

In the fast-paced world of digital innovation, success is often accompanied by a multitude of challenges - like the pitfalls lurking at every turn, threatening to derail the most promising projects. But fret not, this webinar is your key to effective product development! Join us for an enlightening session to empower you to lead your team to greater heights.

Certification

Cloudera

JUNE 24, 2022

Much of the hype around big data and analytics focuses on business value and bottom-line impacts. Those are enormously important in the private and public sectors alike. But for government agencies, there is a greater mission: improving people’s lives. Data makes the most ambitious and even idealistic goals —like making the world a better place — possible.

Government 76

Government Metadata Data Algorithm 76

KDnuggets

JUNE 22, 2022

Let’s get into some of the expectations of data scientists – and the reality they face.

Data Science 156

Data Science Data 156

Rockset

JUNE 21, 2022

We’re excited to announce that Rockset’s new connector with Snowflake is now available and can increase cost efficiencies for customers building real-time analytics applications. The two systems complement each other well, with Snowflake designed to process large volumes of historical data and Rockset built to provide millisecond-latency queries , even when tens of thousands of users are querying the data concurrently.

Kafka 52

Kafka Data Warehouse BI Analytics Application 52

Monte Carlo

JUNE 23, 2022

When it comes to big data quality, bigger data isn’t always better data. But at times we are guilty of forgetting this. At some point in the last two decades, the size of our data became inextricably linked to our ego. The bigger the better. We watched enviously as FAANG companies talked about optimizing hundreds of petabyes in their data lakes or data warehouses.

Big Data 40

Big Data Management Machine Learning Data Warehouse 40

Advertisement

“Reimagined: Building Products with Generative AI” is an extensive guide for integrating generative AI into product strategy and careers featuring over 150 real-world examples, 30 case studies, and 20+ frameworks, and endorsed by over 20 leading AI and product executives, inventors, entrepreneurs, and researchers.

Building