Start Data Engineering
OCTOBER 22, 2022
1. Introduction 2. Data project template 2.1. Prerequisites 2.2. Setup infra 2.3. Tear down infra 3. Set up data infrastructure 3.1. Run data infra on your laptop with containers 3.2. Manage cloud infrastructure with code 4. Set up development workflow 4.1. CI: Automated tests & checks before the merge with GitHub Actions 4.2. CD: Deploy to production servers with GitHub Actions 4.3.
The Pragmatic Engineer
OCTOBER 27, 2022
This issue was written in Oct 2022, sent out to all subscribers of The Pragmatic Engineer Newsletter in October 2022. The observations on how Big Tech hiring will slow down have since been validated, with Meta not only laying off in November, but also rescinding offers in January 2023, and Amazon doing the same. If you want to get the pulse of the industry in your inbox, subscribe.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
KDnuggets
OCTOBER 24, 2022
Preprocessing data for machine learning models is a core general skill for any Data Scientist or Machine Learning Engineer. Follow this guide using Pandas and Scikit-learn to improve your techniques and make sure your data leads to the best possible outcome.
Data Engineering Podcast
OCTOBER 23, 2022
Summary The database market has seen unprecedented activity in recent years, with new options addressing a variety of needs being introduced on a nearly constant basis. Despite that, there are a handful of databases that continue to be adopted due to their proven reliability and robust features. MariaDB is one of those default options that has continued to grow and innovate while offering a familiar and stable experience.
Speaker: Timothy Chan, PhD., Head of Data Science
Are you ready to move beyond the basics and take a deep dive into the cutting-edge techniques that are reshaping the landscape of experimentation? 🌐 From Sequential Testing to Multi-Armed Bandits, Switchback Experiments to Stratified Sampling, Timothy Chan, Data Science Lead, is here to unravel the mysteries of these powerful methodologies that are revolutionizing how we approach testing.
Teradata
OCTOBER 25, 2022
Developing an IT sustainability strategy can bring major positive change across the enterprise, lowering costs and optimizing resource use.
Cloudera
OCTOBER 23, 2022
Demand for both entry-level and highly skilled tech talent is at an all-time high, and companies across industries and geographies are struggling to find qualified employees. And, with 1.1 billion jobs liable to be radically transformed by technology in the next decade, a “ reskilling revolution ” is reaching a critical mass. Already underrepresented populations like workers without a four-year degree are four times more likely to work in highly automatable jobs than individuals with a bachelor’
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
Pinterest Engineering
OCTOBER 28, 2022
Lin Wang | Android Performance Engineer Designed by AJ Oxendine | Software Engineer It’s a well-known fact for Android developers that an app’s manifest (AndroidManifest.xml) holds crucial application declarations. It is rarely monitored after being set up because we assume it hardly ever changes. At Pinterest, however, we have been actively monitoring the manifest after realizing it does change every so often.
U-Next
OCTOBER 28, 2022
Introduction . Artificial Intelligence ( AI technology ) is the latest buzzword in the world of technology. We are moving towards a more intelligent world where machines are able to think, learn and make decisions on their own. AI has been used in various industries for years now. It has been used to improve search engines and provide recommendations based on your past searches. .
DataKitchen
OCTOBER 27, 2022
. Question: What is something the data industry is missing? I think it’s observability-led DataOps. I’ve come to believe that we, as an industry, will not change how people build things they’ve already made. They’re already being Heroes and have pain, unhappiness, and poor results. The first step to enlightenment. The first step in solving that pain is to observe what’s happening with your data and analytics ‘estate’ and stick little thermometers at va
KDnuggets
OCTOBER 28, 2022
If you’re someone in data science or aiming to get into a data science career, this article will give you a comprehensive analysis of the state of the field.
Speaker: Anne Steiner and David Laribee
As a concept, Developer Experience (DX) has gained significant attention in the tech industry. It emphasizes engineers’ efficiency and satisfaction during the product development process. As product managers, we need to understand how a good DX can contribute not only to the well-being of our development teams but also to the broader objectives of product success and customer satisfaction.
Pinterest Engineering
OCTOBER 26, 2022
Bella Huang | Software Engineer, Home Candidate Generation; Raymond Hsu | Engineer Manager, Home Candidate Generation; Dylan Wang | Engineer Manager, Home Relevance In Homefeed, ~30% of recommended pins come from pin to pin-based retrieval. This means that during the retrieval stage, we use a batch of query pins to call our retrieval system to generate pin recommendations.
U-Next
OCTOBER 27, 2022
Introduction . An MIS ( Management Information Systems ) executive is responsible for the management of an organization’s computer systems, applications, and networks. This includes overseeing the information technology (IT) department and ensuring that all platforms, including hardware, software, and telecommunications systems, are running smoothly.
Confluent
OCTOBER 25, 2022
Joining two topics to aggregate data is fundamental in stream processing, but it’s not easy. Learn how to use kcat to debug and ensure two topics use the same keys in the same partitions.
KDnuggets
OCTOBER 28, 2022
In this article, I have explained some tips and tricks to optimize and speed up Python code.
Speaker: Aarushi Kansal, AI Leader & Author and Tony Karrer, Founder & CTO at Aggregage
Software leaders who are building applications based on Large Language Models (LLMs) often find it a challenge to achieve reliability. It’s no surprise given the non-deterministic nature of LLMs. To effectively create reliable LLM-based (often with RAG) applications, extensive testing and evaluation processes are crucial. This often ends up involving meticulous adjustments to prompts.
Data Engineering Weekly
OCTOBER 23, 2022
Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Sign up free to test out the tool today. Editor’s Note: DEW is the reader’s choice & Is Data Catalog living up to the hype?
U-Next
OCTOBER 27, 2022
Introduction . Cybersecurity risk management process is a topic of interest to many people. It’s because cybersecurity is a growing concern for businesses and individuals alike. As we continue to rely on technology more and more, the risk of cyber-attacks grows significantly. . So why should you gain knowledge of cybersecurity? It’s simple. you want to protect yourself and your family from malicious hackers. .
Confluent
OCTOBER 25, 2022
Announcing the complete guide to Schema Registry. In our course, you’ll learn the basics, like how schema registry works, key concepts, how to manage schemas, and more.
KDnuggets
OCTOBER 24, 2022
Learn various algorithms to improve the robustness and performance of machine learning applications. Furthermore, it will help you build a more generalized and stable model.
Advertisement
Are you trying to decide which entity resolution capabilities you need? It can be confusing to determine which features are most important for your project. And sometimes key features are overlooked. Get the Entity Resolution Evaluation Checklist to make sure you’ve thought of everything to make your project a success! The list was created by Senzing’s team of leading entity resolution experts, based on their real-world experience.
Elder Research
OCTOBER 25, 2022
The post Decision Process Improvement (DPI): Better, Faster Decisions appeared first on Elder Research.
Teradata
OCTOBER 28, 2022
Data will drive the business models of next generation commercial vehicle suppliers. Find out how.
Confluent
OCTOBER 27, 2022
How to build a complete motion detection and alerting system to power modern, real-time IoT and data streaming using Confluent.
KDnuggets
OCTOBER 26, 2022
A Data Science Portfolio That Will Land You The Job in 2022 • Is OLAP Dead? • 10 Essential SQL Commands for Data Science • Why TinyML Cases Are Becoming More Popular • Ensemble Learning with Examples.
Speaker: Margaret-Ann Seger, Head of Product, Statsig
Experimentation is often seen as an aspirational practice, especially at smaller, fast-moving companies who are strapped for time and resources. So, how can you get your team making decisions in a more data-driven way while continuing to remain lean and maintaining ship velocity? In this webinar, Margaret-Ann Seger, Head of Product at Statsig, will teach you how to build an experimentation culture from the ground-up, graduating from just getting started with data-driven development to operating
Propel Data
OCTOBER 23, 2022
Learn how to use the incremental model in dbt to manage data streams in your Snowflake warehouse.
U-Next
OCTOBER 28, 2022
Introduction . Project Management is the practice of using techniques, methods, skills, knowledge, and experience to complete specific project goals within predetermined constraints. Final deliverables in Project Management are subject to a limited amount of time and money. . Project management has a final outcome and a limited time frame as opposed to simple “management.” The latter is a continuing process, a crucial aspect that sets it apart from simple “management.”
KDnuggets
OCTOBER 27, 2022
Graph Algorithms for Data Science is a hands-on guide to working with graph-based data in applications like machine learning, fraud detection, and business data analysis. Filled with fascinating and fun projects, demonstrating the ins-and-outs of graphs.
Speaker: David Bard, Principal at VP Product Coaching
In the fast-paced world of digital innovation, success is often accompanied by a multitude of challenges - like the pitfalls lurking at every turn, threatening to derail the most promising projects. But fret not, this webinar is your key to effective product development! Join us for an enlightening session to empower you to lead your team to greater heights.
Propel Data
OCTOBER 25, 2022
This article will demonstrate how to deduplicate events in Snowflake using dbt
U-Next
OCTOBER 27, 2022
Introduction . AI cloud is a promising domain. It has recently gained prominence and has been deployed for various purposes, such as data storage, processing, and software development. The emergence of Artificial Intelligence (AI) has opened new possibilities for cloud computing. . AI is a branch of computer science that deals with making computers intelligent by writing algorithms with human-like characteristics like learning and problem-solving.
Data Engineering Podcast
OCTOBER 23, 2022
Summary Agile methodologies have been adopted by a majority of teams for building software applications. Applying those same practices to data can prove challenging due to the number of systems that need to be included to implement a complete feature. In this episode Shane Gibson shares practical advice and insights from his years of experience as a consultant and engineer working in data about how to adopt agile principles in your data work so that you can move faster and provide more value to
KDnuggets
OCTOBER 27, 2022
Edge ML involves putting ML models on consumer devices where they can independently run inferences without an internet connection, in real-time, and at no cost.
Advertisement
“Reimagined: Building Products with Generative AI” is an extensive guide for integrating generative AI into product strategy and careers featuring over 150 real-world examples, 30 case studies, and 20+ frameworks, and endorsed by over 20 leading AI and product executives, inventors, entrepreneurs, and researchers.
Expert insights. Personalized for you.
147 
Let's personalize your content