Start Data Engineering
OCTOBER 12, 2021
Introduction Testing your data pipeline 1. End-to-end system testing 2. Data quality testing 3. Monitoring and alerting 4. Unit and contract testing Conclusion Further reading Introduction Testing data pipelines are different from testing other applications, like a website backend.
Uber Engineering
OCTOBER 21, 2021
Background. Apache Kafka ® is widely used across Uber’s multiple business lines. Take the example of an Uber ride: When a user opens up the Uber app, demand and supply data are aggregated in Kafka queues to serve fare calculations. … The post Introducing uGroup: Uber’s Consumer Management Framework appeared first on Uber Engineering Blog.
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Confluent
OCTOBER 28, 2021
Kafka Streams is an abstraction over Apache Kafka® producers and consumers that lets you forget about low-level details and focus on processing your Kafka data. You could of course write […].
Data Engineering Podcast
OCTOBER 28, 2021
Summary Streaming data systems have been growing more capable and flexible over the past few years. Despite this, it is still challenging to build reliable pipelines for stream processing. In this episode Eric Sammer discusses the shortcomings of the current set of streaming engines and how they force engineers to work at an extremely low level of abstraction.
Speaker: Timothy Chan, PhD., Head of Data Science
Are you ready to move beyond the basics and take a deep dive into the cutting-edge techniques that are reshaping the landscape of experimentation? From Sequential Testing to Multi-Armed Bandits, Switchback Experiments to Stratified Sampling, Timothy Chan, Data Science Lead, is here to unravel the mysteries of these powerful methodologies that are revolutionizing how we approach testing.
DataKitchen
OCTOBER 22, 2021
The post Tech workers warned they were going to quit. Now, the problem is spiralling out of control first appeared on DataKitchen.
Cloudera
OCTOBER 19, 2021
Airflow has been adopted by many Cloudera Data Platform (CDP) customers in the public cloud as the next generation orchestration service to setup and operationalize complex data pipelines. Today, customers have deployed 100s of Airflow DAGs in production performing various data transformation and preparation tasks, with differing levels of complexity.
Data Engineering Digest brings together the best content for data engineering professionals from the widest variety of industry thought leaders.
Teradata
OCTOBER 27, 2021
Recent events have illustrated the fragility of ultra-lean supply chains. Chief Supply Chain Officers must figure out how to navigate these crises to manage costs, speed & quality of service.
Confluent
OCTOBER 19, 2021
Extensive out-of-the-box functionality, a large user community, and up-to-date, cloud-native features make Spring and its libraries a strong option for anchoring your Apache Kafka® and Confluent Cloud based microservices architecture. […].
Data Engineering Podcast
OCTOBER 22, 2021
Summary The market for business intelligence has been going through an evolutionary shift in recent years. One of the driving forces for that change has been the rise of analytics engineering powered by dbt. Lightdash has fully embraced that shift by building an entire open source business intelligence framework that is powered by dbt models. In this episode Oliver Laslett describes why dashboards aren’t sufficient for business analytics, how Lightdash promotes the work that you are alread
DataKitchen
OCTOBER 18, 2021
The post 5 hot new IT jobs — and why they just might stick first appeared on DataKitchen.
Speaker: Anne Steiner and David Laribee
As a concept, Developer Experience (DX) has gained significant attention in the tech industry. It emphasizes engineers’ efficiency and satisfaction during the product development process. As product managers, we need to understand how a good DX can contribute not only to the well-being of our development teams but also to the broader objectives of product success and customer satisfaction.
Cloudera
OCTOBER 18, 2021
It’s that time of year again… Award season! We are thrilled to announce the finalists of the 2021 Data Impact Awards. This year’s entrants have excelled at demonstrating how innovative data solutions can help solve real-time challenges and positively impact people around the world. . The entries are some of the most remarkable we’ve seen, giving our judges the tough task of selecting an award worthy shortlist.
Start Data Engineering
OCTOBER 11, 2021
Introduction Skills 1. SQL 2. Python 3. Leetcode: data structures and algorithms 4. Data modeling 4.1 Data warehousing 4.2 OLTP 5. Data pipelines 6. Distributed system fundamentals 7. Event streaming 8. System design 9. Business questions 10. Cloud computing 11. Probabilistic data structures (optional) Interview prep, the TL;DR version Conclusion Introduction Are you a student, analyst, engineer, or someone preparing for a data engineering interview and overwhelmed by all the tools and concepts?
U-Next
OCTOBER 19, 2021
INTRODUCTION. The evaluation of the job method determines the value of jobs at intervals a company. Various styles of jobs area unit performed by staff in a company. Some area unit is totally changed in responsibilities to every different area and a few areas similar to happiness to the same cluster. It is important to ascertain or a method to work out the relative value of work and implement clear ways to maintain the plan for equal pay in a company.
Confluent
OCTOBER 6, 2021
Today, an organization’s strategic objective is to deliver innovations for a connected life and to improve the quality of life worldwide. With connected devices comes data, and with data comes […].
Speaker: Aarushi Kansal, AI Leader & Author and Tony Karrer, Founder & CTO at Aggregage
Software leaders who are building applications based on Large Language Models (LLMs) often find it a challenge to achieve reliability. It’s no surprise given the non-deterministic nature of LLMs. To effectively create reliable LLM-based (often with RAG) applications, extensive testing and evaluation processes are crucial. This often ends up involving meticulous adjustments to prompts.
Data Engineering Podcast
OCTOBER 15, 2021
Summary The binding element of all data work is the metadata graph that is generated by all of the workflows that produce the assets used by teams across the organization. The DataHub project was created as a way to bring order to the scale of LinkedIn’s data needs. It was also designed to be able to work for small scale systems that are just starting to develop in complexity.
Teradata
OCTOBER 4, 2021
An interdisciplinary team from Volkswagen, AWS and Teradata have created an intelligent solution that enables greater transparency and efficiency in car body construction. Find out more.
Cloudera
OCTOBER 26, 2021
As Halloween night quickly approaches, there is only one question on every kid’s mind: how can I maximize my candy haul this year with the best possible candy? This kind of question lends itself perfectly to data science approaches that enable quick and intuitive analysis of data across multiple sources. Using Cloudera Machine Learning, the world’s first hybrid data cloud machine learning tooling, let’s take a deep dive into the world of candy analytics to answer the tough question on everyone’s
Start Data Engineering
OCTOBER 12, 2021
1. Introduction 2. E-T-L definition 3. Differences between ETL & ELT 4. Conclusion 5. Further reading 1. Introduction If you are a student, analyst, engineer, or anyone working with data pipelines, you would have heard of ETL and ELT architecture. If you have questions like What is the difference between ETL & ELT? Should I use ETL or ELT pattern for my data pipeline?
Advertisement
Are you trying to decide which entity resolution capabilities you need? It can be confusing to determine which features are most important for your project. And sometimes key features are overlooked. Get the Entity Resolution Evaluation Checklist to make sure you’ve thought of everything to make your project a success! The list was created by Senzing’s team of leading entity resolution experts, based on their real-world experience.
Netflix Tech
OCTOBER 7, 2021
Martin Tingley with Wenjing Zheng , Simon Ejdemyr , Stephanie Lane , and Colin McFarland This is the third post in a multi-part series on how Netflix uses A/B tests to inform decisions and continuously innovate on our products. Need to catch up? Have a look at Part 1 (Decision Making at Netflix) and Part 2 (What is an A/B Test?). Subsequent posts will go into more details on experimentation across Netflix, how Netflix has invested in infrastructure to support and scale experimentation, and the i
Confluent
OCTOBER 29, 2021
At the recent Kafka Summit, Confluent announced the general availability of Stream Governance–the industry’s only governance suite for data in motion. Offered as a fully managed cloud solution, it delivers […].
DataKitchen
OCTOBER 7, 2021
The post How Predictive and Prescriptive Analytics Improve the Call Center Experience first appeared on DataKitchen.
Advancing Analytics: Data Engineering
OCTOBER 22, 2021
Problem Recently on a client project, we wanted to use the Auto Loader functionality in Databricks to easily consume from AWS S3 into our Azure hosted data platform. The reason why we opted for Auto Loader over any other solution is because it natively exists within Databricks and allows us to quickly ingest data from Azure Storage Accounts and AWS S3 Buckets, while using the benefits of Structured Streaming to checkpoint which files it last loaded.
Advertisement
Outdated or absent analytics won’t cut it in today’s data-driven applications – not for your end users, your development team, or your business. That’s what drove the five companies in this e-book to change their approach to analytics. Download this e-book to learn about the unique problems each company faced and how they achieved huge returns beyond expectation by embedding analytics into applications.
Cloudera
OCTOBER 7, 2021
October sees the launch of Partner Appreciation Month and during the next few weeks we will be sharing success stories, updates and interviews with our valued partners across the world. . We’re on a mission to make data and analytics easy and accessible, for everyone, and the hybrid data cloud is how we’ll get there. Today’s world is a hybrid world—there’s hybrid data, hybrid infrastructure, hybrid work—and leading businesses are embracing these changes, unafraid to transform their processes and
Start Data Engineering
OCTOBER 12, 2021
Introduction Setup Common Table Expressions (CTEs) Performance comparison CTE Subquery and derived tables Temp table Trade-offs Tear down Conclusion References Introduction If you are a student, analyst, engineer, or anyone in the data space and are Wondering what CTEs are? Trying to understand CTE performance Then this post is for you. In this post, we go over what CTEs are and compare their performance to the subquery, derived table, and temp table.
Netflix Tech
OCTOBER 27, 2021
Open-Sourcing a Monitoring GUI for Metaflow, Netflix’s ML Platform tl;dr Today, we are open-sourcing a long-awaited GUI for Metaflow. The Metaflow GUI allows data scientists to monitor their workflows in real-time, track experiments, and see detailed logs and results for every executed task. The GUI can be extended with plugins, allowing the community to build integrations to other systems, custom visualizations, and embed upcoming features of Metaflow directly into its views.
Confluent
OCTOBER 21, 2021
How do you continuously process half a terabyte of data in real-time? That’s the exact question we had to answer. Leadnomics is a digital marketing company that helps companies maximize […].
Speaker: David Bard, Principal at VP Product Coaching
In the fast-paced world of digital innovation, success is often accompanied by a multitude of challenges - like the pitfalls lurking at every turn, threatening to derail the most promising projects. But fret not, this webinar is your key to effective product development! Join us for an enlightening session to empower you to lead your team to greater heights.
DataKitchen
OCTOBER 19, 2021
The post Data Quality: Volume, interdependencies can create big problems first appeared on DataKitchen.
ProjectPro
OCTOBER 28, 2021
In this blog, we have mentioned all the topics that are considered as prerequisites for learning machine learning. We have covered all the subjects and the best resources that will help you learn them thoroughly. Upskilling in the era of the Internet has become hassle-free.The Internet has given a platform to experts who can now share their knowledge with a large number of people and help those people in acquiring new skills irrespective of their previous knowledge about the subject.
Cloudera
OCTOBER 28, 2021
I’ve had the pleasure to participate in a few Commercial Lines insurance industry events recently and as a prior Commercial Lines insurer myself, I am thrilled with the progress the industry is making using data and analytics. However, I do not think Commercial Lines insurance gets the credit it deserves for the industry-leading role it has played in analytics.
Start Data Engineering
OCTOBER 12, 2021
Introduction Prerequisites 6 Key Concepts 1. When to Use 2. Partition By 3. Order By 4. Function 5. Lead and Lag 6. Rolling Window Efficiency Considerations Conclusion Further reading References Introduction If work with data, window functions can significantly level up your SQL skills.
Speaker: Jon Harmer, Product Manager for Google Cloud
Move from feature factory to customer outcomes and drive impact in your business! This session will provide you with a comprehensive set of tools to help you develop impactful products by shifting from output-based thinking to outcome-based thinking. You will deepen your understanding of your customers and their needs as well as identifying and de-risking the different kinds of hypotheses built into your roadmap.
Expert insights. Personalized for you.
173 
Let's personalize your content