Start Data Engineering

JUNE 25, 2021

Introduction Objective Setup Pre-requisites Project 1. ETL Code 2. Test 3. Scheduler 4. Presentation 4.1. Formatting, Linting, and Type checks 4.2. Architecture Diagram 4.3. README.md 5. Adding Dashboard to your Profile Future Work Tear down infra Conclusion Further Reading References Introduction Building a data project for your portfolio is hard. Getting hiring managers to read through your Github code is even harder.

Project 130

Project Designing Management Portfolio 130

Cloudera

JUNE 1, 2021

Today marks the beginning of an exciting new chapter for Cloudera. Cloudera will become a private company with the flexibility and resources to accelerate product innovation, cloud transformation and customer growth. Cloudera will benefit from the operating capabilities, capital support and expertise of Clayton, Dubilier & Rice (CD&R) and KKR – two of the most experienced and successful global investment firms in the world recognized for supporting the growth strategies of the businesses

Cloud 144

Cloud Data Lake Big Data Finance 144

Confluent

JUNE 11, 2021

Managing Apache Kafka® clusters can be tricky sometimes. To solve this problem, Confluent Control Center helps you easily manage and monitor your clusters and interact with other Confluent components, such […].

Kafka 138

Kafka Management 138

Uber Engineering

JUNE 22, 2021

Introduction. Uber relies on a containerized microservice architecture. Our need for computational resources has grown significantly over the years, as a consequence of business’ growth. It is an important goal now to increase the efficiency of our computing resources. Broadly … The post Efficient and Reliable Compute Cluster Management at Scale appeared first on Uber Engineering Blog.

Management 130

Management Architecture Engineering IT 130

Advertisement

Many organizations today are unlocking the power of their data by using graph databases to feed downstream analytics, enahance visualizations, and more. Yet, when different graph nodes represent the same entity, graphs get messy. Watch this essential video with Senzing CEO Jeff Jonas on how adding entity resolution to a graph database condenses network graphs to improve analytics and save your analysts time.

Netflix Tech

JUNE 7, 2021

By Alok Tiagi , Hariharan Ananthakrishnan , Ivan Porto Carrero and Keerti Lakshminarayan Netflix has developed a network observability sidecar called Flow Exporter that uses eBPF tracepoints to capture TCP flows at near real time. At much less than 1% of CPU and memory on the instance, this highly performant sidecar provides flow data at scale for network insight.

Transportation 130

Transportation AWS Cloud Metadata 130

Data Engineering Podcast

JUNE 22, 2021

Summary The database is the core of any system because it holds the data that drives your entire experience. We spend countless hours designing the data model, updating engine versions, and tuning performance. But how confident are you that you have configured it to be as performant as possible, given the dozens of parameters and how they interact with each other?

Database 100

Database MySQL PostgreSQL Data Warehouse 100

Cloudera

JUNE 7, 2021

Cloudera has been named as a Strong Performer in the Forrester Wave for Streaming Analytics, Q2 2021. We are excited to be recognized in this wave at, what we consider to be, such a strong position. We are proud to have been named as one of “ The 14 providers that matter most ” in streaming analytics. The report states that richness of analytics, development tool options and near-effortless scalability are what streaming analytics customers should look for in a provider. .

Kafka 101

Kafka Data Ingestion Architecture Cloud 101

Confluent

JUNE 29, 2021

Making changes to a database schema is a natural part of software development. Often, it’s important to carefully manage the timing of changes and keep track of them over time. […].

Management 128

Management Database Process 128

Uber Engineering

JUNE 15, 2021

Introduction to Flaky Tests. Unit testing forms the bedrock of any Continuous Integration (CI) system. It warns software engineers of bugs in newly-implemented code and regressions in existing code, before it is merged. This ensures increased software reliability. It also … The post Handling Flaky Unit Tests in Java appeared first on Uber Engineering Blog.

Java 120

Java Software Engineer Software Engineering Coding 120

Netflix Tech

JUNE 22, 2021

A Script Authoring Specification By: Bhanu Srikanth, Andy Swan, Casey Wilms, Patrick Pearson The Art of Dubbing and Subtitling Dubbing and subtitling are inherently creative processes. At Netflix, we strive to make shows as joyful to watch in every language as in the original language, whether a member watches with original or dubbed audio, closed captions, forced narratives, subtitles or any combination they prefer.

Metadata 69

Metadata Technology Designing Process 69

Speaker: Scott Sehlhorst

We know we want to create products which our customers find to be valuable. Whether we label it as customer-centric or product-led depends on how long we've been doing product management. There are three challenges we face when doing this. The obvious challenge is figuring out what our users need; the non-obvious challenges are in creating a shared understanding of those needs and in sensing if what we're doing is meeting those needs.

Data Engineering Podcast

JUNE 14, 2021

Summary When you build a machine learning model, the first step is always to load your data. Typically this means downloading files from object storage, or querying a database. To speed up the process, why not build the model inside the database so that you don’t have to move the information? In this episode Paige Roberts explains the benefits of pushing the machine learning processing into the database layer and the approach that Vertica has taken for their implementation.

Machine Learning 100

Machine Learning Database Data Warehouse Hadoop 100

AltexSoft

JUNE 14, 2021

In today’s society, insurers can no longer ignore the mounting expectations of customers. Clients now expect insurers to provide different levels of personalization that are fast, adaptable, and up to date. That is why some insurers have gone further to provide insurance and risk management services that can be adjusted and rewritten in real-time depending on the changing risk in the consumer’s life.

Insurance 88

Insurance Medical Machine Learning Algorithm 88

Cloudera

JUNE 7, 2021

At the end of March, we released the first version of Cloudera SQL StreamBuilder as part of CSA 1.3. It enabled users to easily write, run and manage real-time SQL queries on streams from Apache Kafka with an exceptionally smooth user experience. . Since then, we have been working hard to expose the full power of Apache Flink SQL and the existing Data Warehousing tools in CDP to combine it into a state-of-the-art real-time analytics platform.

Kafka 101

Kafka SQL Accessible Accessibility 101

Confluent

JUNE 23, 2021

Al data til folket (all data to the people) is a compelling proposition in an enterprise context. Yet the ability to quickly address integration challenges and deliver data to those […].

Architecture 127

Architecture Data 127

Speaker: Timothy Chan, PhD., Head of Data Science

Are you ready to move beyond the basics and take a deep dive into the cutting-edge techniques that are reshaping the landscape of experimentation? 🌐 From Sequential Testing to Multi-Armed Bandits, Switchback Experiments to Stratified Sampling, Timothy Chan, Data Science Lead, is here to unravel the mysteries of these powerful methodologies that are revolutionizing how we approach testing.

Data Science

Teradata

JUNE 10, 2021

Open Banking platforms require high-reliability, seamless interfaces and should be driven by efficiency. Read more.

Banking 85

Banking Designing 85

DataKitchen

JUNE 25, 2021

In this five-module course, Mike Lampa & Chris Bergh teach data professionals to plan their organization's DataOps program for low errors & fast deployment. The post Standing Up a DataOps Program for Practitioners first appeared on DataKitchen.

Programming 59

Programming Data 59

Data Engineering Podcast

JUNE 8, 2021

Summary The way to build maintainable software and systems is through composition of individual pieces. By making those pieces high quality and flexible they can be used in surprising ways that the original creators couldn’t have imagined. One such component that has gone above and beyond its originally envisioned use case is BookKeeper, a distributed storage system that is optimized for durability and speed.

Data Warehouse 100

Data Warehouse Hadoop Metadata Architecture 100

Netflix Tech

JUNE 1, 2021

Data Engineers of Netflix?—?Interview with Dhevi Rajendran Dhevi Rajendran This post is part of our “Data Engineers of Netflix” interview series, where our very own data engineers talk about their journeys to Data Engineering @ Netflix. Dhevi Rajendran is a Data Engineer on the Growth Data Science and Engineering team. Dhevi joined Netflix in July 2020 and is one of many Data Engineers who have onboarded remotely during the pandemic.

Data Engineering 55

Data Engineering Data Engineer Engineering Software Engineer 55

Speaker: Aarushi Kansal, AI Leader & Author and Tony Karrer, Founder & CTO at Aggregage

Software leaders who are building applications based on Large Language Models (LLMs) often find it a challenge to achieve reliability. It’s no surprise given the non-deterministic nature of LLMs. To effectively create reliable LLM-based (often with RAG) applications, extensive testing and evaluation processes are crucial. This often ends up involving meticulous adjustments to prompts.

Building

Cloudera

JUNE 3, 2021

Cloudera Support’s cluster validations proactively identify known problem signatures contained in customers’ diagnostic data with the goal of increasing cluster health, performance, and overall stability. Cluster validations are included in a customer’s enterprise subscription at no additional cost. All customers with access to the Support case portal will also be able to take advantage of cluster validations.

Programming 91

Programming Consulting Designing Accessible 91

Confluent

JUNE 1, 2021

One of the canonical examples of streaming data is tracking location data over time. Whether it’s ride-sharing vehicles, the position of trains on the rail network, or tracking airplanes waking […].

Data 117

Data Process 117

Teradata

JUNE 20, 2021

Open Banking is re-shaping the landscape of financial services and introducing new types of risks extending beyond data security. Secure open banking is everyone’s responsibility.

Banking 59

Banking Data Security Data 59

DataKitchen

JUNE 30, 2021

MLOps.community leader Demetrios Brinkmann chats with DataKitchen CEO Chris Bergh about the benefits of Data Science teams doing MLOps to pull the pain forward. The post Operationalizing Machine Learning at Scale with MLOps first appeared on DataKitchen.

Machine Learning 52

Machine Learning Data Science Data 52

Advertisement

Outdated or absent analytics won’t cut it in today’s data-driven applications – not for your end users, your development team, or your business. That’s what drove the five companies in this e-book to change their approach to analytics. Download this e-book to learn about the unique problems each company faced and how they achieved huge returns beyond expectation by embedding analytics into applications.

Building

Monte Carlo

JUNE 29, 2021

I guess data runs in the family. Growing up as a kid in the ‘90s, I distinctly remember my father having to bring his laptop everywhere he went with him. Compared to today’s Macbooks and PCs, my dad’s laptop took forever to load and connected to the internet via dial-up, which made an embarrassing noise whenever we were out. The dinner table? Check?

Business Analyst 52

Business Analyst Data Warehouse Algorithm Data 52

Netflix Tech

JUNE 1, 2021

Data Engineers of Netflix?—?Interview with Samuel Setegne Samuel Setegne This post is part of our “Data Engineers of Netflix” interview series, where our very own data engineers talk about their journeys to Data Engineering @ Netflix. Samuel Setegne is a Senior Software Engineer on the Core Data Science and Engineering team. Samuel and his team build tools and frameworks that support data engineering teams across Netflix.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Science 52

Cloudera

JUNE 2, 2021

Data pipelines are in high demand in today’s data-driven organizations. As critical elements in supplying trusted, curated, and usable data for end-to-end analytic and machine learning workflows, the role of data pipelines is becoming indispensable. To keep up, data pipelines are being vigorously reshaped with modern tools and techniques. At Cloudera, we recently introduced several cutting-edge innovations in our Cloudera Data Engineering experience (CDE) as part of our Enterprise Data Cloud pro

Data Pipeline 89

Data Pipeline Data Warehouse Machine Learning Data Architect 89

Confluent

JUNE 3, 2021

What if I told you there is a query your database can’t answer? That would probably surprise you. With decades of effort behind them, databases are one of the most […].

Database 94

Database Process 94

Speaker: Anne Steiner and David Laribee

As a concept, Developer Experience (DX) has gained significant attention in the tech industry. It emphasizes engineers’ efficiency and satisfaction during the product development process. As product managers, we need to understand how a good DX can contribute not only to the well-being of our development teams but also to the broader objectives of product success and customer satisfaction.

Engineering

Teradata

JUNE 30, 2021

As banks learn to adjust to the changes enforced by the COVID pandemic, the attention of customers, regulators & shareholders is returning to another global crisis – climate change.

Banking 59

Banking Data 59

DataKitchen

JUNE 30, 2021

Joe Reis, host of the Data Nerd Herd podcast & Ternary Data CEO & Co-Founder, interviews DataKitchen CEO Chris Bergh about what DataOps is & why it matters. The post DataOps with Chris Bergh first appeared on DataKitchen.

IT 52

IT Data 52

Monte Carlo

JUNE 24, 2021

Monte Carlo , the data reliability company, today announced two new strategic hires to its leadership team: Daniel Day , Head of Marketing, and Jordan Van Horn , Head of Revenue. With experience leading award-winning go-to-market teams at Snowflake and Segment, Day and Van Horn share a deep expertise in the data industry and will help Monte Carlo meet the growing demands as the industry leader in Data Observability.

High Quality Data 52

High Quality Data Data Management IT 52

ProjectPro

JUNE 18, 2021

Why do data scientists prefer Python over Java? Java vs Python for Data Science- Which is better? Which has a better future: Python or Java in 2021? These are the most common questions that our ProjectAdvisors get asked a lot from beginners getting started with a data science career. This blog aims to answer all questions on how Java vs Python compare for data science and which should be the programming language of your choice for doing data science in 2021.

Java 52

Java Data Science Python Programming Language 52

Speaker: David Bard, Principal at VP Product Coaching

In the fast-paced world of digital innovation, success is often accompanied by a multitude of challenges - like the pitfalls lurking at every turn, threatening to derail the most promising projects. But fret not, this webinar is your key to effective product development! Join us for an enlightening session to empower you to lead your team to greater heights.

Certification