Data Engineering Podcast

MAY 26, 2019

Summary Some problems in data are well defined and benefit from a ready-made set of tools. For everything else, there’s Pachyderm, the platform for data science that is built to scale. In this episode Joe Doliner, CEO and co-founder, explains how Pachyderm started as an attempt to make data provenance easier to track, how the platform is architected and used today, and examples of how the underlying principles manifest in the workflows of data engineers and data scientists as they collabor

Data Science 100

Data Science Data Pipeline Data Kafka 100

Uber Engineering

MAY 14, 2019

Uber operates on a global scale across more than 600 cities, with our apps relying entirely on wireless connectivity from over 4,500 mobile carriers. To deliver the real-time performance expected from Uber’s users, our mobile apps require low-latency and highly … The post Employing QUIC Protocol to Optimize Uber’s App Performance appeared first on Uber Engineering Blog.

Engineering 112

Engineering Architecture 112

Confluent

MAY 23, 2019

Let us cut to the chase: Kafka Summit London session videos are available! If you were there, you know what a great time it was, and you know that you had to make sometimes-agonizing decisions about which sessions to attend and which to miss. Well, now you can make all those tradeoffs right by watching the whole catalog. And if you weren’t there? Well, dig in and start learning!

Kafka 109

Kafka Cloud Database Engineering 109

Netflix Tech

MAY 1, 2019

by Guillaume du Pontavice, Phill Williams and Kylee Peña (on behalf of our Streaming Algorithms, Audio Algorithms, and Creative Technologies teams) Remember the epic opening sequence of Stranger Things 2 ? The thrill of that car chase through Pittsburgh not only introduced a whole new set of mysteries, but it returned us to a beloved and dangerous world alongside Dustin, Lucas, Mike, Will and Eleven.

Engineering 109

Engineering Algorithm Media Entertainment 109

Advertisement

Many organizations today are unlocking the power of their data by using graph databases to feed downstream analytics, enahance visualizations, and more. Yet, when different graph nodes represent the same entity, graphs get messy. Watch this essential video with Senzing CEO Jeff Jonas on how adding entity resolution to a graph database condenses network graphs to improve analytics and save your analysts time.

Database

Teradata

MAY 1, 2019

Teradata CMO Martyn Etherington discusses how brands can build, measure, and maintain brand equity. He also explains why customer experience is critical to a brand's success.

Building 97

Building 97

Rockset

MAY 24, 2019

Remote work affords organizations access to more talent and offers workers greater flexibility in their lives. With a vision for everyone to be able to work from anywhere, FULL Creative runs a contact center service using fully remote teams, tapping into the growing share of employees working remotely. FULL agents answer calls on behalf of 7,000 clients of all sizes, from plumbers to parking garages to legal and medical professionals.

Management 52

Management Medical SQL Utilities 52

Zalando Engineering

MAY 8, 2019

AWS SQS polling from sharded Akka Cluster running on Kubernetes NOTE: This blog post requires the reader to have prior knowledge of AWS SQS , Akka Actors and Akka Cluster Sharding. My last post introduced Akka Cluster Sharding as a Distributed Cache running on Kubernetes. As that Proof-of-concept(PoC) proved promising, we started building a high-throughput and low-latency system based on the gained experiences and learnings.

AWS 52

AWS Architecture Systems Process 52

Confluent

MAY 13, 2019

In the last year, we’ve experienced enormous growth on Confluent Cloud, our fully managed Apache Kafka ® service. Confluent Cloud now handles several GB/s of traffic—a 200-fold increase in just six months. As Confluent Cloud has grown, we’ve noticed two gaps that very clearly remain to be filled in managed Apache Kafka services. First, all the Kafka services out there still require you to size and provision a cluster, which inevitably leads to a poor developer experience, over-provisioned capaci

Kafka 100

Kafka Cloud Management Building 100

Netflix Tech

MAY 21, 2019

Lerner?—?using RL agents for test case scheduling By: Stanislav Kirdey , Kevin Cureton , Scott Rick , Sankar Ramanathan Introduction Netflix brings delightful customer experiences to homes on a variety of devices that continues to grow each day. The device ecosystem is rich with partners ranging from Silicon-on-Chip (SoC) manufacturers, Original Design Manufacturer (ODM) and Original Equipment Manufacturer (OEM) vendors.

Amazon Web Services 65

Amazon Web Services Manufacturing Python AWS 65

Teradata

MAY 23, 2019

Using Vantage, Air France-KLM Group performs cross-channel analytics of customer data to provide a seamless experience for their passengers.

Data 96

Data 96

Speaker: Scott Sehlhorst

We know we want to create products which our customers find to be valuable. Whether we label it as customer-centric or product-led depends on how long we've been doing product management. There are three challenges we face when doing this. The obvious challenge is figuring out what our users need; the non-obvious challenges are in creating a shared understanding of those needs and in sensing if what we're doing is meeting those needs.

Certification

Rockset

MAY 17, 2019

We love a good debate. And we love data. So when the existential question of spaces vs. tabs came up in our team, we just had to run a real-time survey and collect thousands of data points around it. While we were at it, we figured it was time to settle the debate around other equally important developer issues like Hint vs. LaCroix, Vim vs. Emacs, and more.

NoSQL 52

NoSQL SQL Kafka Data Pipeline 52

Data Engineering Podcast

MAY 6, 2019

Summary The database market continues to expand, offering systems that are suited to virtually every use case. But what happens if you need something customized to your application? FoundationDB is a distributed key-value store that provides the primitives that you need to build a custom database platform. In this episode Ryan Worl explains how it is architected, how to use it for your applications, and provides examples of system design patterns that can be built on top of it.

Systems 100

Systems MongoDB NoSQL Database 100

InData Labs

MAY 4, 2019

Businesses of mid and large scale have massive amounts of printed documents in daily use. Among them are invoices, receipts, corporate documents, reports, media releases. And millions of them can be handwritten, which makes documents understandable for humans but difficult to read for machines. Basic Concept of OCR Optical character recognition (OCR) algorithms allow computers.

Algorithm 52

Algorithm Process Media Technology 52

Confluent

MAY 2, 2019

When managing Apache Kafka ® clusters at scale, tasks that are simple on small clusters turn into significant burdens. To be fair, a lot of things turn into significant burdens at scale, and it’s Confluent Control Center’s job to ease as many of them as possible. In Confluent Platform 5.2, Control Center has grown a couple of new features that make large deployments a little more pleasant to manage: It has become much better at managing configuration changes among a large number of brokers, and

Kafka 88

Kafka Management Food Consulting 88

Speaker: Timothy Chan, PhD., Head of Data Science

Are you ready to move beyond the basics and take a deep dive into the cutting-edge techniques that are reshaping the landscape of experimentation? 🌐 From Sequential Testing to Multi-Armed Bandits, Switchback Experiments to Stratified Sampling, Timothy Chan, Data Science Lead, is here to unravel the mysteries of these powerful methodologies that are revolutionizing how we approach testing.

Data Science

Netflix Tech

MAY 1, 2019

By Ed Ballot “Creating a good API is hard.”?—? anyone who has created an API used by others As with any API, wrapping your data stream in a Rx observable requires consideration for reasonable error handling and intuitive behavior. The following guidelines are intended to help developers create consistent and intuitive API. Since we frequently create Rx Observables in our Android app, we needed a common understanding of when to use onNext() and when to use onError() to make the API more consisten

Database 63

Database Coding Systems Building 63

Teradata

MAY 14, 2019

Cheryl Wiebe explains why AI for industrial use cases is a more complicated road than it appears.

IT 103

IT 103

Rockset

MAY 6, 2019

Many early adopters of cryptocurrency were individuals at the forefront of this technology, but enterprises are now increasingly getting more involved. As using cryptocurrency for business transactions becomes more commonplace, Decore aims to make accounting as streamlined as possible for companies accepting and sending crypto. Conceived as a “Quickbooks for crypto,” Decore provides accounting solutions for companies that have adopted crypto.

Banking 52

Banking MySQL AWS Architecture 52

Advancing Analytics: Data Engineering

MAY 2, 2019

In the last Docker for Data Science blog we looked at where Docker came from and why it is important. In this blog we will get Docker installed and configured on either Windows or Mac. Installing Docker. Below are instructions for installing Docker on both Windows and on Mac. <important>Before we begin, there are a few different methods for installing Docker on Windows and Mac.

Data Science 52

Data Science Data Accessible Accessibility 52

Speaker: Aarushi Kansal, AI Leader & Author and Tony Karrer, Founder & CTO at Aggregage

Software leaders who are building applications based on Large Language Models (LLMs) often find it a challenge to achieve reliability. It’s no surprise given the non-deterministic nature of LLMs. To effectively create reliable LLM-based (often with RAG) applications, extensive testing and evaluation processes are crucial. This often ends up involving meticulous adjustments to prompts.

Building

Zalando Engineering

MAY 15, 2019

A closer look at how the Linux kernel influences Redis memory management Recently, I was talking to a long-time friend, previous university colleague and former boss, who mentioned the fact that Redis was failing to persist data to disk in low memory conditions. For that reason, he advised to never let a Redis in-memory dataset to be bigger than 50% of the system memory.

Datasets 55

Datasets Python Systems Coding 55

Confluent

MAY 14, 2019

A technology community is made up of people. Without people writing code, writing tutorials, welcoming newcomers, giving presentations, and answering questions, what we have is not a community, but just a set of Git repositories. We’re grateful that the Apache Kafka ® and Confluent communities are composed of hundreds of thousands of people all over the world who are constantly doing all of these things.

Programming 83

Programming Kafka Education Coding 83

Cloudera

MAY 9, 2019

The number one challenge that enterprises struggle with their IoT implementation is not being able to measure if they are successful or not with it. Most of the enterprises start an IoT initiative without assessing their potential prior hand to be able to complete it. Even if they complete it, they lack the ability to identify and correlate the success metrics with key business goals.

Manufacturing 52

Manufacturing Data Ingestion Architecture Data Governance 52

Teradata

MAY 2, 2019

Ritu Jain explains the value of the Teradata Real Time Interaction Manager (RTIM) and why personalized customer experiences are so critical for marketers.

Management 81

Management 81

Advertisement

Outdated or absent analytics won’t cut it in today’s data-driven applications – not for your end users, your development team, or your business. That’s what drove the five companies in this e-book to change their approach to analytics. Download this e-book to learn about the unique problems each company faced and how they achieved huge returns beyond expectation by embedding analytics into applications.

Building

Rockset

MAY 31, 2019

Live dashboards can help organizations make sense of their event data and understand what's happening in their businesses in real time. Marketing managers constantly want to know how many signups there were in the last hour, day, or week. Product managers are always looking to understand which product features are working well and most heavily utilized.

BI 40

BI Java Data SQL 40

Netflix Tech

MAY 30, 2019

By Juliano Moraes , David Henry , Corey Grunewald & Jim Isaacs Recently Netflix has started building mobile apps to bring technology and innovation to our Studio Physical Productions , the portion of the business responsible for producing our TV shows and movies. Our very first mobile app is called Prodicle and was built for Android & iOS using the same reactive architecture in both platforms, which allowed us to build 2 apps from scratch in 3 months with 4 software engineers.

Architecture 62

Architecture Coding Software Engineer Software Engineering 62

Zalando Engineering

MAY 26, 2019

This blog post describes how we manage the process of proposing, reviewing and approving projects to become open source, while at the same time ensuring project code follows our compliance rules, and the maintainers of the projects are aware of their responsibilities. See our formal release guidelines Overview The process involves five steps that take the project from internal source code, through a review phase to our incubator, which eventually results in the project being graduated into our t

Project 40

Project PostgreSQL Coding Machine Learning 40

Confluent

MAY 2, 2019

When managing Apache Kafka ® clusters at scale, tasks that are simple on small clusters turn into significant burdens. To be fair, a lot of things turn into significant burdens at scale, and it’s Confluent Control Center’s job to ease as many of them as possible. In Confluent Platform 5.2, Control Center has grown a couple of new features that make large deployments a little more pleasant to manage: It has become much better at managing configuration changes among a large number of brokers, and

Kafka 57

Kafka Management Food Consulting 57

Speaker: Anne Steiner and David Laribee

As a concept, Developer Experience (DX) has gained significant attention in the tech industry. It emphasizes engineers’ efficiency and satisfaction during the product development process. As product managers, we need to understand how a good DX can contribute not only to the well-being of our development teams but also to the broader objectives of product success and customer satisfaction.

Engineering

Cloudera

MAY 28, 2019

Gartner states that “By 2022, 75% of new end-user solutions leveraging machine learning (ML) and AI techniques will be built with commercial instead of open source platforms” ¹. Spoiler alert: it’s not because data scientists will stop relying on open source for the latest innovation in ML algorithms and development environments. But rather as businesses look to operationalize machine learning capabilities at scale, they’ll turn increasingly to commercial platforms, with connectors to open so

Data Science 47

Data Science Transportation Machine Learning Algorithm 47

Teradata

MAY 21, 2019

Chad Meley explains how the concept of compound interest can be applied to your data and analytics investment strategy.

Data Analytics 89

Data Analytics Data 89

Rockset

MAY 17, 2019

The best way to answer questions about user behavior is often to gather data. A common pattern is to track user clicks throughout a product, then perform analytical queries on the resulting data, getting a holistic understanding of user behavior. In my case, I was curious to get a pulse of developer preferences on several divisive questions. So, I built a simple survey and gathered tens of thousands of data points from developers on the Internet.

Building 52

Building SQL NoSQL Data 52

Teradata

MAY 6, 2019

Heading to Teradata Universe 2019? Camille Schmidt lists the "8 Places to Visit in Denver" while attending the flagship conference.

81

81

Speaker: David Bard, Principal at VP Product Coaching

In the fast-paced world of digital innovation, success is often accompanied by a multitude of challenges - like the pitfalls lurking at every turn, threatening to derail the most promising projects. But fret not, this webinar is your key to effective product development! Join us for an enlightening session to empower you to lead your team to greater heights.

Certification