Data Engineering Podcast

OCTOBER 28, 2018

Summary Jupyter notebooks have gained popularity among data scientists as an easy way to do exploratory analysis and build interactive reports. However, this can cause difficulties when trying to move the work of the data scientist into a more standard production environment, due to the translation efforts that are necessary. At Netflix they had the crazy idea that perhaps that last step isn’t necessary, and the production workflows can just run the notebooks directly.

Scala 100

Scala Python Data Engineering Data Engineer 100

Uber Engineering

OCTOBER 17, 2018

Uber is committed to delivering safer and more reliable transportation across our global markets. To accomplish this, Uber relies heavily on making data-driven decisions at every level, from forecasting rider demand during high traffic events to identifying and addressing bottlenecks … The post Uber’s Big Data Platform: 100+ Petabytes with Minute Latency appeared first on Uber Engineering Blog.

Big Data 109

Big Data Transportation Data Engineering 109

Cloudera

OCTOBER 1, 2018

What an evening! Last week Cloudera hosted over 150 attendees at our annual EMEA Partner Summit in Amsterdam with attendees from over 21 countries across the region. Representatives from across the Cloudera ecosystem came together to hear from company executives and EMEA leadership as well as interactive sessions on Machine Learning, AI and Data Analytics, Cloud and Platform as well training and certification opportunities.

Machine Learning 40

Machine Learning Consulting Cloud Certification 40

Zalando Engineering

OCTOBER 24, 2018

A Scala 3 Experiment I'll start this post by admitting that I’ve never gone deeply into any kind of Scala coding on the typelevel. It's not what I, as a common application (or microservice) developer, usually need. Having stated that, of course, I might be missing out on a whole world of opportunities for better code without knowing. And because of that, I put some effort into trying to understanding the features of Scala that might sound strange, overly-theoretical, and maybe even useless, at f

Scala 40

Scala Coding Database IT 40

Speaker: Timothy Chan, PhD., Head of Data Science

Are you ready to move beyond the basics and take a deep dive into the cutting-edge techniques that are reshaping the landscape of experimentation? 🌐 From Sequential Testing to Multi-Armed Bandits, Switchback Experiments to Stratified Sampling, Timothy Chan, Data Science Lead, is here to unravel the mysteries of these powerful methodologies that are revolutionizing how we approach testing.

Data Science

Rockset

OCTOBER 19, 2018

I love open-source but open-source software for data infrastructure is on the way out. There, I said it. And you might think I've got a screw loose, given the broad adoption of open source today, but hear me out. Yes, open source is ubiquitous in data management today, but the era of open-source innovation is all but over. In the age of public cloud, there is no longer a reason to build or use open source for data infrastructure, and a new category of software I'm labeling open services will ren

MongoDB 40

MongoDB Hadoop Kafka Data Warehouse 40

ProjectPro

OCTOBER 5, 2018

Hadoop-as-a-Service: The Need Of The Hour For Superior Business Solutions.InsideBigData.com, September 7, 2018 Hadoop is the cornerstone of the big data industry, however, the challenges involved in maintaining the hadoop network has led to the development and growth of Hadoop-as-a-Service (HaaS) market.Industry research reveals that the global Hadoop-as-a-Service market is anticipated to reach $16.2 billion by 2020 growing a a compound annual growth rate of 70.8% from 2014 to 2020.With market l

Hadoop 40

Hadoop BI MongoDB Big Data 40

Uber Engineering

OCTOBER 30, 2018

Cluster management, a common software infrastructure among technology companies, aggregates compute resources from a collection of physical hosts into a shared resource pool, amplifying compute power and allowing for the flexible use of data center hardware. At Uber, cluster management … The post Peloton: Uber’s Unified Resource Scheduler for Diverse Cluster Workloads appeared first on Uber Engineering Blog.

Management 103

Management Engineering Technology MySQL 103

Cloudera

OCTOBER 3, 2018

We’ve just announced that Cloudera and Hortonworks have agreed to merge to form a single company. I want to explain the thinking behind the deal and the combination. Rob Bearden from Hortonworks has written up a post sharing his thoughts, as well. First, remember the history of Apache Hadoop. Google built an innovative scale-out platform for data storage and analysis in the late 1990s and early 2000s, and published research papers about their work.

Hadoop 75

Hadoop Cloud Data Storage Big Data 75

Zalando Engineering

OCTOBER 9, 2018

How we revolutionized the way we worked agile One and a half years ago we started something new at Zalando. We asked all producers of our department to join one team with the purpose of helping us create great teams to get things done in the best way possible. Where did we start from? The producer role had been introduced at Zalando to provide a team with whatever it lacked at a certain moment in time, be it a roadmap, team building, process improvement, documentation or even testing.

Generalist 40

Generalist Retail Recruitment BI 40

Rockset

OCTOBER 30, 2018

Prior to Rockset, I spent eight years at Facebook building out their big data infrastructure and online data infrastructure. All the software we wrote was deployed in Facebook's private data centers, so it was not till I started building on the public cloud that I fully appreciated its true potential. Facebook may be the very definition of a web-scale company, but getting hardware still required huge lead times and extensive capacity planning.

Cloud 40

Cloud IT MongoDB Hadoop 40

Speaker: Anne Steiner and David Laribee

As a concept, Developer Experience (DX) has gained significant attention in the tech industry. It emphasizes engineers’ efficiency and satisfaction during the product development process. As product managers, we need to understand how a good DX can contribute not only to the well-being of our development teams but also to the broader objectives of product success and customer satisfaction.

Engineering

Netflix Tech

OCTOBER 31, 2018

Netflix Media Database?—?the Media Timeline Data Model In the previous post in this series, we described some important Netflix business needs as well as traits of the media data system?—?called “ N etflix M edia D ata B ase” (NMDB) that is used to address them. The curious reader might have noticed that a majority of these characteristics relate to properties of the data managed by NMDB.

Media 54

Media Metadata Data MongoDB 54

Data Engineering Podcast

OCTOBER 14, 2018

Summary With the growth of the Hadoop ecosystem came a proliferation of implementations for the Hive table format. Unfortunately, with no formal specification, each project works slightly different which increases the difficulty of integration across systems. The Hive format is also built with the assumptions of a local filesystem which results in painful edge cases when leveraging cloud object storage for a data lake.

Data Lake 100

Data Lake Big Data Cloud Hadoop 100

Uber Engineering

OCTOBER 30, 2018

Cluster management, a common software infrastructure among technology companies, aggregates compute resources from a collection of physical hosts into a shared resource pool, amplifying compute power and allowing for the flexible use of data center hardware. At Uber, cluster management … The post Peloton: Uber’s Unified Resource Scheduler for Diverse Cluster Workloads appeared first on Uber Engineering Blog.

Management 64

Management Engineering Technology MySQL 64

Cloudera

OCTOBER 30, 2018

Federated Learning is a technology that allows you to build machine learning systems when your datacenter can’t get direct access to model training data. The data remains in its original location, which helps to ensure privacy and reduces communication costs. Privacy and reduced communication make federated learning a great fit for smartphones and edge hardware, healthcare and other privacy-sensitive use cases, and industrial applications such as predictive maintenance.

Machine Learning 40

Machine Learning Healthcare Manufacturing Accessible 40

Speaker: Aarushi Kansal, AI Leader & Author and Tony Karrer, Founder & CTO at Aggregage

Software leaders who are building applications based on Large Language Models (LLMs) often find it a challenge to achieve reliability. It’s no surprise given the non-deterministic nature of LLMs. To effectively create reliable LLM-based (often with RAG) applications, extensive testing and evaluation processes are crucial. This often ends up involving meticulous adjustments to prompts.

Building

Zalando Engineering

OCTOBER 3, 2018

Essential building blocks for strong leadership that enables people to grow and achieve results The story of how I ended up working for Zalando in Berlin starts with a LinkedIn message from Joseph Wilkinson, one of our tech recruiters. In tech, we get a lot of messages on LinkedIn, but this one was different and made me very interested to know more about Zalando.

Recruitment 40

Recruitment Building Engineering Designing 40

Data Engineering Podcast

OCTOBER 9, 2018

Summary One of the most complex aspects of managing data for analytical workloads is moving it from a transactional database into the data warehouse. What if you didn’t have to do that at all? MemSQL is a distributed database built to support concurrent use by transactional, application oriented, and analytical, high volume, workloads on the same hardware.

PostgreSQL 100

PostgreSQL BI Data Warehouse Machine Learning 100

Zalando Engineering

OCTOBER 31, 2018

Why I advocate a practice of no estimates as a software engineer Before I get to the topic, I would like to clarify one thing: I don’t want to ban estimations generally from software development, as there are good and solid reasons for it. In a nutshell, business needs to be predictable. I want to show a software developer's view on how to reduce or even get rid of endless estimations meetings with doubtful outcomes.

Software Engineer 52

Software Engineer Software Engineering Coding Building 52

Cloudera

OCTOBER 30, 2018

451 Research Analyst Sheryl Kingstone, and Cloudera’s Steve Totman recently discussed how a growing number of organizations are replacing legacy Customer 360 systems with Customer Insights Platforms ( watch the replay here ). In this blog post, Sheryl outlines how next-gen CIP applications are delivering a better customer experience, and why businesses are relying on CIPs as their preferred path to customer insights.

Unstructured Data 42

Unstructured Data Data Lake Algorithm Machine Learning 42

Advertisement

Are you trying to decide which entity resolution capabilities you need? It can be confusing to determine which features are most important for your project. And sometimes key features are overlooked. Get the Entity Resolution Evaluation Checklist to make sure you’ve thought of everything to make your project a success! The list was created by Senzing’s team of leading entity resolution experts, based on their real-world experience.

Project

Zalando Engineering

OCTOBER 17, 2018

The six month journey of the customer inbox multi-disciplinary team The customer inbox multi-disciplinary area operates in the Fashion Store pillar of the Zalando platform organization. The purpose of the Customer Inbox Unit is to serve customers personal and practical fashion messages, through multiple channels, i.e. “Target the customers at the right time, at the right place.

Consulting 40

Consulting Management Engineering Project 40