Data Engineering Podcast

JANUARY 14, 2018

Summary As we scale our systems to handle larger volumes of data, geographically distributed users, and varied data sources the requirement to distribute the computational resources for managing that information becomes more pronounced. In order to ensure that all of the distributed nodes in our systems agree with each other we need to build mechanisms to properly handle replication of data and conflict resolution.

Data Pipeline 100

Data Pipeline Data Engineering Data Engineer Project 100

U-Next

JANUARY 9, 2018

Success in today’s businesses has taken several meanings. Apart from just pay hikes and promotions, success has gotten new dimensions that have been of very recent origins. Today, success has become synonymous with happiness at a workplace, challenging tasks, compensatory rewards, incentives, authoritative job profiles, influential role, and more. The current talent pools in organizations have become wiser and more mature than their previous generation counterparts.

Recruitment 73

Recruitment Technology Management Process 73

Cloudera

JANUARY 12, 2018

I mentioned in an earlier blog titled, “Staffing your big data team, ” that data engineers are critical to a successful data journey. That said, most companies that are early in their journey lack a dedicated engineering group. And the longer it takes to put a team in place, the likelier it is that your big data project will stall. The data engineering team is responsible for collecting and ingesting batch and stream-oriented data, inventorying the data, working through ingest bottlenecks, and d

Big Data 54

Big Data Data Engineering Data Engineer Engineering 54

Zalando Engineering

JANUARY 24, 2018

How Zalando helps its engineering teams navigate the tech landscape Zalando has more than 200 engineering teams, which regularly face tricky technology choices. To help them make good decisions, we created the Zalando Tech Radar as a "navigation" tool. Inspired by ThoughtWorks , it assigns each technology to one of four rings — Adopt, Trial, Assess and Hold — which represents the current consensus within Zalando.

Building 40

Building Technology Utilities Coding 40

Advertisement

Many organizations today are unlocking the power of their data by using graph databases to feed downstream analytics, enahance visualizations, and more. Yet, when different graph nodes represent the same entity, graphs get messy. Watch this essential video with Senzing CEO Jeff Jonas on how adding entity resolution to a graph database condenses network graphs to improve analytics and save your analysts time.

Database

Maxime Beauchemin

JANUARY 7, 2018

Batch data processing  — historically known as ETL —  is extremely challenging. It’s time-consuming, brittle, and often unrewarding. Not only that, it’s hard to operate, evolve, and troubleshoot. In this post, we’ll explore how applying the functional programming paradigm to data engineering can bring a lot of clarity to the process. This post distills fragments of wisdom accumulated while working at Yahoo, Facebook, Airbnb and Lyft, with the perspective of well over a decade of data warehousing

Data Engineering 183

Data Engineering Data Engineer Data Process Process 183

Dataquest

JANUARY 10, 2018

In previous blog posts , we have described the Postgres database and ways to interact with it using Python. Those posts provided the basics, but if you want to work with databases in production systems, then it is necessary to know how to make your queries faster and more efficient. To understand what efficiency means in Postgres, it’s important to learn how Postgres works under the hood.

Building 52

Building PostgreSQL Database Python 52

ProjectPro

JANUARY 2, 2018

News on Hadoop - December 2017 Apache Impala gets top-level status as open source Hadoop tool.TechTarget.com, December 1, 2017. The massively parallel processing engine born at Cloudera acquired the status of a top-level project within the Apache Foundation. The main objective of Impala is to provide SQL-like interactivity to big data analytics just like other big data tools - Hive, Spark SQL, Drill, HAWQ , Presto and others.

Hadoop 52

Hadoop Big Data Datasets Machine Learning 52

Cloudera

JANUARY 19, 2018

The New Year is a great time to make resolutions, but it’s also a great time to reflect on the previous year. Before we get too far into 2018, let’s take a look at the ten most popular Cloudera VISION blogs from 2017. Today is an important day in the life of Cloudera. On April 28, 2017, Mike Olson , as one of the founders of Cloudera, writes about the initial public offering, and what the milestone means.

Insurance 45

Insurance Machine Learning Data Science Big Data 45

Zalando Engineering

JANUARY 8, 2018

Leveraging the full power of a functional programming language In Zalando Dublin, you will find that most engineering teams are writing their applications using Scala. We will try to explain why that is the case and the reasons we love Scala. This content is coming both from my own experience and the team I'm working with in building the new Zalando Customer Data Platform.

Scala 40

Scala Bytes Java Programming 40

Cloudera

JANUARY 8, 2018

As the GDPR clock ticks down to implementation, it is clear that this will not be a non-event like the Millennium Bug – it will happen and there will be dire consequences, potentially company-closures, in the event of non-compliance. The three certainties in life: death, taxes and GDPR. 1999 was a milestone year for the development of technology.

Relational Database 41

Relational Database Government Data Warehouse Big Data 41

Speaker: Scott Sehlhorst

We know we want to create products which our customers find to be valuable. Whether we label it as customer-centric or product-led depends on how long we've been doing product management. There are three challenges we face when doing this. The obvious challenge is figuring out what our users need; the non-obvious challenges are in creating a shared understanding of those needs and in sensing if what we're doing is meeting those needs.

Certification

Cloudera

JANUARY 9, 2018

This was an amazing year for our inaugural “Cybersecurity On Call” season. It was truly an honor hosting amazing guests as we explored the world of cybersecurity. From industry thought leaders, to New York Times best sellers, to hackers, I learned a ton about the future of cybersecurity and I hope you did as well. Today’s episode won’t be our usual programming, today is our end of the year special where we will dive into our top five tips from this year’s season.

Government 40

Government Programming IT Management 40

Cloudera

JANUARY 29, 2018

To quote Sam Walton, Walmart’s founder, “There is only one boss. The customer. And he can fire everybody in the company from the chairman on down, simply by spending his money somewhere else”. This very much forms the lens for our focus here at Cloudera Asia Pacific. And it is this unwavering passion and commitment that has driven the team to strive for the very best for our customers and partners, and milestones that we have collectively attained since 2015.

Cloud 40

Cloud Banking Big Data Skills Big Data 40

Data Engineering Podcast

JANUARY 28, 2018

Summary Sharing data across multiple computers, particularly when it is large and changing, is a difficult problem to solve. In order to provide a simpler way to distribute and version data sets among collaborators the Dat Project was created. In this episode Danielle Robinson and Joe Hand explain how the project got started, how it functions, and some of the many ways that it can be used.

Data 100

Data Project Electronics Data Management 100

Zalando Engineering

JANUARY 10, 2018

We talk to Han and Kashif from Zalando Research Employer Branding Specialist Data Science, Nana Yamazaki catches up with the team using literal fashion icons in Deep Learning. Tell us about Fashion-MNIST. What did you want to accomplish? Fashion-MNIST is a freely available dataset of Zalando articles that most importantly has the same format as the MNIST dataset.

Deep Learning 52

Deep Learning Datasets Machine Learning Data Science 52

Speaker: Timothy Chan, PhD., Head of Data Science

Are you ready to move beyond the basics and take a deep dive into the cutting-edge techniques that are reshaping the landscape of experimentation? 🌐 From Sequential Testing to Multi-Armed Bandits, Switchback Experiments to Stratified Sampling, Timothy Chan, Data Science Lead, is here to unravel the mysteries of these powerful methodologies that are revolutionizing how we approach testing.

Data Science

Zalando Engineering

JANUARY 31, 2018

How we deployed RabbitMQ on AWS In an effort to move away from our legacy monolithic service, we decided take on the challenge of building a new communication platform based on a micro service architecture, which would be more focused and more easily manageable. The challenge was exciting and big; we had to make crucial decisions early on, decisions that we would have to live with for the foreseeable future.

Cloud 40

Cloud AWS Programming Language Utilities 40

Zalando Engineering

JANUARY 3, 2018

Reducing ops effort while maintaining Kafka and Zookeeper This post is targeted to those looking for ways to reduce ops effort while maintaining Kafka and Zookeeper deployments on AWS and also improving their availability and stability. In a nutshell, we are going to explain how using Elastic Network Interfaces can improve over a straight out of the box setup.

Kafka 40

Kafka AWS Data Ingestion Process 40

Cloudera

JANUARY 3, 2018

Building the right team is as important as assembling the right IT infrastructure – and the needs differ just as dramatically. A traditional BI and analytics organization consists of three main groups: Analysts that develop reports often using sample data. The data management team – modelers that take requests, find data, and develop models to answer the questions.

Big Data 40

Big Data Data Architect Data Science Programming Language 40

Cloudera

JANUARY 11, 2018

I often meet with our customers to help them understand how to connect modern technology to business success. The ever-present question at these encounters is “Where do I start?” For them, they may understand that they need a data-driven strategy or the culture may aim to take a shift to being guided by data. These are often goals set by the executive team with little guidance on how to execute or implement.

Data Warehouse 40

Data Warehouse Telecommunication Media Retail 40

Speaker: Aarushi Kansal, AI Leader & Author and Tony Karrer, Founder & CTO at Aggregage

Software leaders who are building applications based on Large Language Models (LLMs) often find it a challenge to achieve reliability. It’s no surprise given the non-deterministic nature of LLMs. To effectively create reliable LLM-based (often with RAG) applications, extensive testing and evaluation processes are crucial. This often ends up involving meticulous adjustments to prompts.

Building

Zalando Engineering

JANUARY 22, 2018

Building an aggregated view of data in the event-driven microservice architecture In the world of microservices, where a domain model gets decomposed into related, but independently handled entities, we often face the challenge of building an aggregate view of the data that brings together different parts of that model. While this can already be interesting with “traditional” designs, the move to event-driven architectures can magnify these difficulties, especially with simplistic event streams.

Media 40

Media Kafka Architecture Designing 40

Data Engineering Podcast

JANUARY 21, 2018

Summary The majority of the conversation around machine learning and big data pertains to well-structured and cleaned data sets. Unfortunately, that is just a small percentage of the information that is available, so the rest of the sources of knowledge in a company are housed in so-called “Dark Data” sets. In this episode Alex Ratner explains how the work that he and his fellow researchers are doing on Snorkel can be used to extract value by leveraging labeling functions written by

Machine Learning 100

Machine Learning Data Pipeline Big Data Data Collection 100

Zalando Engineering

JANUARY 17, 2018

How to talk about design in the agile world How we improved design communication in the Retail Ops Team With an agile and lean approach, most of us here at Zalando changed the way we build digital products. Design processes also evolved,  with  designers usually working alongside cross-functional product teams. But, at first, one thing did not change too much: how we talk about the design.

Designing 40

Designing Retail Process IT 40