Start Data Engineering

AUGUST 29, 2021

1. Introduction 2. Understanding your data engineering task 2.1. Data infrastructure overview 2.2. What exactly 2.3. Why exactly 2.4. Current state 2.5. Downstream impact 3. Delivering your data engineering task 3.1. How 3.2. Breakdown into sub-tasks 3.3. Delivering the finished task 4. Conclusion 5. Further reading 1. Introduction Congratulations! You are given a quick overview of the business and data architecture and are assigned your very first data engineering task.

Data Engineering 130

Data Engineering Data Engineer Engineering Data Architecture 130

Uber Engineering

SEPTEMBER 2, 2021

Introduction. Uber’s GSS (Global Scaled Solutions) team runs scaled programs for diverse products and businesses, including but not limited to Eats, Rides, and Freight. The team transforms Uber’s ideas into agile, global solutions by designing and implementing scalable solutions. One … The post Streaming Real-Time Analytics with Redis, AWS Fargate, and Dash Framework appeared first on Uber Engineering Blog.

AWS 144

AWS Programming Engineering Designing 144

Confluent

AUGUST 31, 2021

Today, as part of our expanded partnership with Elastic, we are announcing an update to the fully managed Elasticsearch Sink Connector in Confluent Cloud. This update allows you to take […].

Cloud 120

Cloud Data Management Kafka 120

Cloudera

SEPTEMBER 1, 2021

The more an enterprise wants to know about itself and its business prospects, the more data it needs to collect and analyze. Additionally, the more data it collects and stores, the better its ability to know customers, to find new ones, and to provide more of what they want to buy. Sounds simple, but a surprising majority of U.S. companies (about two-thirds, according to CIO.com ) are only now getting tuned in to become fully functioning data-driven enterprises by starting new initiatives, scali

Hadoop 96

Hadoop Data Utilities Consulting 96

Advertisement

Many organizations today are unlocking the power of their data by using graph databases to feed downstream analytics, enahance visualizations, and more. Yet, when different graph nodes represent the same entity, graphs get messy. Watch this essential video with Senzing CEO Jeff Jonas on how adding entity resolution to a graph database condenses network graphs to improve analytics and save your analysts time.

Database

Netflix Tech

AUGUST 30, 2021

By Benson Ma , Alok Ahuja Introduction At Netflix, hundreds of different device types, from streaming sticks to smart TVs, are tested every day through automation to ensure that new software releases continue to deliver the quality of the Netflix experience that our customers enjoy. In addition, Netflix continuously works with its partners (such as Roku, Samsung, LG, Amazon) to port the Netflix SDK to their new and upcoming devices (TVs, smart boxes, etc), to ensure the quality bar is reached be

Management 85

Management Kafka Transportation Cloud 85

Teradata

AUGUST 29, 2021

Accelerating drug discovery and development with Teradata Vantage on AWS.

Pharmaceutical 98

Pharmaceutical AWS 98

Cloudera

SEPTEMBER 2, 2021

The shift to cloud has been accelerating, and with it, a push to modernize data pipelines that fuel key applications. That is why cloud native solutions which take advantage of the capabilities such as disaggregated storage & compute, elasticity, and containerization are more paramount than ever. At Cloudera, we introduced Cloudera Data Engineering (CDE) as part of our Enterprise Data Cloud product — Cloudera Data Platform (CDP) — to meet these challenges.

Data Engineering 82

Data Engineering Data Engineer Engineering Utilities 82

Grouparoo

SEPTEMBER 2, 2021

The newest release of Grouparoo has a few updates that make working with data easier. Staying sync with your data warehouse. If rows are deleted in your data warehouse, then Grouparoo profiles get deleted. Combine or use logic to make profile properties. Use code to re-mix your data and get the perfect formats. New destinations: Mixpanel, Mailjet Profile deletion Data systems are often quite good at ingesting new data, but things get complicated when it gets deleted.

Data Warehouse 52

Data Warehouse Data Pipeline Database Systems 52

Datakin

SEPTEMBER 2, 2021

Blog Data Quality + Data Lineage = Written by Peter Hicks on Sep 2, 2021 In a prior life, I dwelled in the day-to-day cycles of an e-commerce platform. I worked with a quite generalized system with orders, products, variants, SKUs, and customers that pined for every discount they could come by. The system built around the core business schema was the kind of chaos that data engineers are all too familiar with; large volumes of clickstream data, etl_warehouses, read replicas, and machine learning

Bytes 52

Bytes Food Datasets Data Pipeline 52

Advancing Analytics: Data Engineering

SEPTEMBER 2, 2021

In late 2020, Databricks introduced Databricks Labs a collection of Terraform Providers that gives you the ability to deploy nearly all Databricks resources onto Azure and Amazon Web Services (AWS) cloud platforms. Meaning you can deploy Databricks workspace, clusters, secrets, libraries, notebooks and automated jobs (and many more) at the time of provisioning the infrastructure, making it easy to manage and configure Databricks.

Amazon Web Services 52

Amazon Web Services AWS Coding Cloud 52

Speaker: Scott Sehlhorst

We know we want to create products which our customers find to be valuable. Whether we label it as customer-centric or product-led depends on how long we've been doing product management. There are three challenges we face when doing this. The obvious challenge is figuring out what our users need; the non-obvious challenges are in creating a shared understanding of those needs and in sensing if what we're doing is meeting those needs.

Certification

Cloudera

AUGUST 31, 2021

Cloudera and Accenture demonstrate strength in their relationship with an accelerator called the Smart Data Transition Toolkit for migration of legacy data warehouses into Cloudera Data Platform. Accenture’s Smart Data Transition Toolkit . Data warehousing is the backbone of every data driven organization , providing mission critical analytics. Today, modern data warehousing has evolved to meet the intensive demands of the newest analytics required for a business to be data driven.

Data Warehouse 82

Data Warehouse Database-centric Metadata Cloud 82

Monte Carlo

AUGUST 31, 2021

Incident management isn’t just for software engineers. With the rise of data platforms and the data-as-a-product mentality, building more reliable processes and workflows to handle data quality has emerged as a top concern for data engineers. In a previous post , we discussed how to set up automatic detection and alerting for bad data; now, guest author Glen Willis shares how the best data teams handle triaging and severity assessment for your broken data pipelines with the help of an emerging r

Data Pipeline 52

Data Pipeline Software Engineer Software Engineering Data 52

Rockset

AUGUST 30, 2021

Until Now: The Slow Crawl from Batch to Real-Time Analytics The world is moving from batch to real-time analytics but it's been at a crawl. Apache Kafka has made acquiring real-time data more mainstream, but only a small sliver are turning batch analytics, run nightly, into real-time analytical dashboards with alerts and automatic anomaly detection.

SQL 52

SQL Kafka MongoDB MySQL 52

Teradata

AUGUST 29, 2021

Data-driven services can help merchant acquirers add value to their core capabilities. However, to succeed, they need to be armed with the necessary data governance capabilities & know-how.

Data Governance 52

Data Governance Government Data 52

Speaker: Timothy Chan, PhD., Head of Data Science

Are you ready to move beyond the basics and take a deep dive into the cutting-edge techniques that are reshaping the landscape of experimentation? 🌐 From Sequential Testing to Multi-Armed Bandits, Switchback Experiments to Stratified Sampling, Timothy Chan, Data Science Lead, is here to unravel the mysteries of these powerful methodologies that are revolutionizing how we approach testing.

Data Science

Pipeline Data Engineering

AUGUST 28, 2021

Data engineering salon. News and interesting reads about the world of data. Building a data team at a mid-stage startup: a short story Erik Bernhardsson, Working on something, "Bernco" The data culture is driven both from above (the CEO pushing for it) as well as from below (people in the trenches). It's OK to fail if at least you learned something from it.

SQL 52

SQL Data Data Engineering Data Engineer 52

Eventbrite Engineering

SEPTEMBER 1, 2021

In my two most recent blog posts, I talked about how to write a Long-Term Technical Vision and a Golden Path. These are future-looking and high-level artifacts so the question I keep hearing is: do I need to give up coding to grow in my career and become a Technical Fellow? In this post I will … Continue reading "A day in the life of a Technical Fellow" The post A day in the life of a Technical Fellow appeared first on Engineering Blog.

Coding 40

Coding Engineering 40

Zalando Engineering

SEPTEMBER 1, 2021

Long time readers of this blog will remember that back in 2019, we published a feature on the benefits of rotating engineers between teams. For those of you who have not seen it, the article described an initiative that aimed to establish cross-functional knowledge sharing, encourage cross team collaboration, and bring greater product awareness, by providing engineers with an opportunity to work on different teams within our Developer Productivity department.

Engineering 40

Engineering Architecture Process Systems 40

Teradata

AUGUST 31, 2021

A core finance foundation, supported by the right data management tools, creates a trusted, auditable, and traceable source of all things financial. Read more.

Building 52

Building Finance Data Management Management 52

Speaker: Aarushi Kansal, AI Leader & Author and Tony Karrer, Founder & CTO at Aggregage

Software leaders who are building applications based on Large Language Models (LLMs) often find it a challenge to achieve reliability. It’s no surprise given the non-deterministic nature of LLMs. To effectively create reliable LLM-based (often with RAG) applications, extensive testing and evaluation processes are crucial. This often ends up involving meticulous adjustments to prompts.

Building

RudderStack

AUGUST 30, 2021

Learn to use dbt & RudderStack Reverse ETL to leverage the power of your data warehouse to sync enriched users, audiences, and other data to downstream tools.

SQL 40

SQL Data Warehouse Data 40

ProjectPro

AUGUST 28, 2021

The most trusted way to learn and master the art of machine learning is to practice hands-on projects. Projects help you create a strong foundation of various machine learning algorithms and strengthen your resume. But as the saying goes the voyage of a thousand miles starts with a single footstep, we present to you a 50 first steps guide on your machine learning journey.

Portfolio 52

Portfolio Project Datasets Algorithm 52

Netflix Tech

SEPTEMBER 3, 2021

By Alex Borysov , Ricky Gardiner Background At Netflix, we heavily use gRPC for the purpose of backend to backend communication. When we process a request it is often beneficial to know which fields the caller is interested in and which ones they ignore. Some response fields can be expensive to compute, some fields can require remote calls to other services.

Designing 98

Designing Java Bytes Utilities 98

Data Engineering Podcast

SEPTEMBER 3, 2021

Summary The term "data platform" gets thrown around a lot, but have you stopped to think about what it actually means for you and your organization? In this episode Lior Gavish, Lior Solomon, and Atul Gupte share their view of what it means to have a data platform, discuss their experiences building them at various companies, and provide advice on how to treat them like a software product.

Designing 100

Designing Building SQL BI 100

Advertisement

Are you trying to decide which entity resolution capabilities you need? It can be confusing to determine which features are most important for your project. And sometimes key features are overlooked. Get the Entity Resolution Evaluation Checklist to make sure you’ve thought of everything to make your project a success! The list was created by Senzing’s team of leading entity resolution experts, based on their real-world experience.

Project

RudderStack

AUGUST 30, 2021

See how you can leverage RudderStack to easily track Marketo form submissions without disrupting Marketo or your marketing team.

Data 40

Data 40

Dataquest

SEPTEMBER 3, 2021

Meet Gino Parages, a former sales and IT business analyst with no coding skills who decided it was time to learn coding to give his career a boost. He chose Dataquest to help him achieve his learning goals and land the job he wanted. Here’s his story… Q: First, what are your preferred pronouns? A: He/him Q: All right, Gino! What’s your current job title?

Business Analyst 52

Business Analyst Python Data Science Healthcare 52

RudderStack

SEPTEMBER 1, 2021

This post explores how RudderStack’s warehouse-first approach separates it from the traditional marketing CDP.

Data Warehouse 40

Data Warehouse Data IT 40

RudderStack

SEPTEMBER 2, 2021

This post explores three interesting ways you can use RudderStack to unlock the power of machine learning.

Machine Learning 40

Machine Learning 40

Speaker: Anne Steiner and David Laribee

As a concept, Developer Experience (DX) has gained significant attention in the tech industry. It emphasizes engineers’ efficiency and satisfaction during the product development process. As product managers, we need to understand how a good DX can contribute not only to the well-being of our development teams but also to the broader objectives of product success and customer satisfaction.

Engineering

Data Engineering Podcast

SEPTEMBER 1, 2021

Summary The Presto project has become the de facto option for building scalable open source analytics in SQL for the data lake. In recent months the community has focused their efforts on making it the fastest possible option for running your analytics in the cloud. In this episode Dipti Borkar discusses the work that she and her team are doing at Ahana to simplify the work of running your own PrestoDB environment in the cloud.

Data Lake 100

Data Lake Cloud AWS SQL 100