Zalando Engineering

NOVEMBER 19, 2023

What this article is about What kind of changes we had to make to the codebase How we did the actual upgrade What challenges we faced How we did the data transfer How the data was kept in sync What this article is not A step-by-step guide on how to upgrade Elasticsearch (read on to find out why).

Scala 86

Scala Java Coding IT 86

dbt Developer Hub

DECEMBER 19, 2023

Overview ​ Over the course of my three years running the Partner Engineering team at dbt Labs, the most common question I've been asked is, How do we integrate with dbt? This also allows us to skip the intro and get to the fun conversations so much faster, like what a joint solution for our customers would look like.

Metadata 52

Metadata Cloud Accessible Accessibility 52

AltexSoft

SEPTEMBER 13, 2023

If you’re an engineer or a decision-maker at a company planning to add generative AI features to its applications, the prompts you use are crucial. This makes understanding how to create good prompts, also known as prompt engineering , very important for your projects. What is prompt engineering and why does it matter?

Engineering 59

Engineering Hospitality Transportation Algorithm 59

Netflix Tech

MAY 19, 2023

Authorizing these identities is done not only by application name, but by specifying specific attributes on which to match. At authZ check time, provide the attributes for the identity to check, e.g. “can app bar in us-west-2 access this document.” What was problematic about this design? This is a Caveated SpiceDB relationship.

PostgreSQL 99

PostgreSQL Systems Accessible Accessibility 99

Knowledge Hut

DECEMBER 26, 2023

Data Engineering is typically a software engineering role that focuses deeply on data – namely, data workflows, data pipelines, and the ETL (Extract, Transform, Load) process. According to reports by DICE Insights, the job of a Data Engineer is considered the top job in the technology industry in the third quarter of 2020.

Data Engineering 98

Data Engineering Data Engineer Engineering Pipeline-centric 98

Towards Data Science

JULY 13, 2023

Building a maintainable and modular LLM application stack with Hamilton in 13 minutes LLM Applications are dataflows, use a tool specifically designed to express them LLM stacks. Hamilton is great for describing any type of dataflow , which is exactly what you’re doing when building an LLM powered application. Stack overview.

Building 74

Building Database-centric Database Coding 74

Scott Logic

MAY 7, 2024

Test Harness App With that sorted, attention turned to what our test harness app would actually look like. Admittedly it wasn’t very robust and probably (definitely) not up to best practices but each deployment would only last for a couple of hours at most. This blog explains how we got around it.

Cloud 52

Cloud Java AWS Coding 52

Maxime Beauchemin

JANUARY 20, 2017

I joined Facebook in 2011 as a business intelligence engineer. By the time I left in 2013, I was a data engineer. We were data engineers! Data Engineering? At the same time, data engineering was the slightly younger sibling, but it was going through something similar. I wasn’t promoted or assigned to this new role.

Data Engineering 130

Data Engineering Data Engineer Engineering ETL Tools 130

Knowledge Hut

DECEMBER 29, 2023

Whatever you see when you visit a website - the different types of buttons and other UI components, media, texts, forms, animations, etc. Please also take a look at the details on what is markdown here. A Front-End Developer is a type of Software Engineer who works on front-end development. are all added using HTML.

Portfolio 98

Portfolio Amazon Web Services Programming Language Coding 98

Knowledge Hut

DECEMBER 22, 2023

But first, let us see what code editors and IDEs are. What is a Code Editor? What is Python IDE? Over the years, Python language has evolved enormously with the contribution of developers. Python is one of the most popular programming languages. A code editor is built for editing and modifying source code.

Python 97

Python Coding Programming Language Data Science 97

AltexSoft

JUNE 26, 2023

What is data collection? Data collection is a methodical practice aimed at acquiring meaningful information to build a consistent and complete dataset for a specific business purpose — such as decision-making, answering research questions, or strategic planning. Data engineering explained in 14 minutes.

Data Collection 52

Data Collection Machine Learning Unstructured Data Non-relational Database 52

Towards Data Science

MAY 2, 2023

This post is part of my recent effort in bringing more cloud-oriented data engineering posts. For example, you can access the AWS UI on your browser, search for S3 and create a new bucket manually, or write a code in Python to create this same instance making a request on the AWS API. Definitely an unpleasant process.

Data Pipeline 74

Data Pipeline Management AWS Data 74

AltexSoft

OCTOBER 21, 2022

Kafka can continue the list of brand names that became generic terms for the entire type of technology. In this article, we’ll explain why businesses choose Kafka and what problems they face when using it. What is Kafka? What does the high-performance data project have to do with the real Franz Kafka’s heritage?

Kafka 93

Kafka Hadoop ETL Tools Big Data 93

Tweag

MAY 24, 2023

Tweagers have an engineering mantra — Functional. I, Test It’s hardly a revelation that testing the code we write — and automating that process — is an essential element of software engineering practice. In a dynamically typed language, like Python, it can be tempting to just throw tests together and call it a day!

Python 104

Python Programming Language Programming Coding 104

Snowflake

APRIL 19, 2023

Welcome to the third blog post in our series highlighting Snowflake’s data ingestion capabilities, covering the latest on Snowpipe Streaming (currently in public preview) and how streaming ingestion can accelerate data engineering on Snowflake. What is Snowpipe Streaming?

Data Ingestion 57

Data Ingestion Kafka Java Data Pipeline 57

Zalando Engineering

MAY 13, 2024

In the past we encountered numerous challenges, including hidden CSS overrides, local conditional logic, debatable API additions, and duplicated implementations. This article will focus instead on the challenges encountered in the development of theming capabilities.

Designing 62

Designing Systems Architecture Coding 62

dbt Developer Hub

JULY 17, 2023

Whether you are creating your pipelines into dbt for the first time or just adding a new model once in a while, good documentation and testing should always be a priority for you and your team. So in this post I am going to help you 10x your documentation and testing speed by using ChatGPT to do most of the leg work for you.

Coding 52

Coding Database Programming Process 52

ProjectPro

FEBRUARY 8, 2023

For data engineers and data scientists , AWS Glue is a powerful data integration tool that enables them to streamline the ETL process and minimize the amount of manual effort involved in data integration activities. Table of Contents What is AWS Glue? Do ETL and data integration activities seem complex to you? billion by 2026?

AWS 98

AWS Scala Metadata Data Lake 98

Monte Carlo

FEBRUARY 8, 2024

Data observability has been one of the hottest emerging data engineering technologies the last several years. This momentum shows no signs of stopping with data quality and reliability becoming a central topic in the data product and AI conversations taking place across organizations of all types and sizes.

BI 59

BI Data Pipeline Data Warehouse Data 59

AltexSoft

MAY 12, 2023

We will discuss the different data types, storage and management options, and various techniques and tools for unstructured data analysis. What is unstructured data? Definition and examples Unstructured data , in its simplest form, refers to any data that does not have a pre-defined structure or organization.

Unstructured Data 59

Unstructured Data NoSQL Hadoop Data Lake 59

AltexSoft

NOVEMBER 1, 2023

This article explains what named entity recognition is, its working principles, and how it is used in real life. What is Named Entity Recognition (NER)? Named entity recognition (NER) is a subfield within natural language processing (NLP) that focuses on identifying and classifying specific data points from textual content.

Deep Learning 52

Deep Learning Machine Learning Datasets Algorithm 52

DoorDash Engineering

MARCH 7, 2023

While most engineering tooling at DoorDash is focused on making safe incremental improvements to existing systems, in part by testing in production (learn more about our end-to-end testing strategy ), this is not always the best approach when launching an entirely new business line. Additionally, some require a paid subscription.

AWS 119

AWS PostgreSQL Database SQL 119

AltexSoft

NOVEMBER 17, 2021

The problem of document classification pertains to the library, information, and computer sciences. In this article, we’ll explore the essence of document classification, and study the main approaches to categorizing files based on their content. What is document classification? Read our separate article on that.

Machine Learning 75

Machine Learning Insurance Medical Healthcare 75

Confluent

MAY 21, 2019

At first, you may think to use REST APIs—most programming languages have frameworks that make it very easy to implement REST APIs, so this is a common first choice. REST APIs define the HTTP methods that are used and the request and response payloads that are expected. What will the profile service publish?

Kafka 110

Kafka Insurance Architecture Database 110

Monte Carlo

MARCH 28, 2024

What does that mean? Table of Contents What’s the difference between data-as-a-product and data-as-a-service? What are the components of data-as-a-product? What is data-as-a-product? What’s the difference between data-as-a-product and data-as-a-service? What are the components of data-as-a-product?

Data 52

Data High Quality Data Machine Learning Accessible 52

Scott Logic

MAY 31, 2023

For a while now I’ve been keeping a notebook where I record brief definitions of these new terms as I encounter them. Model architecture - complex AI models often have various discrete components, each of which are a neural network, that have a specific function. Model size is often measured in terms of the number of parameters it has.

Datasets 59

Datasets Media Architecture Finance 59

Knowledge Hut

JANUARY 30, 2024

What is data extraction? Once stored in the destination repository, the data can undergo further transformations, such as data cleansing, feature engineering, statistical analysis, and machine learning. What is the purpose of extracting data?

ETL Tools 52

ETL Tools Database-centric Data Mining Data Cleanse 52

Knowledge Hut

FEBRUARY 7, 2023

What is Kubernetes? What Does Kubernetes Cheat Sheet Mean? A Kubernetes Cheat Sheet is a document that helps developers to understand the Kubernetes concepts. Kubernetes is an open-source system for automating containerized applications' deployment, scaling, and management.

Architecture 98

Architecture Accessible Accessibility Management 98

Snowflake

AUGUST 16, 2023

At Snowflake Summit, we announced a wave of product innovations: Snowpark ML Modeling API, Snowflake Native App Framework, Dynamic Tables and more. What remains challenging is how streaming data is brought together with batch data. Visit Snowflake Documentation for more details. If you missed out, not to worry! Learn more here.

Scala 52

Scala Transportation Kafka Data Lake 52

Lyft Engineering

JANUARY 30, 2023

These model-based decisions include price optimization for rides, incentives allocation for drivers, fraud detection, ETA prediction, and innumerable others that impact how riders move and drivers earn. Making real-time inferences with machine learning (ML) models at scale is complex. Constraints imposed by our legacy system.

Machine Learning 92

Machine Learning Systems Python Coding 92

Monte Carlo

APRIL 16, 2022

Whether or not you agree with Chad’s approach detailed below, what can’t be disputed is how his opinions have generated a tremendous amount of debate. Whether or not you agree with Chad’s approach detailed below, what can’t be disputed is how his opinions have generated a tremendous amount of debate. Treating data like an API.

Data Warehouse 52

Data Warehouse Architecture Data Government 52

Ascend.io

JANUARY 3, 2023

This approach should improve the full data lifecycle, including knowing what data exists, where it is, how to access it, and how to understand it. Table of Contents What Are Data Products? Interpolating between these views and talking to our customers, we have arrived at a useful definition that lends itself to action.

Data 52

Data Data Lake Business Intelligence Big Data 52

Rock the JVM

SEPTEMBER 3, 2023

Introduction One of the simplest and well documented ways to build a web API is to follow the REST paradigm. REST APIs provide a simple and uniform way to access data and not only through URLs, across the web. In this article we will try to develop a basic skeleton for a REST API using Play and Scala.

Scala 52

Scala Database Project Coding 52

DareData

NOVEMBER 28, 2023

These practices and methodologies are commonly known as MLOps, short for Machine Learning Operations and they bridge the gap between data science and software engineering, ensuring the pillars of experimentation: reproducibility, performance, scalability and monitorization. But that is not all!

Data Science 52

Data Science Medical Machine Learning Data 52

Knowledge Hut

MAY 2, 2024

As per Apache, “ Apache Spark is a unified analytics engine for large-scale data processing ” Spark is a cluster computing framework, somewhat similar to MapReduce but has a lot more capabilities, features, speed and provides APIs for developers in many languages like Scala, Python, Java and R. billion (2019 - 2022).

Scala 52

Scala Hospitality Healthcare Retail 52

Monte Carlo

JANUARY 9, 2024

The change in high-level metrics set off alarm bells, summoning their colleagues on product and engineering teams to run to the rescue and determine what was going wrong with the customer experience. I think most people recognize that having some type of API or expectation on the data is very useful. “I

Pipeline-centric 52

Pipeline-centric Database-centric Data Manufacturing 52

Rockset

MAY 17, 2022

That changed when NoSQL databases such as key-value and document stores came on the scene. Now companies could take a massive data set, organize it into simple pairs of key values or documents and instantly perform lookups and other simple queries. Complex SQL queries have long been commonplace in business intelligence (BI).

SQL 52

SQL NoSQL Hadoop MongoDB 52