Picnic Engineering

FEBRUARY 6, 2024

Picnic’s infrastructure design focuses on a rapidly scalable cloud solution. We will share why we set up on-premises servers in the first place, why we planned our migration back and how we executed this in a safe and stable fashion. How do you approach planning such a migration? Secondly, you create a plan set-up for success.

Cloud 97

Cloud Transportation Cloud Computing Systems 97

Data Engineering Podcast

DECEMBER 3, 2023

This episode is brought to you by Datafold – a testing automation platform for data engineers that finds data quality issues for every part of your data workflow, from migration to deployment. Datafold has recently launched a 3-in-1 product experience to support accelerated data migrations. Data lakes are notoriously complex.

Systems 130

Systems Designing Data Lake SQL 130

Snowflake

NOVEMBER 30, 2023

We’re extremely familiar with just how tricky a cloud migration can be, especially when it involves moving historical business data. Choosing a migration approach involves balancing cloud strategy, architecture needs and business priorities. How many tables and views will be migrated, and how much raw data?

Data Warehouse 93

Data Warehouse Banking Data Cloud 93

DoorDash Engineering

FEBRUARY 27, 2024

Following the migration to DoorDash’s Search Engine, we saw a 50% p99.9 We also designed the service to be a one-stop solution for all DoorDash teams that need a search engine. Conclusion We spent 2023 implementing the Search Engine and migrating DoorDash to it. latency reduction and a 75% hardware cost decrease.

Engineering 129

Engineering Systems Designing Architecture 129

Seattle Data Guy

SEPTEMBER 2, 2023

SSIS, short for SQL Server Integration Service, is an essential data migration tool for modern businesses. It’s a comprehensive solution to your data management needs. In today’s business landscape,… Read more The post What Is SSIS and Should You Use It?

IT 130

IT Aggregated Data SQL Database 130

Data Engineering Podcast

SEPTEMBER 3, 2023

In this episode Adrian Brudaru explains how it works, the benefits that it provides over other data integration solutions, and how you can start building pipelines today. If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration.

Data Integration 162

Data Integration BI SQL Python 162

Christophe Blefari

OCTOBER 14, 2023

A microservice approach for DAG authoring using datasets — The idea is to apply SE patterns to pipelines like migration , broadcast and aggregate. This time Xebia team propose an usage of dynamic task mapping to do it (link to Github repo with multiple solutions). Migration recipes can apply whatever the software you use.

Python 130

Python Datasets Data Banking 130

Knowledge Hut

MARCH 5, 2024

AWS Migration Challenges and How to Overcome Them AWS Challenge 1: Legacy Application Compatibility Migrating legacy applications to AWS can face compatibility issues due to outdated dependencies and configurations. Solution: Conduct a thorough application assessment, update dependencies, and refactor code where necessary.

AWS 52

AWS Amazon Web Services Cloud Computing PostgreSQL 52

Data Engineering Podcast

NOVEMBER 5, 2023

In this episode Lukas Fittl shares some hard-won wisdom about the causes and solution of many performance bottlenecks and the work that he is doing to shine some light on PostgreSQL to make it easier to understand how to keep it running smoothly. When an application is slow, there is a good probability that the database needs some attention.

PostgreSQL 147

PostgreSQL Data Lake High Quality Data SQL 147

The Pragmatic Engineer

AUGUST 3, 2023

In the future, the Dynamic Links service will shut down, but you will have at least 12 months from the announcement date to migrate. They instead seem to suggest to use other solutions going forward: Android App Links for Android apps iOS Universal Links for iOS apps Google Play Instant and iOS App Clips as a way to preview an app.

Metadata 157

Metadata Engineering Building Technology 157

Engineering at Meta

JANUARY 26, 2023

The technical journey discusses the motivations, challenges, and technical solutions employed for warehouse schematization, especially a change to the wire serialization format employed in Meta’s data platform for data interchange related to Warehouse Analytics Logging. Migrations are hard. Why did we migrate to Tulip?

Bytes 102

Bytes Data Engineering Coding 102

Netflix Tech

MAY 4, 2023

Migrating Critical Traffic At Scale with No Downtime — Part 1 Shyam Gala , Javier Fernandez-Ivern , Anup Rokkam Pratap , Devang Shah Hundreds of millions of customers tune into Netflix every day, expecting an uninterrupted and immersive streaming experience.

Utilities 135

Utilities Systems Architecture Coding 135

Hevo

APRIL 20, 2024

Effective data migration is the key to overcoming the challenges associated with today’s data-driven world. The AWS Aurora Postgres to Databricks integration offers data storage and analytics solutions that help unlock the full potential of your organization’s operational data.

AWS 52

AWS Data Integration Data Storage Database 52

Knowledge Hut

MARCH 28, 2024

As a result, organizations are looking to capitalize on cloud-based data solutions. Job Role 1: Azure Data Engineer Azure Data Engineers develop, deploy, and manage data solutions with Microsoft Azure data services. Develop predictive models and data-driven solutions to address business challenges.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Snowflake

DECEMBER 20, 2023

More than 7,700 customers rely on ServiceNow’s platform and solutions to optimize processes, break down silos and drive business value. Migrating to Snowflake helped ServiceNow democratize data, accelerate innovation and increase efficiency. ServiceNow is focused on making the world work better for everyone.

IT 63

IT Data Government Data Security 63

Precisely

MAY 11, 2023

A host of others have started fresh, implementing SAP’s next-generation cloud-enabled ERP for the first time coupled with SAP Solution Manager. Read our eBook How to Solve the Top 10 Customer Master Data Challenges in SAP Learn more about SAP S/4HANA migration and the benefits of automation with Precisely. Read our free ebook.

Data Cleanse 52

Data Cleanse Management Systems Government 52

Cloudera

SEPTEMBER 26, 2023

Based on a partnership designed to bring IBM’s advanced data and AI solutions to more organizations across the expansive Apache Open Source Database ecosystem, IBM Technology is partnering with Cloudera as our preferred partner for addressing real time data movement built on Cloudera’s Data Flow leveraging Kafka.

Kafka 90

Kafka Technology IT Government 90

Snowflake

JANUARY 24, 2024

Governments face many challenges that could be addressed almost immediately by the effective application of AI solutions accompanied by the use of data management platforms. federal government by SaaS solutions achieving FedRAMP (Federal Risk and Authorization Management Program) certification. This will be enabled in the U.S.

Government 90

Government Data Management Cloud Data 90

Hevo

APRIL 5, 2024

Data analytics involves storing, managing, and processing data from different sources and analyzing it thoroughly to develop solutions for our business problems. Therefore, JSON Snowflake data migration is crucial […]

Data 52

Data Data Analytics Process Management 52

Monte Carlo

NOVEMBER 20, 2023

Far from a mere “copy-paste” job, data migrations can be deceptively tricky and one of the most challenging aspects of a data engineer’s job. But as businesses pivot and technologies advance, data migrations are—regrettably—unavoidable. Table of Contents What makes data migrations complex?

MongoDB 52

MongoDB ETL Tools Database Data 52

Cloudera

JULY 14, 2023

However, as there are already 25 million terabytes of data stored in the Hive table format, migrating existing tables in the Hive table format into the Iceberg table format is necessary for performance and cost. In all the use cases we are trying to migrate a table named “events.” They are meant to be a general guide.

Metadata 70

Metadata Data Warehouse Big Data Ecosystem Java 70

DoorDash Engineering

FEBRUARY 7, 2023

This article will take a deep dive into the steps we took to migrate over to CockroachDB and how we ensured that migration was smooth and successful. How we migrated to CockroachDB: We migrated from the legacy database to CockroachDB in four major milestones. Adding any caching logic will be easier for better performance.

PostgreSQL 119

PostgreSQL Retail SQL Database 119

Pinterest Engineering

NOVEMBER 7, 2023

From dev velocity and service stability improvements to seamless migrations from native client to PSC, we would like to share some of our findings from running a unified PubSub client library in production. We have observed 0 incidents so far with migrations and do not expect this number to increase.

Kafka 98

Kafka Java Software Engineer Software Engineering 98

The Pragmatic Engineer

JUNE 13, 2023

‘I can’t speak for Booking.com, but in some cases I believe companies consider migration to the cloud as an opportunity to “restart” their infrastructure and solve existing scale/capacity issues. So far, we haven’t been able to prove such merit in the migration.

Cloud 189

Cloud Utilities Database BI 189

Knowledge Hut

MARCH 19, 2024

So, let's discuss the AWS cloud migration case study and its importance in getting a better understanding of the topic in detail. They offer vital lessons and inspiration for anyone considering or already using AWS by providing insights into the tactics, solutions, and best practices businesses use the AWS Cloud Engineer program.

AWS 52

AWS Amazon Web Services Hospitality Cloud Computing 52

Knowledge Hut

DECEMBER 26, 2023

PaaS PaaS (Platform as a Service) solution uses third-party service providers to deliver hardware and software tools to enterprises over the internet. Actual SaaS solutions are executed from remote locations across the world on cloud servers. Features: Migrates data across 20 sources Cloud backup and recovery 17.

Cloud Computing 75

Cloud Computing Cloud Amazon Web Services AWS 75

Monte Carlo

OCTOBER 2, 2023

As the team began to look at modernizing their data stack and migrating to the cloud, finding data quality issues at the source was going to become a major focus. Finding a scalable solution to data quality was a top priority for Abacus’s data modernization efforts. It was time for a change — a big one.

Pharmaceutical 72

Pharmaceutical Healthcare Machine Learning Architecture 72

Knowledge Hut

JANUARY 3, 2024

It is said that more businesses around the world will increase their investments in Cloud technology solutions, and there has been a sudden spike in Cloud deployments ever since the COVID-19 pandemic started. Some of the benefits introduced by cloud solutions include data security, flexibility, efficiency, and high performance.

Cloud Computing 52

Cloud Computing Cloud Amazon Web Services Government 52

Knowledge Hut

MARCH 18, 2024

There are already over 100 distinct Azure solutions accessible to customers, ranging from management tools that can give you app analytics that will identify, prioritize, and diagnose any problems in your services to AI machine learning services. Migration Challenges 14. This is the context in which Microsoft Azure is beneficial.

Portfolio 52

Portfolio Cloud Computing Machine Learning Cloud 52

Cloudera

MARCH 22, 2024

That’s why Cloudera and Red Hat , renowned for their open-source solutions, have teamed up to bring Red Hat Enterprise Linux ( RHEL ) to Cloudera on public cloud as the operating system for all of our public cloud platform images. For those transitioning from CentOS Linux (or other distributions), Cloudera will offer an easy migration path.

Cloud 112

Cloud Certification Systems Architecture 112

Snowflake

AUGUST 30, 2023

When using data platforms that don’t support comprehensive cross-cloud replication, customers have to build custom solutions which lead to longer deployment cycles and a significantly higher total cost of ownership. Thus, a quick and efficient cloning process is an important feature to have in any data management solution.

Scala 87

Scala Government Database Cloud 87

Knowledge Hut

NOVEMBER 16, 2023

Once the transition is complete, I hand over the responsibility to a cloud engineer or another team member to maintain the new system and oversee the entire migration process. Job Summary of an Azure Cloud Engineer As an Azure cloud engineer, my responsibilities include planning and deploying ongoing cloud migrations across organizations.

Cloud 52

Cloud Engineering Education Cloud Computing 52

Knowledge Hut

MARCH 22, 2024

AWS solutions architect career opportunities have grown multiplefold. Keeping all that in mind, Let me provide you with some invaluable insights about AWS solutions architect jobs, demand, benefits, industries, and companies, as well as tips to kickstart your career as an AWS Solutions Architect.

AWS 52

AWS Amazon Web Services Cloud Computing Google Cloud 52

Lyft Engineering

NOVEMBER 29, 2023

Streaming solutions require setting up a mutability window per entity (ex. After a careful and deep analysis of cost, infrastructure management, and overall feature benefits, we decided to expand on ClickHouse and sunset Druid, migrating existing Druid use cases to ClickHouse. Overall, our migration experience went smoothly.

Kafka 104

Kafka Data Ingestion Datasets Architecture 104

Snowflake

APRIL 28, 2023

Migrating data warehouse workloads from long-standing legacy systems can be a time-consuming, error-prone, and logistically challenging process. Based on our experience working with thousands of customers on this very journey, we’ve learned that having the right partners can lead to a more successful and streamlined migration process.

Data Warehouse 52

Data Warehouse Consulting Programming Cloud 52

Knowledge Hut

NOVEMBER 16, 2023

You will be introduced to all the basic concepts required to design AWS solutions. AWS Certified Solutions Architect – Associate AWS Solutions Architect certification exam validates your skills in designing distributed applications. It is not a definitive guide containing steps for the migrating process.

AWS 52

AWS Certification Amazon Web Services Machine Learning 52

Snowflake

SEPTEMBER 14, 2023

Unbound by the limitations of a legacy on-premises solution, its multi-cluster shared data architecture separates compute from storage, allowing data teams to easily scale up and down based on their needs. Here are a few materials to help you get started: Start planning a migration to Snowflake with one of our Migration Kits.

Data Architecture 89

Data Architecture Architecture Hadoop Data Warehouse 89