Pinterest Engineering

AUGUST 18, 2022

The HBase Ecosystem, though having various advantages like strong consistency at row level in high volume requests, flexible schema, low latency access to data, Hadoop integration, etc. In this blog post, we will first learn the various approaches considered for data migration with their trade offs.

Data Ingestion 52

Data Ingestion Hadoop Database Kafka 52

Cloudera

JUNE 25, 2021

Many Cloudera customers are making the transition from being completely on-prem to cloud by either backing up their data in the cloud, or running multi-functional analytics on CDP Public cloud in AWS or Azure. The Replication Manager service facilitates both disaster recovery and data migration across different environments.

Cloud 68

Cloud Data Lake Cloud Storage Metadata 68

Propel Data

JANUARY 9, 2023

Apollo Server 3 is officially deprecated, so it’s time to upgrade to Apollo Server 4. This post explains the expected and unexpected changes

Data Analytics 52

Data Analytics Data 52

ArcGIS

JULY 6, 2023

This third blog in a series provides a set of sample tools to migrate a personal geodatabase from ArcMap, to a file or mobile geodatabase in ArcGIS Pro.

Data 52

Data Data Management Management 52

Cloudera

APRIL 14, 2021

The blog “ Migrating Apache NiFi Flows from HDF to CFM with Zero Downtime ” detailed how many common NiFi dataflows can be easily migrated when the Hortonworks DataFlow and Cloudera Flow Management clusters are running side-by-side. The ZooKeeper service supporting the HDF NiFi cluster is not included in the migration.

Professional Services 52

Professional Services Data Data Process Process 52

RandomTrees

SEPTEMBER 6, 2020

Snowflake Overview A data warehouse is a critical part of any business organization. Lot of cloud-based data warehouses are available in the market today, out of which let us focus on Snowflake. Snowflake is an analytical data warehouse that is provided as Software-as-a-Service (SaaS).

Cloud Storage 52

Cloud Storage Data Ingestion Data Cleanse Data Warehouse 52

Netflix Tech

MAY 4, 2023

Migrating Critical Traffic At Scale with No Downtime — Part 1 Shyam Gala , Javier Fernandez-Ivern , Anup Rokkam Pratap , Devang Shah Hundreds of millions of customers tune into Netflix every day, expecting an uninterrupted and immersive streaming experience.

Utilities 135

Utilities Systems Architecture Coding 135

The Pragmatic Engineer

AUGUST 10, 2023

Stack Overflow shares website data with some of its most active members with reputations higher than 25,000. I’d like to call out how neat this approach is and how it contributes to transparency, even when the data isn’t flattering. However, the traffic data turned out to not account for a Google Analytics change.

Retail 171

Retail Utilities Software Engineer Software Engineering 171

Pinterest Engineering

NOVEMBER 7, 2023

Jeff Xiang | Software Engineer, Logging Platform Vahid Hashemian | Software Engineer, Logging Platform Jesus Zuniga | Software Engineer, Logging Platform At Pinterest, data is ingested and transported at petabyte scale every day, bringing inspiration for our users to create a life they love.

Kafka 98

Kafka Java Software Engineer Software Engineering 98

Hevo

APRIL 26, 2024

Microsoft SQL Server is commonly used to manage controls/permissions, backup, recovery, and data migration — but it may not be the best option for you. Through this blog post, we will be listing out the top 10 SQL Server alternatives for you to consider while investigating the best fit.

SQL 52

SQL Management IT Data 52

The Pragmatic Engineer

AUGUST 10, 2023

A blog post suggests traffic is down 50% at Stack Overflow, due to ChatGPT gaining popularity. I reached out to Stack Overflow for more details: the company admitted a drop, but it’s only 14% as per data shared with me. What kind of migration is causing a payout outage at Booking.com?

Engineering 189

Engineering Recruitment Retail Software Engineer 189

Knowledge Hut

MARCH 5, 2024

In this blog article, I will explain what specific difficulties AWS challenges companies have when utilizing AWS and how can they best utilize managed service providers (MSPs) to assist them grow their business. A managed database running on MySQL, PostgreSQL, Oracle, or SQL Server can be used to store the data.

AWS 52

AWS Amazon Web Services Cloud Computing PostgreSQL 52

databricks

JUNE 6, 2023

Apache Parquet is one of the most popular open source file formats in the big data world today. Being column-oriented, Apache Parquet allows.

Data Lake 79

Data Lake Big Data Data Data Engineering 79

Christophe Blefari

MARCH 17, 2023

If you live in a cave or if you only read my newsletter to get news about the data world you might have missed that GPT-4 has been announced and released this week. In the article Grab team explain how to migrated from roles to attributes autorisation on Kafka. How to be a good data analyst without good data.

Data 130

Data SQL Deep Learning Kafka 130

Cloudera

JULY 14, 2023

Introduction For more than a decade now, the Hive table format has been a ubiquitous presence in the big data ecosystem, managing petabytes of data with remarkable efficiency and scale. Some of the common issues include constrained schema evolution, static partitioning of data, and long planning time because of S3 directory listings.

Metadata 69

Metadata Data Warehouse Big Data Ecosystem Java 69

Cloudera

FEBRUARY 1, 2024

Delivering transformational innovation and accurate business decisions requires harnessing the full potential of your organization’s entire data ecosystem. Ultimately, this boils down to how reliable and trustworthy the underlying data that feeds your insights and applications is. How does Cloudera support Day 2 operations?

Cloud 81

Cloud Architecture Utilities Designing 81

Hevo

APRIL 26, 2024

Amazon Redshift is a serverless, fully managed leading data warehouse in the market, and many organizations are migrating their legacy data to Redshift for better analytics. In this blog, we will discuss the best Redshift ETL tools that you can use to load data into Redshift.

ETL Tools 40

ETL Tools Data Warehouse Management Data 40

Cloudera

JUNE 13, 2023

Data and analytics are the essential engines in this strategy which entails upgrading the bank’s core banking technology to deliver reliable, efficient, and secure services for their customers while safeguarding critical data.

Banking 76

Banking Professional Services Insurance Retail 76

Data Engineering Weekly

FEBRUARY 4, 2024

RudderStack is the Warehouse Native CDP, built to help data teams deliver value across the entire data activation lifecycle, from collection to unification and activation. Joe Reis: Definition of Data Modeling & What Data Modeling Is not Joe raised a very fundamental question in data engineering.

Data Engineering 62

Data Engineering Data Engineer Engineering PostgreSQL 62

Cloudera

SEPTEMBER 26, 2023

Organizations increasingly rely on streaming data sources not only to bring data into the enterprise but also to perform streaming analytics that accelerate the process of being able to get value from the data early in its lifecycle.

Kafka 89

Kafka Technology IT Government 89

Cloudera

JULY 7, 2021

To that end, we released a few blog posts to help you migrate from HDF to CFM: Migrating NiFi flows from HDF to CFM on CDF with no downtime. No Data Loss and No Service Interruption – HDF to CFM Rolling Migration. Migration is recommended. In-place migration with AM2CM. Note that when CDP 7.1.7

Management 104

Management Java Certification Process 104

Christophe Blefari

FEBRUARY 24, 2023

Data engineering team moving data manually ( credits ) Dear readers, I hope you had a great week. This is something I struggle with, I really like writing, I really like this newsletter, I really like the blog, but it takes me one day per week to be done. For the last 2 newsletters I've tried to ask your for paying support.

Kafka 130

Kafka Data Lake Data Storage Data 130

Cloudera

JANUARY 31, 2024

Replication is a crucial capability in distributed systems to address challenges related to fault tolerance, high availability, load balancing, scalability, data locality, network efficiency, and data durability. SRM replicates data at high performance and keeps topic properties in sync across clusters.

Management 68

Management Kafka Big Data Cloud 68

Christophe Blefari

JANUARY 7, 2024

This edition of Data News will focus on the end of 2023 with a good retrospective about me and my activities—content and freelancing. My only target is, to be honest, to grow my blog, having 4000 people who trusted my enough to enter their email and validate the subscription is just crazy. More about on this soon.

Data 130

Data SQL Python Data Engineering 130

Cloudera

FEBRUARY 4, 2022

This blog post provides an overview of the HBase to CDP Operational Database (COD) migration process. CDP Operational Database enables developers to quickly build future-proof applications that are architected to handle data evolution. The following is an overview of the HBase migration steps: Prepare for the migration.

Database 75

Database Cloud Process Building 75

Christophe Blefari

JUNE 3, 2023

I wanted to write more about Microsoft Fabric and the states of data that were published last week but I'll do it another time. The article is saying that it lays down with Japanese new strategy to become a leader in AI technologies, by removing barriers on training data they hope to open doors. Or several needles.

Data Pipeline 130

Data Pipeline Data SQL Algorithm 130

Data Engineering Weekly

MARCH 3, 2024

RudderStack is the Warehouse Native CDP, built to help data teams deliver value across the entire data activation lifecycle, from collection to unification and activation. Editor’s Note: Chennai, India Meetup - March-08 Update We are thankful to Ideas2IT to host our first Data Hero’s meetup.

Data Engineering 67

Data Engineering Data Engineer Pipeline-centric Engineering 67

Cloudera

MARCH 22, 2024

Consider it like fortifying the foundation of a house: moving to RHEL is about laying a stronger, more supportive groundwork for your data platform. For those transitioning from CentOS Linux (or other distributions), Cloudera will offer an easy migration path.

Cloud 110

Cloud Certification Systems Architecture 110

Databand.ai

AUGUST 30, 2023

7 Data Testing Methods, Why You Need Them & When to Use Them Helen Soloveichik August 30, 2023 What Is Data Testing? Data testing involves the verification and validation of datasets to confirm they adhere to specific requirements. This is part of a series of articles about data quality.

Data Validation 52

Data Validation Data Integration Data Database 52

databricks

MARCH 26, 2024

As new Generative AI capabilities continue to emerge with heightened customer expectations, data modernization and migration to the cloud have become critical success.

Banking 85

Banking Data Solutions Data Cloud 85

Lyft Engineering

NOVEMBER 29, 2023

Sub-second query systems allow for near real-time data explorations and low latency, high throughput queries, which are particularly well-suited for handling time-series data. For our customers, this means faster analytics on near real-time data and decision making. An example of how we use Druid rollup at Lyft.

Kafka 104

Kafka Data Ingestion Datasets Architecture 104

Knowledge Hut

MARCH 28, 2024

Data science has become one of the most trending fields today. Data engineering is one of them. According to AnalytixLabs , the data science market is expected to be worth USD 230.80 This demonstrates the increasing need for Microsoft Certified Data Engineers. That’s where data engineers are on the go.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Warehouse 52

Snowflake

JULY 27, 2023

In this guest blog post, HomeToGo’s director of data, Stephan Claus, explains why the company migrated to Snowflake to meet its data needs. This article is based on Stephan’s presentation during the Snowflake Data World Tour 2022. Over the course of this journey, HomeToGo’s data needs have evolved considerably.

Data Architecture 52

Data Architecture Architecture Building Structured Data 52

Netflix Tech

NOVEMBER 29, 2023

As noted in an earlier blog post , we began developing an HDR variant of VMAF; let’s call it HDR-VMAF. A vital aspect of such development is subjective testing with HDR encodes in order to generate training data. 1) depicts the migration of traffic from fixed bitrates to DO encodes. The graphic below (Fig.

Metadata 100

Metadata Electronics Algorithm Software Engineer 100

Cloudera

OCTOBER 16, 2023

Cloudera recently released a fully featured Open Data Lakehouse , powered by Apache Iceberg in the private cloud, in addition to what’s already been available for the Open Data Lakehouse in the public cloud since last year. SDX Integration: Provides common security and governance policies, as well as data lineage and auditing.

Cloud 75

Cloud Kafka SQL Data 75

Data Engineering Weekly

JUNE 18, 2023

Data Engineering Weekly Is Brought to You by RudderStack RudderStack provides data pipelines that make it easy to collect data from every application, website, and SaaS platform, then activate it in your warehouse and business tools. Data management is critical for any organization to succeed in this AI world.

Data Engineering 52

Data Engineering Data Engineer Engineering MySQL 52

Christophe Blefari

DECEMBER 31, 2023

Before moving on to the Data News, a bit of personal news, in December, I took part in the MotherDuck meetup in Berlin. End of January, on the 31st I'll speak at a Modern Data Stack conf in Paris, still about DuckDB, but this time in French. Enjoy this last 2023 Data News. How to know if your data team is successful?

Data 100

Data Cloud Storage Datasets Python 100