Data Engineering Weekly

MARCH 3, 2024

Here is the agenda, 1) Data Application Lifecycle Management - Harish Kumar( Paypal) Hear from the team in PayPal on how they build the data product lifecycle management (DPLM) systems. Visit rudderstack.com to learn more. There will be food, networking, and real-world talks around data engineering. link] Nvidia: What Is Sovereign AI?

Data Engineering 67

Data Engineering Data Engineer Pipeline-centric Engineering 67

Snowflake

AUGUST 21, 2023

In this blog post, we’ll review the core data mesh principles, highlight how both organizations and modern data platforms are putting those principles into action, and demonstrate just how achievable a secure and efficient data mesh architecture can be. How, then, are modern data teams finding success with the data mesh?

Pipeline-centric 52

Pipeline-centric Architecture Government Data Architect 52

Snowflake

SEPTEMBER 11, 2023

This wide adoption of legislation requires organizations that store or process personally identifiable information (PII) to have greater control over that data, and better transparency about how they store it. Among the many reasons Snowflake is integral to an organization’s data strategy is the out-of-the-box security-related features.

Cloud 76

Cloud Electronics Government Data Governance 76

Snowflake

SEPTEMBER 13, 2023

Or you’re going to enable restaurants to reduce their food waste. The problem was no one really knew how to figure out what it was worth. The question should first be broken down into two parts: How to charge for my data? The question should first be broken down into two parts: How to charge for my data? But it’s yours.

Food 99

Food Retail Hospitality Data 99

Snowflake

JULY 27, 2023

In this guest blog post, HomeToGo’s director of data, Stephan Claus, explains why the company migrated to Snowflake to meet its data needs. This article is based on Stephan’s presentation during the Snowflake Data World Tour 2022. Something that is especially handy is Snowflake’s support for semi-structured data.

Data Architecture 52

Data Architecture Architecture Building Structured Data 52

DoorDash Engineering

OCTOBER 17, 2023

do so effectively, we had to find ways to reduce our SRM rate. In this post, we explore some of the common examples of SRM failures we experienced, the solutions we’ve implemented to solve these issues, and how we raised awareness of these solutions internally to dramatically reduce our SRM rate.

Education 87

Education Kafka Algorithm Data Warehouse 87

Snowflake

JUNE 27, 2023

At Snowflake Summit 2022, we introduced a new way of building apps with the Snowflake Native App Framework. Today, we are excited to bring the power of the Snowflake Native App Framework to developers around the world with the public preview in AWS. Snowflake Native Apps are now available in Snowflake Marketplace.

AWS 52

AWS MySQL Google Cloud Coding 52

Snowflake

FEBRUARY 14, 2023

As a result, they can drive fulfillment innovations such as operationalizing click-and-collect and reducing delivery windows from weeks to hours. Here’s how a few of our retail customers are using Snowflake to optimize fulfillment and delivery, leading to a great customer experience.

Retail 52

Retail Portfolio Cloud Accessible 52

Databand.ai

AUGUST 30, 2023

A well-defined data quality strategy enables better decision-making based on reliable information and reduces risks associated with poor-quality data. A data quality strategy details the processes, tools, and techniques employed to ensure your company’s data is accurate, consistent, complete, and up-to-date.

Building 52

Building Data Cleanse Data Governance Datasets 52

Databand.ai

AUGUST 30, 2023

This helps you make informed decisions about how best to clean and preprocess this information before using it in downstream processes like analytics or model training. It offers insights into patterns, relationships between attributes, columns and tables (if applicable), frequency distributions for categorical variables or values and more.

Data Cleanse 52

Data Cleanse Data Pipeline Datasets Data Validation 52

Databand.ai

JULY 11, 2023

Data Quality Platform: Benefits, Key Features, and How to Choose Eric Jones July 11, 2023 What Is a Data Quality Platform? By automating many of the processes involved in data quality management, data quality platforms can help organizations reduce errors, streamline workflows, and make better use of their data assets.

Data Cleanse 52

Data Cleanse Telecommunication High Quality Data BI 52

Monte Carlo

APRIL 28, 2022

Most data pros know Snowflake’s pricing model is consumption based–you pay for what you use. What many don’t know is Snowflake actually WANTS you to optimize your costs and has provided helpful features to rightsize your consumption. The key with Snowflake cost optimization initiatives is they too need to be right sized.

Data Warehouse 52

Data Warehouse Bytes SQL AWS 52

Databand.ai

AUGUST 30, 2023

This helps you make informed decisions about how best to clean and preprocess this information before using it in downstream processes like analytics or model training. Data Testing Tools: Key Capabilities and 6 Tools You Should Know Helen Soloveichik August 30, 2023 What Are Data Testing Tools?

Data Cleanse 52

Data Cleanse Data Validation Data Pipeline Datasets 52

Data Engineering Weekly

APRIL 16, 2023

The blog narrates the difference between deterministic and non-deterministic system design and the challenges of building LLM applications in production. However, as the data stack consolidation accelerates, we will see a much better permission model from Snowflake and Databricks. #3 Sign up free to test out the tool today.

Data Engineering 52

Data Engineering Data Engineer Engineering Pipeline-centric 52

DataKitchen

AUGUST 27, 2021

DataOps is fundamentally about eliminating errors, reducing cycle time, building trust and increasing agility. Figure 1 shows how the revenue curves diverge between a weak and robust product launch. They will be able to produce high-quality, on-demand insight that consistently leads to successful business decisions.

Pharmaceutical 98

Pharmaceutical Pipeline-centric Data Lake Data Analytics 98

DataKitchen

NOVEMBER 4, 2021

The term process hub is new to most people, so let’s look at a concrete example of how a process hub is used in a real-world application. The bottom line is how to attain analytic agility? Repetitive data process and query work reduce the time the commercial pharma analytics team has to create original insight.

Process 98

Process Data Process Pharmaceutical Data Lake 98

Striim

NOVEMBER 30, 2022

How does Striim Cloud bring value to the AWS ecosystem? How does Striim Cloud bring value to the AWS ecosystem? When you run Striim on AWS, it lets you create real-time data pipelines for Redshift, S3, Kinesis, Databricks, Snowflake and RDS for enterprise workloads.

AWS 52

AWS Cloud Management MySQL 52

Rockset

JANUARY 3, 2023

This blog compiles real-time data predictions from industry leaders so you know what’s coming in 2023. Enterprises will examine how to leverage real-time data to mitigate risk and find more value in margins and operational costs.”

Data Warehouse 52

Data Warehouse Education Database Data 52

Data Engineering Weekly

NOVEMBER 6, 2022

I plan to write a series of blogs on Schemata and Data Contract in the coming weeks. Martin kindly stepped in for me to give the update for my promised blog posts. Uber: How Uber Optimizes the Timing of Push Notifications using ML and Linear Programming In-House notifications are a significant lead generator for online commerce.

Data Engineering 40

Data Engineering Data Engineer Engineering Machine Learning 40

Rockset

AUGUST 30, 2021

Rockset: Real-time Analytics Built for the Cloud Rockset is doing for real-time analytics what Snowflake did for batch. It eliminates the cost and complexity around data preparation, performance tuning and operations, helping to accelerate the movement from batch to real-time analytics.

SQL 52

SQL Kafka MongoDB MySQL 52

Monte Carlo

AUGUST 31, 2023

Data downtime can cost companies upwards of millions of dollars per year , not to mention customer trust. Her team was responsible for managing reporting for the broader business, from generating dashboards for her CEO to use during All Hands meetings to setting strategy to reduce customer churn based on user metrics.

Data Lake 52

Data Lake Data Pipeline Unstructured Data Data Warehouse 52

Snowflake

APRIL 24, 2024

Building top-tier enterprise-grade intelligence using LLMs has traditionally been prohibitively expensive and resource-hungry, and often costs tens to hundreds of millions of dollars. As researchers, we have grappled with the constraints of efficiently training and inferencing LLMs for years. Arctic is efficiently intelligent and truly open.

Amazon Web Services 145

Amazon Web Services SQL AWS Architecture 145

Snowflake

APRIL 18, 2024

But as much as security operation center (SOC) analysts have come to rely on solutions like Splunk, there is one complaint that comes up for some: Costs can quickly add up. This blog post explores how Snowflake can help with this challenge. Now there are a few ways to ingest data into Snowflake.

Data Lake 78

Data Lake Data Ingestion Bytes Cloud Computing 78

ProjectPro

DECEMBER 7, 2021

This blog will give you an in-depth knowledge of what is a data pipeline and also explore other aspects such as data pipeline architecture, data pipeline tools, use cases, and so much more. Features of a Data Pipeline Data Pipeline Architecture How to Build an End-to-End Data Pipeline from Scratch? What is a Big Data Pipeline?

Data Pipeline 52

Data Pipeline Architecture Kafka AWS 52

Snowflake

DECEMBER 20, 2023

And once GPUs are obtained, operating the infrastructure requires specialized expertise and increases overhead costs that can constrain delivery of innovation. And to extend flexibility to virtually any language, framework or library as well as hardware of choice including GPUs, Snowflake built Snowpark Container Services.

Scala 117

Scala Government Java AWS 117

Christophe Blefari

MARCH 31, 2023

We are slowly approaching the 2-years anniversary of the blog and the newsletter. To be honest time flies and I’d have preferred to do more for the blog in the start of the year but my freelancing activities and my laziness took me so much. It will costs $6,25 per TB compared to $5 before. But obviously it changed.

Bytes 130

Bytes Data Google Cloud Education 130

Ascend.io

JUNE 6, 2023

Understanding and controlling cloud costs is a fundamental part of how Ascend manages the cloud infrastructure of our dedicated deployment customers. But if reading is not your thing, dive right into the fascinating details of how we master cloud costs in the video below.

Cloud 52

Cloud Data Pipeline Data Ingestion Cloud Storage 52

Snowflake

APRIL 19, 2023

Welcome to the third blog post in our series highlighting Snowflake’s data ingestion capabilities, covering the latest on Snowpipe Streaming (currently in public preview) and how streaming ingestion can accelerate data engineering on Snowflake.

Data Ingestion 57

Data Ingestion Kafka Java Data Pipeline 57

Monte Carlo

JANUARY 16, 2024

In this blog post, we’ll look at six innovations that are shaping the future of the data warehousing, as well as challenges and considerations that organizations should keep in mind. Similarly, Snowflake’s new Dynamic Tables and Snowpipe Streaming simplifies the traditionally complex process of managing batch and streaming data.

Data Lake 52

Data Lake Data Warehouse Unstructured Data AWS 52

Snowflake

MARCH 2, 2023

Snowflake enables organizations to be data-driven by offering an expansive set of features for creating performant, scalable, and reliable data pipelines that feed dashboards, machine learning models, and applications. As part of this, we are also supporting Snowpipe Streaming as an ingestion method for our Snowflake Connector for Kafka.

Kafka 123

Kafka Data Ingestion Data Pipeline Cloud Storage 123

Snowflake

AUGUST 30, 2023

In today’s digital age, the growth and success of an enterprise heavily rely on how it manages and leverages its data. Snowflake offers such an advanced replication mechanism that can be set up with just a few SQL commands in minutes. However, when it comes to enterprise-grade requirements certain key features are indispensable.

Scala 88

Scala Government Database Cloud 88

phData: Data Engineering

MARCH 8, 2023

Fivetran, a cloud-based automated data integration platform, has emerged as a leading choice among businesses looking for an easy and cost-effective way to unify their data from various sources. In today’s increasingly data-driven world, businesses need reliable and efficient ways to collect, analyze, and draw insights from their data.

IT 96

IT Data Warehouse Data Ingestion Data Integration 96

Snowflake

JUNE 28, 2023

Since its launch two years ago, Snowpark has been empowering data scientists, data engineers, and application developers to streamline their architectures, accelerate development, and boost the performance of data engineering and ML/AI workloads on Snowflake.

Python 52

Python Accessible Accessibility Pipeline-centric 52

Snowflake

AUGUST 22, 2023

“The future is what you make it,” declared Sheila Jordan , Chief Digital Technology Officer at Honeywell, in the opening keynote of this year’s Snowflake Summit keynote. Snowflake’s World of Data Collaboration, the theme of Snowflake Summit , was a world of data products—extending beyond data to include AI models and applications.

Metadata 71

Metadata Data AWS Business Analyst 71

Snowflake

NOVEMBER 2, 2023

At Snowflake, we don’t think prescribing a single pattern for all customers to adopt suits their interests. For example, customers who need a centralized store of data in large volume and variety – including JSON, text files, documents, images, and video – have built their data lake with Snowflake.

Data Lake 89

Data Lake Data Warehouse Cloud Unstructured Data 89

Data Science Blog: Data Engineering

SEPTEMBER 19, 2023

With the evolution of cloud computing, many organizations are now migrating their Data Warehouse Systems to the cloud for better scalability, flexibility, and cost-efficiency. Cost-Efficiency in Resource Allocation Cloud resources are priced based on usage, so efficient allocation is crucial for managing costs.

Data Warehouse 52

Data Warehouse Coding Systems Cloud 52

Snowflake

JANUARY 31, 2023

A number of blog posts, mainly from competitors, call out Snowflake as expensive. But in our conversations with customers, we frequently hear that Snowflake provides the best value for the money of any enterprise data platform option available today. So what gives?

Cloud 69

Cloud Data Programming Language SQL 69