Snowflake

APRIL 22, 2024

By leveraging advanced technology and data analytics, Magnite offers a comprehensive suite of tools and services designed to maximize ad revenue for publishers while helping them effectively reach their target audiences. Magnite was operating its Snowflake data platform on AWS US West, whereas SpringServe had its presence on AWS US East.

AWS 68

AWS Cloud Storage Cloud Technology 68

Cloudera

JANUARY 31, 2024

Replication is a crucial capability in distributed systems to address challenges related to fault tolerance, high availability, load balancing, scalability, data locality, network efficiency, and data durability. SRM replicates data at high performance and keeps topic properties in sync across clusters.

Management 69

Management Kafka Big Data Cloud 69

Engineering at Meta

MAY 16, 2023

Over the last few years, we have implemented MySQL Raft, a Raft consensus engine that was integrated with MySQL to build a replicated state machine. The deployment includes thousands of machines running over several regions and data centers across multiple continents. This was a data path–only protocol.

MySQL 136

MySQL Building Engineering Database 136

Data Engineering Podcast

FEBRUARY 5, 2023

In that time there have been a number of generational shifts in how data engineering is done. Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today?

Data Engineering 147

Data Engineering Data Engineer Engineering PostgreSQL 147

Data Engineering Podcast

JANUARY 22, 2023

Summary The most interesting and challenging bugs always happen in production, but recreating them is a constant challenge due to differences in the data that you are working with. Building your own scripts to replicate data from production is time consuming and error-prone.

PostgreSQL 130

PostgreSQL MongoDB MySQL Machine Learning 130

Snowflake

AUGUST 30, 2023

In today’s digital age, the growth and success of an enterprise heavily rely on how it manages and leverages its data. There are multiple enterprise data platforms in the market, each offering its distinct capabilities. Finally, replication is easy to configure and deploy. What makes it truly effective?

Scala 87

Scala Government Database Cloud 87

Rockset

FEBRUARY 22, 2024

Introduction Indexes are a crucial part of proper data modeling for all databases, and DynamoDB is no exception. DynamoDB's secondary indexes are a powerful tool for enabling new access patterns for your data. First, DynamoDB horizontally scales your databases by spreading your data across multiple partitions under the hood.

Database 80

Database Relational Database Accessible Accessibility 80

Ascend.io

FEBRUARY 15, 2024

A common hurdle that many data engineering teams face is how to choose the right data replication strategy. This article breaks down the complexities of data replication strategies in a way that’s easy to understand. That can include the creation of historical data that is not available in the source system.

Datasets 52

Datasets Database Data Warehouse Data 52

Snowflake

DECEMBER 12, 2023

Iceberg Tables bring the easy management and great performance of Snowflake to data stored externally in an open source format. Learn more about Iceberg Tables here. Learn more here. It further reduces the attack surface on shared data by limiting the scope of access granted to roles and users on the consumer account.

Metadata 83

Metadata Python Government AWS 83

Knowledge Hut

MARCH 14, 2024

It is compatible with spatial, JSON, XML, and relational data structures. There are three ways to deploy Microsoft Azure SQL Database: Elastic pool Single database Managed instances Azure SQL vs. SQL Server Microsoft Azure SQL is a cloud-based relational database technology that allows customers to host and utilize data as a service.

Database 52

Database SQL Relational Database BI 52

Snowflake

JUNE 27, 2023

The world is undergoing a remarkable transformation fueled by data. Organizations have accumulated silos across their data infrastructure to support various workloads, languages, tools, and formats because of technology limitations. Learn more about the continuous performance improvements we make to the platform on an ongoing basis.

Data Governance 54

Data Governance Unstructured Data Government SQL 54

Snowflake

JANUARY 29, 2024

Imagine seamlessly integrating your Google Analytics data with Snowflake, allowing you to combine it effortlessly with other key sources like CRM, ERP, social media metrics, email campaign data, and whatever data sources compose the full scope of your data estate.

Raw Data 103

Raw Data Aggregated Data Data Government 103

Snowflake

SEPTEMBER 13, 2023

In August, Snowflake released new features around Snowpark for Python, DevOps, pipeline replication, and more. Read on to learn more about the full set of features that were just announced. Read Snowflake Documentation to learn how to set up your development environment for Snowpark Python. Learn more. Learn more.

Python 76

Python SQL Data Pipeline Data Ingestion 76

Data Engineering Podcast

FEBRUARY 11, 2023

Summary Data is a team sport, but it's often difficult for everyone on the team to participate. For a long time the mantra of data tools has been "by developers, for developers", which automatically excludes a large portion of the business members who play a crucial role in the success of any data project.

PostgreSQL 130

PostgreSQL SQL Machine Learning Data Warehouse 130

Data Engineering Podcast

JANUARY 29, 2023

Data analysts create reports that are used by the business to understand and direct the business, but the process is very labor and time intensive. What are the most interesting, unexpected, or challenging lessons that you have learned while working on Omni? Can you describe what Omni Analytics is and the story behind it?

Business Intelligence 130

Business Intelligence PostgreSQL Building BI 130

Engineering at Meta

MARCH 18, 2024

Logarithm’s data model Logarithm represents logs as a named log stream of (host-local) time-ordered sequences of immutable unstructured text, corresponding to a single log file. A common example of metadata is rank ID in machine learning (ML) training, when multiple sequences of log lines are multiplexed into a single log stream (e.g.,

Engineering 89

Engineering Metadata Architecture Designing 89

LinkedIn Engineering

DECEMBER 19, 2023

Co-authors: Arjun Mohnot , Jenchang Ho , Anthony Quigley , Xing Lin , Anil Alluri , Michael Kuchenbecker LinkedIn operates one of the world’s largest Apache Hadoop big data clusters. Historically, deploying code changes to Hadoop big data clusters has been complex.

Big Data 92

Big Data Hadoop Metadata Data 92

The Pragmatic Engineer

MARCH 12, 2024

They come up with test cases and data. Code reviews reduce the need to pair while working on a task, allowing engineers to keep up with changes and learn from each other. Meanwhile, others could have the advantage of a specific context that cannot be replicated. The tester. Also responsible for scaffolding of tests. 

Engineering 253

Engineering Programming Language Hospitality Programming 253

Snowflake

MARCH 16, 2023

What if it was as easy as just a few clicks to get ServiceNow data directly into your Snowflake account so you could combine it with other data sources, including ERPs, HRs, and CRMs? The connector provides instant access to up-to-date ServiceNow data without the need to manually integrate against API endpoints. Well, now it is.

Hospitality 110

Hospitality BI Finance Government 110

Precisely

OCTOBER 26, 2023

The Top 3 Drivers for IBM i High Availability There are three primary factors driving decisions about HA solutions: High-performance real-time replication. Organizations need flexibility and scalability, ensuring that every change in the source system is replicated in the target system as quickly as possible. Data integrity.

Cloud Computing 64

Cloud Computing Cloud Kafka Data Integration 64

Snowflake

AUGUST 16, 2023

Applications Snowflake Native App Framework now available in AWS – public preview Snowflake Native Apps are an entirely new way to put data to work. Learn more here. Learn more here. Data comes in a continuous manner, and often a separate architecture is required to handle streaming data. Learn more here.

Scala 52

Scala Transportation Kafka Data Lake 52

Precisely

JANUARY 18, 2024

As IT leaders continuously embrace innovation fueled by connections between mainframe systems and modern cloud data platforms, it’s clear that a “modernize in place” approach is favored over the old “rip and replace” way of operating. Ready to unlock the value of all your enterprise data?

Technology 52

Technology AWS Cloud Computing Data Lake 52

Snowflake

JULY 12, 2023

In the early beginnings of the integration, associates shared customer and supply chain data between the two companies’ Snowflake accounts in Microsoft Azure. While 5 TBs of data in 2,000 tables was replicated from the HD Pro account into the HD Supply account, in parallel Snowflake enabled data sharing to continue between the accounts.

Cloud 52

Cloud Hospitality Google Cloud Healthcare 52

Christophe Blefari

OCTOBER 20, 2023

Read my dbt multi-project guide 📺 On the content side I'll also present next week the Fancy Data Stack project at the Data Engineering And Machine Learning Summit 2023 organised by Seattle Data Guy. This post gives great insights about the impact on the data platform team.

Generalist 130

Generalist Entertainment NoSQL Datasets 130

Snowflake

APRIL 25, 2023

As a global company with teams located across the Americas, Europe, and Asia, Pfizer views data as playing a critical role in every aspect of its operations, from research and development to manufacturing and distribution. Pfizer now securely shares data sets across different business units using Snowflake in AWS U.S.,

Manufacturing 54

Manufacturing AWS Cloud Accessible 54

Snowflake

APRIL 23, 2024

If you’re a Snowflake customer using ServiceNow’s popular SaaS application to manage your digital workloads, data integration is about to get a lot easier — and less costly. The connector provides immediate access to up-to-date ServiceNow data without the need to manually integrate against API endpoints.

Data Warehouse 78

Data Warehouse Data Integration Consulting Metadata 78

DoorDash Engineering

JANUARY 30, 2024

I will share what we learned as we made our fleet much more performant and cost-effective. Apache Cassandra is an open-source, distributed NoSQL database management system designed to handle large amounts of data across a wide range of commodity servers. Data is replicated across multiple nodes, ensuring no single point of failure.

NoSQL 84

NoSQL Database Systems Relational Database 84

RandomTrees

DECEMBER 20, 2023

The Next GEN Edge AI Application provides an immersive, intuitive and fun learning experience while saving time and money with multiple programs like Ranking, Classification and Designing. AI workflows at the edge use data originating from centralized data centers (cloud, devices) and data originating from human sources (edge).

IT 52

IT Hospitality Healthcare Deep Learning 52

Data Engineering Podcast

FEBRUARY 27, 2022

Summary There are a wealth of options for managing structured and textual data, but unstructured binary data assets are not as well supported across the ecosystem. As organizations start to adopt cloud technologies they need a way to manage the distribution, discovery, and collaboration of data across their operating environments.

Unstructured Data 100

Unstructured Data Cloud Management Metadata 100

Precisely

MARCH 20, 2023

With real-time logical replication, IBM shops can dramatically reduce risk and achieve faster, smoother migrations, along with switchable HA/DR cloud environments for ongoing flexibility and resiliency. Cloud-based HA/DR for IBM i Assure MIMIX replicates your complete IBM i environment, enabling lower cost, lower risk migration to the cloud.

Cloud 52

Cloud Banking Cloud Computing Data Integration 52

Precisely

APRIL 11, 2023

It’s a common scenario across industries: a new strategic initiative is launching, and you need trustworthy, relevant data to analyze and support the best outcomes for the business. What existing data does the organization have to support my initiative? What existing data does the organization have to support my initiative?

Accessible 52

Accessible Accessibility Metadata Data Integration 52

Christophe Blefari

APRIL 14, 2023

Anyway, here the weekly Data News, written faster than usual. This is just a normal evolution of deep learning systems. Hot takes on the Modern Data Stack — Matt gives 5 hot takes about the MDS. This time he writes about the new marketing approach of the modern data stack ecosystem.

Datasets 130

Datasets Data Deep Learning SQL 130

Data Engineering Podcast

AUGUST 12, 2019

Summary The extract and load pattern of data replication is the most commonly needed process in data engineering workflows. Fivetran is a platform that does the hard work for you and replicates information from your source systems into whichever data warehouse you use. Integration of multiple data sources (e.g.

Media 100

Media Data Warehouse Data Big Data 100

Knowledge Hut

NOVEMBER 17, 2023

In essence, it is a type of artificial intelligence that, like human intelligence, is able to understand, learn knowledge, reason, and apply it to a variety of activities and domains. Limited learning capacity, designed for specific tasks and requires human intervention for major adaptations. What is Weak AI?

Medical 52

Medical Algorithm Utilities Computer Science 52

Data Engineering Podcast

JANUARY 15, 2023

Summary The modern data stack has made it more economical to use enterprise grade technologies to power analytics at organizations of every scale. At the Modern Data Company they created the DataOS platform as a means of driving your full analytics lifecycle through code, while providing automatic knowledge graphs and data discovery.

Coding 130

Coding Building PostgreSQL Data Lake 130

Data Engineering Podcast

MARCH 22, 2021

For analytical systems, the only way to provide this reliably is by implementing change data capture (CDC). Unfortunately, this is a non-trivial undertaking, particularly for teams that don’t have extensive experience working with streaming data and complex distributed systems.

Data Warehouse 100

Data Warehouse Metadata Data Lake Hadoop 100

Data Engineering Podcast

JANUARY 8, 2023

Summary Managing end-to-end data flows becomes complex and unwieldy as the scale of data and its variety of applications in an organization grows. Part of this complexity is due to the transformation and orchestration of data living in disparate systems. Missing data? Start trusting your data with Monte Carlo today!

PostgreSQL 130

PostgreSQL Data Lake Data Warehouse BI 130