Start Data Engineering

NOVEMBER 14, 2023

1. Introduction 2. What is an Open Table Format (OTF) 3. Why use an Open Table Format (OTF) 3.0. Setup 3.1. Evolve data and partition schema without reprocessing 3.2. See previous point-in-time table state, aka time travel 3.3. Git like branches & tags for your tables 3.4. Handle multiple reads & writes concurrently 4. Conclusion 5. Further reading 6.

Data 322

Data 322

Jesse Anderson

NOVEMBER 14, 2023

A Guest Post by Ole Olesen-Bagneux In this blog post I would like to describe a new data team, that I call ‘the data discovery team’. It’s a team that connects naturally into the constellation of the three data teams Operations team Data engineering team Data Science team as described in Jesse Anderson’s book Data Teams (2020) Before I explain what the data discovery team should do, it is necessary to add a bit of context on the concept of data discovery itself.

Metadata 147

Metadata Data Science Big Data Data 147

Waitingforcode

NOVEMBER 14, 2023

Have you written your first successful Apache Flink job and are still wondering the high-level API translates into the executable details? I did and decided to answer the question in the new blog post.

130

130

Confessions of a Data Guy

NOVEMBER 14, 2023

Show me the money. That’s what it’s all about. I have a question for you, to tickle your ears and mind. Get you out of that humdrum funk you are in. Here is my question, riddle me this all you hobbits. “Of what use is, and what good does the best and most advanced architecture […] The post Fleetclusters for Databricks + AWS to reduce Costs. appeared first on Confessions of a Data Guy.

AWS 113

AWS Architecture Data IT 113

Advertisement

Generative AI is upending the way product developers & end-users alike are interacting with data. Despite the potential of AI, many are left with questions about the future of product development: How will AI impact my business and contribute to its success? What can product managers and developers expect in the future with the widespread adoption of AI?

Data

Precisely

NOVEMBER 14, 2023

When speaking to organizations about data integrity , and the key role that both data governance and location intelligence play in making more confident business decisions, I keep hearing the following statements: “For any organization, data governance is not just a nice-to-have! “ “Everyone knows that 80% of data contains location information. Why are you still telling us this, Monica?

Data Governance 120

Data Governance Government Metadata Retail 120

ArcGIS

NOVEMBER 14, 2023

Here's everything new in ArcGIS Pro 3.2 from the Geodatabase Team. Schema Reports, 64-bit OIDs, Big Integer fields, new date fields, etc.

Data Management 126

Data Management Management Data 126

ArcGIS

NOVEMBER 14, 2023

This blog details the new features and enhancements that were add for deep learning using the Image Analyst extension - for Pro 3.2.

Deep Learning 119

Deep Learning 119

Netflix Tech

NOVEMBER 14, 2023

By Abhinaya Shetty , Bharath Mummadisetty This blog post will cover how Psyberg helps automate the end-to-end catchup of different pipelines, including dimension tables. In the previous installments of this series, we introduced Psyberg and delved into its core operational modes: Stateless and Stateful Data Processing. Now, let’s explore the state of our pipelines after incorporating Psyberg.

Metadata 97

Metadata Data Pipeline Scala Data Workflow 97

KDnuggets

NOVEMBER 14, 2023

Read on to find out everything you need to become a SAS Certified Machine Learning Engineer.

Machine Learning 127

Machine Learning Engineering 127

Netflix Tech

NOVEMBER 14, 2023

By Abhinaya Shetty , Bharath Mummadisetty In the inaugural blog post of this series, we introduced you to the state of our pipelines before Psyberg and the challenges with incremental processing that led us to create the Psyberg framework within Netflix’s Membership and Finance data engineering team. In this post, we will delve into a more detailed exploration of Psyberg’s two primary operational modes: stateless and stateful.

Data Process 90

Data Process Process Metadata Finance 90

Advertisement

Many organizations today are unlocking the power of their data by using graph databases to feed downstream analytics, enahance visualizations, and more. Yet, when different graph nodes represent the same entity, graphs get messy. Watch this essential video with Senzing CEO Jeff Jonas on how adding entity resolution to a graph database condenses network graphs to improve analytics and save your analysts time.

Database

KDnuggets

NOVEMBER 14, 2023

Discover how to run a small language model on your local CPU in just seven easy steps.

121

121

ArcGIS

NOVEMBER 14, 2023

This article is specific to ICEYE SAR satellite data and is part of a blog series on sensor support in ArcGIS Pro.

Data 107

Data Education 107

Snowflake

NOVEMBER 14, 2023

See why Snowflake’s healthcare customers rate the Data Cloud high in performance and cost savings. Each year, KLAS Research interviews thousands of healthcare professionals about the IT solutions and services their organizations use. Since 1996, the analyst firm has been leading the healthcare IT (HIT) industry in providing accurate, honest and impartial insights about vendor solutions and customer satisfaction metrics.

Healthcare 84

Healthcare Data Warehouse Data Governance Cloud 84

Confluent

NOVEMBER 14, 2023

Learn how Confluent helps financial services modernize payment platforms. Ensure interoperability between legacy payments messaging data while standardizing on the new ISO 20022 format.

Architecture 67

Architecture Data 67

Advertisement

Start-ups & SMBs launching products quickly must bundle dashboards, reports, & self-service analytics into apps. Customers expect rapid value from your product (time-to-value), data security, and access to advanced capabilities. Traditional Business Intelligence (BI) tools can provide valuable data analysis capabilities, but they have a barrier to entry that can stop small and midsize businesses from capitalizing on them.

Data Analysis

ArcGIS

NOVEMBER 14, 2023

Learn how to create and apply a custom vertical transformation in ArcGIS Pro.

Systems 92

Systems Data Management Management Data 92

DoorDash Engineering

NOVEMBER 14, 2023

DoorDash proactively embeds privacy into our products. As an example of how we do so, we delve here into an engineering effort to maintain user privacy. We will show how geomasking address data allows DoorDash to protect user privacy while maintaining local analytic capabilities. Privacy engineering overview To facilitate deliveries, users must give us some personal information, including such things as names, addresses, and phone numbers, in a Drive API request.

Engineering 55

Engineering Kafka Process Database 55

KDnuggets

NOVEMBER 14, 2023

Have our Sci-Fi dreams become reality?

83

83

Monte Carlo

NOVEMBER 14, 2023

Does it feel colder in here or is it all this Apache Iceberg talk? Over the last few months, Apache Iceberg has come to the forefront as a promising new open-source table format that removes many of the largest barriers to lakehouse adoption – namely, the high-latency and lack of OLTP (Online Transaction Processing) support afforded by Apache Hive. Databricks announced that Delta tables metadata will also be compatible with the Iceberg format, and Snowflake has also been moving aggressively to i

Data Lake 64

Data Lake Metadata Data Warehouse SQL 64

Speaker: Scott Sehlhorst

We know we want to create products which our customers find to be valuable. Whether we label it as customer-centric or product-led depends on how long we've been doing product management. There are three challenges we face when doing this. The obvious challenge is figuring out what our users need; the non-obvious challenges are in creating a shared understanding of those needs and in sensing if what we're doing is meeting those needs.

Certification

Ascend.io

NOVEMBER 14, 2023

Conway’s law states that organizations will invariably design systems that mirror their internal communication and organizational structures. This foundational insight into the very fabric of organizational behavior also applies to how many enterprises are approaching the AI opportunity. If you look closely, the solutions being proposed in your organization will likely reflect current departmental silos, legacy objectives, internal politics, and traditional power centers.

Pipeline-centric 52

Pipeline-centric Government Data Pipeline Recruitment 52

Cloudera

NOVEMBER 14, 2023

Of the many things one might take for granted, access to banking and financial services may not immediately come to mind. But as a thought experiment, imagine trying to buy a home or a car without the ability to take out a loan. Try depending on cash payments from your employer, or relying on alternative banking solutions like short-term payday loans, check-cashing services, and prepaid debit cards.

Banking 73

Banking Education Unstructured Data Algorithm 73

Monte Carlo

NOVEMBER 14, 2023

Imagine, if you will, a world where data just… flows. No hiccups. No “Oops, wrong format.” Just smooth, seamless operations. This is the world that data orchestration tools aim to create. Data orchestration tools minimize manual intervention by automating the movement of data within data pipelines. Similar to a traffic director for information, data orchestration tools gather data from various locations, organize it into a usable format, and then activate it for analysis and consumption.

Pipeline-centric 40

Pipeline-centric Google Cloud Data Workflow Python 40

KDnuggets

NOVEMBER 14, 2023

In alignment with President Biden's recent Executive Order emphasizing safe, secure, and trustworthy AI, we share our Trusted AI (TAI) lessons learned two years into the course of our US Federally funded TAI research projects.

Project 96

Project 96

Advertisement

While data platforms, artificial intelligence (AI), machine learning (ML), and programming platforms have evolved to leverage big data and streaming data, the front-end user experience has not kept up. Holding onto old BI technology while everything else moves forward is holding back organizations. Traditional Business Intelligence (BI) aren’t built for modern data platforms and don’t work on modern architectures.

BI

Scott Logic

NOVEMBER 14, 2023

In this post we’ll take a look at queues, logs and pub/sub systems in order to understand the options for sending data asynchronously between services. We’ll provide examples of each and discuss the tradeoffs that must be made. Introduction In any organisation there is a need to distribute data from a source system to other systems. This is especially true with the modern micro-services architecture.

Technology 52

Technology Kafka AWS Data 52