Precisely

JANUARY 9, 2025

Does the LLM capture all the relevant data and context required for it to deliver useful insights? Not to mention the crazy stories about Gen AI making up answers without the data to back it up!) Are we allowed to use all the data, or are there copyright or privacy concerns? But simply moving the data wasnt enough.

Data Integration 118

Data Integration Hadoop Data Warehouse Data Lake 118

Towards Data Science

JANUARY 30, 2025

Building more efficient AI TLDR : Data-centric AI can create more efficient and accurate models. I experimented with data pruning on MNIST to classify handwritten digits. What if I told you that using just 50% of your training data could achieve better results than using the fulldataset? Image byauthor.

Database-centric 106

Database-centric Datasets Data Architecture 106

Data Engineering Weekly

DECEMBER 28, 2024

In this post, we delve into predictions for 2025, focusing on the transformative role of AI agents, workforce dynamics, and data platforms. For professionals across domains—data engineers, AI engineers, and data scientists—the message is clear: adapt or become obsolete.

Unstructured Data 112

Unstructured Data Metadata Data Government 112

Start Data Engineering

JUNE 14, 2024

Run Data Pipelines 2.1. Introduction Whether you are new to data engineering or have been in the data field for a few years, one of the most challenging parts of learning new frameworks is setting them up! Introduction 2. Run on codespaces 2.2. Run locally 3. Projects 3.1. Projects from least to most complex 3.2. Conclusion 1.

Data Engineering 239

Data Engineering Data Engineer Project Engineering 239

Engineering at Meta

JANUARY 22, 2025

Data lineage is an instrumental part of Metas Privacy Aware Infrastructure (PAI) initiative, a suite of technologies that efficiently protect user privacy. It is a critical and powerful tool for scalable discovery of relevant data and data flows, which supports privacy controls across Metas systems.

Data Warehouse 115

Data Warehouse SQL Programming Language Data 115

Monte Carlo

NOVEMBER 26, 2024

Here’s where leading futurist and investor Tomasz Tunguz thinks data and AI stands at the end of 2024—plus a few predictions of my own. 2025 data engineering trends incoming. Small data is the future of AI (Tomasz) 7. The lines are blurring for analysts and data engineers (Barr) 8. Table of Contents 1.

Data Engineering 105

Data Engineering Data Engineer Engineering Unstructured Data 105

Precisely

OCTOBER 31, 2024

Key Takeaways: Data mesh is a decentralized approach to data management, designed to shift creation and ownership of data products to domain-specific teams. Data fabric is a unified approach to data management, creating a consistent way to manage, access, and share data across distributed environments.

Data Architecture 68

Data Architecture Architecture Metadata Government 68

Seattle Data Guy

MARCH 19, 2025

Data integration is critical for organizations of all sizes and industriesand one of the leading providers of data integration tools is Talend, which offers the flagship product Talend Studio. In 2023, Talend was acquired by Qlik, combining the two companies data integration and analytics tools under one roof.

Data Pipeline 130

Data Pipeline Data Integration Data Big Data 130

Christophe Blefari

MARCH 15, 2024

Saying mainly that " Sora is a tool to extend creativity " Last point Mira has been mocked and criticised online because as a CTO she wasn't able to say on which public / licensed data Sora has been trained on. Pandera, a data validation library for dataframes, now supports Polars.

Metadata 272

Metadata Data Data Warehouse Software Engineering 272

Data Engineering Podcast

APRIL 23, 2023

Meroxa was created to enable teams of all sizes to deliver real-time data applications. In this episode DeVaris Brown discusses the types of applications that are possible when teams don't have to manage the complex infrastructure necessary to support continuous data flows. Can you describe what Meroxa is and the story behind it?

Data Lake 277

Data Lake Kafka Machine Learning Data Warehouse 277

Analytics Vidhya

NOVEMBER 8, 2023

In the data-driven world […] The post Monitoring Data Quality for Your Big Data Pipelines Made Easy appeared first on Analytics Vidhya. Determine success by the precision of your charts, the equipment’s dependability, and your crew’s expertise. A single mistake, glitch, or slip-up could endanger the trip.

Big Data 253

Big Data Data Pipeline Data IT 253

Towards Data Science

DECEMBER 10, 2024

Three Zero-Cost Solutions That Take Hours, NotMonths A data quality certified pipeline. Source: unsplash.com In my career, data quality initiatives have usually meant big changes. Whats more, fixing the data quality issues this way often leads to new problems. Create a custom dashboard for your specific data qualityproblem.

PostgreSQL 75

PostgreSQL Data Python SQL 75

Seattle Data Guy

JANUARY 18, 2025

However, PDF files also present multiple challenges when it… Read more The post What Is PDFMiner And Should You Use It – How To Extract Data From PDFs appeared first on Seattle Data Guy.

IT 130

IT Education Data Designing 130

Analytics Vidhya

FEBRUARY 28, 2023

Introduction Data science has taken over all economic sectors in recent times. To achieve maximum efficiency, every company strives to use various data at every stage of its operations.

Data Science 290

Data Science Data Preparation Big Data Data 290

Confessions of a Data Guy

JANUARY 8, 2025

You know, for all the hoards of content, books, and videos produced in the “Data Space” over the last few years, famous or others, it seems I find there are volumes of information on the pieces and parts of working in Data. appeared first on Confessions of a Data Guy.

Data Storage 124

Data Storage Data Pipeline Data IT 124

Engineering at Meta

APRIL 28, 2025

Managing and understanding large-scale data ecosystems is a significant challenge for many organizations, requiring innovative solutions to efficiently safeguard user data. To address these challenges, we made substantial investments in advanced data understanding technologies, as part of our Privacy Aware Infrastructure (PAI).

Metadata 79

Metadata Data Utilities Data Warehouse 79

Precisely

JANUARY 30, 2025

Key Takeaways: Prioritize metadata maturity as the foundation for scalable, impactful data governance. Recognize that artificial intelligence is a data governance accelerator and a process that must be governed to monitor ethical considerations and risk.

Data Governance 102

Data Governance Government Metadata Data 102

databricks

MAY 5, 2025

Atlassian recently partnered with Databricks to power new data sharing capabilities from Atlassian Analytics, using the Delta Sharing protocol. This partnership allows Atlassian to provide

Data 112

Data 112

Towards Data Science

DECEMBER 16, 2024

Agentic AI, small data, and the search for value in the age of the unstructured datastack. Heres where leading futurist and investor Tomasz Tunguz thinks data and AI stands at the end of 2024plus a few predictions of myown. 2025 data engineering trends incoming. Search: tools that leverage a corpus of data to answer questions 3.

Unstructured Data 80

Unstructured Data Data Food Data Engineering 80

Seattle Data Guy

NOVEMBER 13, 2024

Much of the data we have used for analysis in traditional enterprises has been structured data. However, much of the data that is being created and will be created comes in some form of unstructured format. However, the digital era… Read more The post What is Unstructured Data?

Unstructured Data 130

Unstructured Data Process Structured Data Data 130

Analytics Vidhya

SEPTEMBER 20, 2023

Data engineering plays a pivotal role in the vast data ecosystem by collecting, transforming, and delivering data essential for analytics, reporting, and machine learning. Aspiring data engineers often seek real-world projects to gain hands-on experience and showcase their expertise.

Data Engineering 273

Data Engineering Data Engineer Coding Project 273

Monte Carlo

NOVEMBER 12, 2024

A data engineering architecture is the structural framework that determines how data flows through an organization – from collection and storage to processing and analysis. It’s the big blueprint we data engineers follow in order to transform raw data into valuable insights. How Does Uber Know Where to Go?

Architecture 96

Architecture Data Engineering Data Engineer Engineering 96

Confessions of a Data Guy

JANUARY 17, 2025

A Deep Dive into Databricks Labs’ DQX: The Data Quality Game Changer for PySpark DataFrames Recently, a LinkedIn announcement caught my eyeand honestly, it had me on the edge of my seat. Databricks Labs has unveiled DQX, a Python-based Data Quality framework explicitly designed for PySpark DataFrames.

Python 100

Python Data Designing IT 100

Cloudera

NOVEMBER 13, 2024

We are excited to announce the acquisition of Octopai , a leading data lineage and catalog platform that provides data discovery and governance for enterprises to enhance their data-driven decision making. This dampens confidence in the data and hampers access, in turn impacting the speed to launch new AI and analytic projects.

Metadata 90

Metadata Management Data Governance Government 90

Confessions of a Data Guy

JUNE 20, 2024

After 10 years of Data Engineering work, I think it’s time to hang up the proverbial hat and ride off into the sunset, never to be seen again. Sometimes I wonder if I’ve learned anything […] The post What I’ve Learned After A Decade Of Data Engineering appeared first on Confessions of a Data Guy.

Data Engineer 223

Data Engineer Data Engineering Engineering Data 223

Start Data Engineering

AUGUST 13, 2024

Data structures and algorithms to know 2.1. Introduction 2. Dictionary 2.3. Counter (from collections module) 2.7. Graph search 2.8.1 Depth First Search (DFS) 2.8.2. Breadth First Search BFS 2.9. Binary Search 3. Common DSA questions asked during DE interviews 3.1. Intervals 3.

Algorithm 200

Algorithm Data Engineering Data Engineer Engineering 200

Data Engineering Podcast

FEBRUARY 18, 2024

Summary A data lakehouse is intended to combine the benefits of data lakes (cost effective, scalable storage and compute) and data warehouses (user friendly SQL interface). Data lakes are notoriously complex. Join in with the event for the global data community, Data Council Austin.

Data Lake 262

Data Lake High Quality Data Data Warehouse Google Cloud 262

Start Data Engineering

APRIL 22, 2024

Most data projects use Docker to set up the data infra locally (and often in production). Communicate between containers and local OS 2.2.2. Start containers with docker CLI or compose 3. Conclusion 1. Introduction Docker can be overwhelming to start with.

Data Engineering 225

Data Engineering Data Engineer Engineering Data 225

Simon Späti

MARCH 28, 2025

As AI applications multiply quickly, vector technologies have become a frontier that data engineers must explore. The database landscape has reached 394 ranked systems across multiple categoriesrelational, document, key-value, graph, search engine, time series, and the rapidly emerging vector databases.

Technology 130

Technology PostgreSQL MySQL Database 130

Start Data Engineering

MAY 28, 2024

Building efficient data pipelines with DuckDB 4.1. Use DuckDB to process data, not for multiple users to access data 4.2. Cost calculation: DuckDB + Ephemeral VMs = dirt cheap data processing 4.3. Processing data less than 100GB? Introduction 2. Project demo 3. Use DuckDB 4.4.

Data Pipeline 219

Data Pipeline Python Building Data 219