Data Engineering Podcast

JUNE 17, 2021

Summary Working with unstructured data has typically been a motivation for a data lake. Kirk Marple has spent years working with data systems and the media industry, which inspired him to build a platform for automatically organizing your unstructured assets to make them more valuable. No more scripts, just SQL.

Unstructured Data 100

Unstructured Data Data Warehouse Metadata Media 100

Towards Data Science

DECEMBER 14, 2023

Why a funnel is the centre of the war between data’s heaviest hitters Continue reading on Towards Data Science »

Unstructured Data 69

Unstructured Data Data Science Data Data Engineering 69

KDnuggets

JANUARY 26, 2022

Let's investigate the current need that enterprise organizations have to rapidly parse through unstructured data and examine several data management trends that are highly relevant in 2022.

Unstructured Data 88

Unstructured Data Data Data Management Management 88

Data Engineering Podcast

DECEMBER 11, 2022

Summary Data is one of the core ingredients for machine learning, but the format in which it is understandable to humans is not a useful representation for models. Embedding vectors are a way to structure data in a way that is native to how models interpret and manipulate information. images, audio, video, etc.)

Unstructured Data 100

Unstructured Data Machine Learning Data Engineering Data Engineer 100

KDnuggets

MAY 8, 2023

Learn how to find million-dollar insights from the data using exploratory analysis for your next data science project with Python.

Unstructured Data 108

Unstructured Data Data Analysis Data Science Data 108

KDnuggets

MAY 10, 2023

HuggingChat Python API: Your No-Cost Alternative • Exploratory Data Analysis Techniques for Unstructured Data • Stop Doing this on ChatGPT and Get Ahead of the 99% of its Users • ChatGPT as a Personalized Tutor for Learning Data Science Concepts • The Ultimate Open-Source Large Language Model Ecosystem

Unstructured Data 87

Unstructured Data Data Analysis Python Data Science 87

Hevo

MAY 10, 2024

Data has become a core component of society in the 21st century. One industry that is heavily reliant on data is the commerce sector. Specifically, Data Collection is a multi-billion dollar business that helps companies make critical business decisions and draw insights into their customers.

Unstructured Data 40

Unstructured Data Data Collection Data IT 40

Snowflake

JULY 10, 2023

With this new Snowpark capability, data engineers and data scientists can process any type of file directly in Snowflake, regardless if files are stored in Snowflake-managed storage or externally. Previously, working with these large and complex files would require a unique set of tools, creating data silos.

Unstructured Data 40

Unstructured Data Python Process Scala 40

Data Engineering Podcast

AUGUST 14, 2021

Summary The vast majority of data tools and platforms that you hear about are designed for working with structured, text-based data. What do you do when you need to manage unstructured information, or build a computer vision model? Hightouch is the easiest way to sync data into the platforms that your business teams rely on.

Unstructured Data 100

Unstructured Data Machine Learning Data Lake SQL 100

Snowflake

FEBRUARY 5, 2024

And it’s no wonder — this new technology has the potential to revolutionize the industry by augmenting the value of employee work, driving organizational efficiencies, providing personalized customer experiences, and uncovering new insights from vast amounts of data. One potential source is data sharing.

Unstructured Data 100

Unstructured Data Banking Government Insurance 100

Data Engineering Podcast

FEBRUARY 27, 2022

Summary There are a wealth of options for managing structured and textual data, but unstructured binary data assets are not as well supported across the ecosystem. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform.

Unstructured Data 100

Unstructured Data Cloud Management Metadata 100

Snowflake

FEBRUARY 6, 2024

Maverick replies, “The data on the MIG is inaccurate.” While flying may be more automated now, the importance of accurate and diverse data for aviation safety remains — and is likely even more critical. In two recent airplane accidents, automated systems aboard a Boeing 737 MAX made decisions based on inaccurate data.

Unstructured Data 103

Unstructured Data Retail Data Manufacturing 103

Analytics Vidhya

FEBRUARY 25, 2023

Introduction A data lake is a centralized and scalable repository storing structured and unstructured data. The need for a data lake arises from the growing volume, variety, and velocity of data companies need to manage and analyze.

Cloud Storage 157

Cloud Storage Data Lake Cloud Unstructured Data 157

KDnuggets

AUGUST 14, 2019

Processing unstructured text data in real-time is challenging when applying NLP or NLU. Find out how an alternative, called Domain-Specific Language Processing, can mine valuable information from data by following your guidance and using the language of your business.

Unstructured Data 90

Unstructured Data Process Data 90

Snowflake

MARCH 19, 2024

A company’s data strategy is always in motion. Some emerging approaches may be seen in our newly released Snowflake Data Trends 2024 , looking at how users in the Data Cloud are working with their data. For the report, we focused on two broad aspects of data strategy. The most marked finding was around governance.

Data Governance 118

Data Governance Government Unstructured Data Programming Language 118

Knowledge Hut

APRIL 23, 2024

In the modern data-driven landscape, organizations continuously explore avenues to derive meaningful insights from the immense volume of information available. Two popular approaches that have emerged in recent years are data warehouse and big data. Data warehousing offers several advantages.

Data Warehouse 52

Data Warehouse Big Data Unstructured Data Hadoop 52

Snowflake

MARCH 18, 2024

This time it’s all about ensuring that the data and the platform where it’s processed are ready for the new AI models. But there’s still a long way to go in an environment where the volume, velocity and complexity of data and data types is constantly increasing. And it’s not just any type of data. petabytes daily in 2021.

Unstructured Data 87

Unstructured Data Government Data Pipeline Cloud 87

Hevo

APRIL 20, 2024

In the early 2000s, organizations started dealing with more semi-structured and unstructured data, which consisted of images, videos, log files, text, and sensor data. They needed a storage solution that was more flexible than a data warehouse.

Data Lake 52

Data Lake Unstructured Data Data Warehouse Data 52

Knowledge Hut

APRIL 25, 2024

Big data vs machine learning is indispensable, and it is crucial to effectively discern their dissimilarities to harness their potential. Big Data vs Machine Learning Big data and machine learning serve distinct purposes in the realm of data analysis.

Machine Learning 52

Machine Learning Big Data Unstructured Data Data Mining 52

Snowflake

APRIL 20, 2023

Generative AI and large language models (LLMs) are revolutionizing many aspects of both developer and non-coder productivity with automation of repetitive tasks and fast generation of insights from large amounts of data. LLMs have the potential to help both developers and less-technically inclined users make sense of the world’s data.

Building 117

Building Unstructured Data Government Coding 117

Data Engineering Podcast

JUNE 19, 2022

Summary Data analysis is a valuable exercise that is often out of reach of non-technical users as a result of the complexity of data systems. Atlan is the metadata hub for your data ecosystem. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Metadata 130

Metadata Unstructured Data MongoDB MySQL 130

Knowledge Hut

APRIL 23, 2024

Big data and data mining are neighboring fields of study that analyze data and obtain actionable insights from expansive information sources. Big data encompasses a lot of unstructured and structured data originating from diverse sources such as social media and online transactions.

Data Mining 52

Data Mining Big Data Database-centric Unstructured Data 52

Jesse Anderson

SEPTEMBER 14, 2023

There has been quite a bit of writing covering GPT and LLMs from data science and business perspectives. I haven’t seen much from the data engineering side. Let me share my perspective, having been in data and AI for a while and using LLMs before they became popular. How can we use LLMs in data engineering? Be careful.

Data Engineering 147

Data Engineering Data Engineer Engineering NoSQL 147

Snowflake

NOVEMBER 2, 2023

Over the years, the technology landscape for data management has given rise to various architecture patterns, each thoughtfully designed to cater to specific use cases and requirements. These patterns include both centralized storage patterns like data warehouse , data lake and data lakehouse , and distributed patterns such as data mesh.

Data Lake 91

Data Lake Data Warehouse Cloud Unstructured Data 91

Hevo

MAY 24, 2023

Data drives the business world, and a significant amount of that data is unstructured. This implies that traditional relational databases can not cater to the needs of organizations seeking to store and manipulate this unstructured data. NoSQL Databases […]

NoSQL 52

NoSQL Unstructured Data Relational Database Database 52

Data Engineering Weekly

APRIL 7, 2024

Poor data quality and unlcear data ownership remains the top challenges for the data teams. Data Mesh continuously gaining popularity among the enterprises. It is a stark difference from the Gartner report about data mesh. I guess only the time will tell who wins in the data mesh vs data fabric war.

Data Engineering 70

Data Engineering Data Engineer Engineering Unstructured Data 70

Data Engineering Podcast

JUNE 26, 2022

Summary The proliferation of sensors and GPS devices has dramatically increased the number of applications for spatial data, and the need for scalable geospatial analytics. Atlan is the metadata hub for your data ecosystem. Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code.

Datasets 130

Datasets Unstructured Data Metadata MongoDB 130

Knowledge Hut

APRIL 23, 2024

Data Science and Business intelligence are popular terms in every business domain these days. Though both have data as the fundamental aspect, their uses, and operations vary. Data Science is the field that focuses on gathering data from multiple sources using different tools and techniques.

Business Intelligence 52

Business Intelligence Data Science BI Unstructured Data 52

Knowledge Hut

MAY 23, 2024

Data science has become the fastest-growing field and a buzzword that every industry knows and leverages. This vast stream of interdisciplinary domains deals with data in different ways. It helps companies understand data and obtain meaningful insights from it. Each of these tools comes with a set of some of these usages.

Data Science 52

Data Science MongoDB Programming Language Unstructured Data 52

Monte Carlo

FEBRUARY 12, 2024

To differentiate and expand the usefulness of these models, organizations must augment them with first-party data – typically via a process called RAG (retrieval augmented generation). Today, this first-party data mostly lives in two types of data repositories. Quality : Is the data itself anomalous?

Database 72

Database Unstructured Data Data Pipeline Metadata 72

Hevo

MAY 24, 2024

For decades, traditional On-Premise Data Warehouses have been tightly coupled with Data Storage and Computing, making them difficult to scale.

Unstructured Data 52

Unstructured Data Data Warehouse Data Storage Data 52

Towards Data Science

APRIL 6, 2023

Principles, practices, and examples for ensuring high quality data flows Source: DreamStudio (generated by author) Nearly 100% of companies today rely on data to power business opportunities and 76% use data as an integral part of forming a business strategy. Data quality is critical to delivering good customer experiences.

Unstructured Data 80

Unstructured Data Metadata Data Coding 80

Precisely

SEPTEMBER 6, 2023

According to a recent report from Drexel University’s LeBow Center for Business Analytics , 77% of data and analytics professionals say that data-driven decision-making is an important goal of data programs. However, fewer than half of survey respondents rate their trust in data as “high” or “very high.”

Data Governance 52

Data Governance Big Data Government Unstructured Data 52

Monte Carlo

AUGUST 25, 2023

Modern companies are ingesting, storing, transforming, and leveraging more data to drive more decision-making than ever before. Data teams need to balance the need for robust, powerful data platforms with increasing scrutiny on costs. But, the options for data storage are evolving quickly. Let’s dive in.

Data Lake 52

Data Lake Data Warehouse Unstructured Data Raw Data 52

Monte Carlo

JANUARY 5, 2024

You know what they always say: data lakehouse architecture is like an onion. …ok, Data lakehouse architecture combines the benefits of data warehouses and data lakes, bringing together the structure and performance of a data warehouse with the flexibility of a data lake. ok, so maybe they don’t say that.

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

Data Engineering Podcast

JUNE 12, 2022

Summary Building a well rounded and effective data team is an iterative process, and the first hire can set the stage for future success or failure. Trupti Natu has been the first data hire multiple times and gone through the process of building teams across the different stages of growth.

Metadata 100

Metadata Unstructured Data Business Intelligence MongoDB 100

The Modern Data Company

JANUARY 22, 2024

The Modern Data Company has been given an honorable mention in Gartner’s 2023 Magic Quadrant for Data Integration. In response, The Modern Data Company emerged, driven by a clear mission: to revolutionize data management and address challenges posed by a diverse and rapidly evolving data environment.

Data Integration 98

Data Integration Metadata Government Unstructured Data 98