Snowflake

SEPTEMBER 7, 2023

Snowpark is the set of libraries and runtimes that enables data engineers, data scientists and developers to build data engineering pipelines, ML workflows, and data applications in Python, Java, and Scala. Airbyte enables customers to continuously load data from external APIs and databases into Snowflake.

Amazon Web Services 119

Amazon Web Services AWS Government Python 119

Snowflake

APRIL 8, 2024

The list of Top 10 semi-finalists is a perfect example: we have use cases for cybersecurity, gen AI, food safety, restaurant chain pricing, quantitative trading analytics, geospatial data, sales pipeline measurement, marketing tech and healthcare. Our sincere thanks go out to everyone who participated in this year’s competition.

Pipeline-centric 127

Pipeline-centric Food Healthcare Unstructured Data 127

DataKitchen

MAY 10, 2024

The Five Use Cases in Data Observability: Fast, Safe Development and Deployment (#4) Introduction The integrity and functionality of new code, tools, and configurations during the development and deployment stages are crucial. This process is critical as it ensures data quality from the onset. Will I Create A Failed Deploy?

Data Ingestion 40

Data Ingestion Datasets Coding Data 40

Monte Carlo

FEBRUARY 12, 2024

To differentiate and expand the usefulness of these models, organizations must augment them with first-party data – typically via a process called RAG (retrieval augmented generation). Today, this first-party data mostly lives in two types of data repositories. Quality : Is the data itself anomalous?

Database 72

Database Unstructured Data Data Pipeline Metadata 72

DataKitchen

MAY 10, 2024

The Five Use Cases in Data Observability: Mastering Data Production (#3) Introduction Managing the production phase of data analytics is a daunting challenge. Overseeing multi-tool, multi-dataset, and multi-hop data processes ensures high-quality outputs.

Raw Data 40

Raw Data Data Ingestion Datasets Data 40

Rockset

APRIL 18, 2023

To highlight these new capabilities, we built a search demo using OpenAI to create embeddings for Amazon product descriptions and Rockset to generate relevant search results. Organizations have continued to accumulate large quantities of unstructured data, ranging from text documents to multimedia content to machine and sensor data.

Unstructured Data 105

Unstructured Data Metadata Machine Learning SQL 105

Snowflake

FEBRUARY 21, 2024

We previously announced Snowflake’s Unistore workload , which continues Snowflake’s legacy of breaking down data silos by uniting transactional and analytical data in a consistent and governed platform. Primary Key, Foreign Key and Unique Constraints enforce data integrity within your application.

Government 101

Government AWS Data Architect Architecture 101

Cloudera

OCTOBER 4, 2023

Fast moving data and real time analysis present us with some amazing opportunities. Every organization has some data that happens in real time, whether it is understanding what our users are doing on our websites or watching our systems and equipment as they perform mission critical tasks for us. Don’t blink — or you’ll miss it!

Telecommunication 89

Telecommunication Data Warehouse Java Manufacturing 89

Data Engineering Weekly

MARCH 24, 2024

OpenAI has more production deployments, but Google is catching on to it. link] Kai Waehner: The Data Streaming Landscape 2024 This is a comprehensive overview of the state of the data streaming landscape in 2024. Data Mesh is certainly a hotly debated topic in the data industry.

Data Engineering 62

Data Engineering Data Engineer Engineering Metadata 62

Snowflake

AUGUST 16, 2023

At Snowflake Summit, we announced a wave of product innovations: Snowpark ML Modeling API, Snowflake Native App Framework, Dynamic Tables and more. Applications Snowflake Native App Framework now available in AWS – public preview Snowflake Native Apps are an entirely new way to put data to work. If you missed out, not to worry!

Scala 52

Scala Transportation Kafka Data Lake 52

Rockset

SEPTEMBER 26, 2023

For the best results, these models must be informed by relevant data. AI-powered applications almost always need access to real-time data to deliver accurate results in a responsive user experience that the market has come to expect. Stale and siloed data can limit the potential value of AI to your customers and your business.

Metadata 64

Metadata Kafka Cloud Database 64

Knowledge Hut

JULY 3, 2023

In today's fast-paced and data-driven world, users increasingly depend on real-time intuition to get an aggressive side and define a plan of action. This is where real-time data ingestion comes into the picture. Data is collected from various sources such as social media feeds, website interactions, log files and processing.

Data Ingestion 52

Data Ingestion Pipeline-centric Google Cloud Media 52

Rockset

DECEMBER 14, 2023

At Rockset, we continually look for ways to give our customers better visibility into the product. As part of the work that apiserver would do when completing a query execution request, it would create a log that would eventually be ingested into the _events collection. These are sources which ingest data originating from Rockset.

Metadata 64

Metadata Systems Project Designing 64

Rockset

APRIL 26, 2023

This is part three in Rockset’s Making Sense of Real-Time Analytics (RTA) on Streaming Data series. In part 1 , we covered the technology landscape for real-time analytics on streaming data. The majority of anomaly detectors require streaming data, real-time data and historical data in order to generate inferences.

Architecture 52

Architecture Transportation Data Lake Insurance 52

Snowflake

SEPTEMBER 14, 2023

More than 50% of data leaders recently surveyed by BCG said the complexity of their data architecture is a significant pain point in their enterprise. Your technology stack should accommodate growth—in data volumes as well as in your business. It should foster collaboration across functions.

Data Architecture 98

Data Architecture Architecture Hadoop Data Warehouse 98

Rockset

OCTOBER 17, 2023

Data integrity is important and changes are often intimidating as they can disrupt data in unexpected ways. Rollback in time Data coming into Rockset goes through optional ingest transformations and indexing, operations that add overhead in terms of time and cost if you need to reingest that data for any reason.

SQL 52

SQL Data Integration IT Data 52

Ascend.io

AUGUST 15, 2023

This same principle holds true in data management. You require a comprehensive solution that addresses every facet, from ingestion and transformation to orchestration and reverse ETL. It’s no surprise, then, that the quest for Fivetran alternatives is on the rise as organizations set their sights on a more holistic data approach.

Data Management 52

Data Management Management Data Ingestion Data Pipeline 52

DataKitchen

OCTOBER 20, 2023

In today’s data-driven landscape, Data and Analytics Teams i ncreasingly face a unique set of challenges presented by Demanding Data Consumers who require a personalized level of Data Observability. Data Observability platforms often need to deliver this level of customization.

Insurance 70

Insurance Pharmaceutical Data Data Ingestion 70

Data Engineering Weekly

MAY 12, 2024

This year, we added some additional features to bring the data community together. Gloss Genius: How We Migrated From dbt Cloud and Scaled Our Data Development Gloss Genius describes its migration journey from dbt cloud to Airflow + custom Github actions. At scale, it becomes impossible to enrich the data assets manually.

Data Engineering 52

Data Engineering Data Engineer Engineering Data Lake 52

Monte Carlo

JANUARY 5, 2024

You know what they always say: data lakehouse architecture is like an onion. …ok, Data lakehouse architecture combines the benefits of data warehouses and data lakes, bringing together the structure and performance of a data warehouse with the flexibility of a data lake. Ingestion layer 2.

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

Monte Carlo

JANUARY 5, 2024

You know what they always say: data lakehouse architecture is like an onion. …ok, Data lakehouse architecture combines the benefits of data warehouses and data lakes, bringing together the structure and performance of a data warehouse with the flexibility of a data lake. Ingestion layer 2.

Architecture 52

Architecture Data Lake Metadata Unstructured Data 52

Scott Logic

APRIL 22, 2024

Several of the tech giants are quietly developing powerful and user-friendly tools to do just that; we recently attended a workshop run by Amazon, to show off some of their products in this area and get hands-on experience of some of AWS’s preview capabilities.

BI 97

BI Software Engineer Software Engineering Algorithm 97

Precisely

MARCH 7, 2024

They can ingest information as soon as it becomes available, summarize lengthy narrative content, and offer guidance to employees who manage the claims process. Yet experts warn that without proactive attention to data quality and data governance, AI projects could face considerable roadblocks.

Insurance 72

Insurance Process Medical Data Governance 72

Snowflake

JANUARY 26, 2023

Accessing data from the manufacturing shop floor is one of the key topics of interest with the majority of cloud platform vendors due to the pace of Industry 4.0 practices is the ability to collect and analyze vast amounts of data, allowing for improved efficiency, accuracy, and decision-making. Industry 4.0, cannot be overstated.

Data Ingestion 52

Data Ingestion Manufacturing Unstructured Data Architecture 52

Monte Carlo

MAY 3, 2024

It’s 2024, and the data estate has changed. Data systems are more diverse. But even though the data landscape is evolving, many enterprise data organizations are still managing data quality the “old” way: with simple data quality monitoring. For JetBlue’s data engineering team, that’s Microsoft Teams.

High Quality Data 52

High Quality Data Data Architecture Media 52

Data Engineering Weekly

MARCH 17, 2024

Compliance is mandatory, with strict penalties for violations, emphasizing the importance of data scientists familiarizing themselves with the law to avoid prohibited AI uses and ensure ethical, safe AI development. It discusses the significance of data governance, sharing history, and generative AI's impact on data economy standards.

Data Engineering 70

Data Engineering Data Engineer Engineering Data Governance 70

Cloudera

JULY 21, 2022

Co-author: Mike Godwin, Head of Marketing, Rill Data. Cloudera has partnered with Rill Data, an expert in metrics at any scale, as Cloudera’s preferred ISV partner to provide technical expertise and support services for Apache Druid customers. Data is made queryable in real time. Cloudera Data Warehouse).

BI 87

BI Digital Media Data Warehouse Kafka 87

Databand.ai

JULY 19, 2023

Complete Guide to Data Ingestion: Types, Process, and Best Practices Helen Soloveichik July 19, 2023 What Is Data Ingestion? Data Ingestion is the process of obtaining, importing, and processing data for later use or storage in a database. In this article: Why Is Data Ingestion Important?

Data Ingestion 52

Data Ingestion Process Data Cleanse Data Governance 52

Snowflake

DECEMBER 5, 2023

Engaging with partners and customers — and showcasing what’s new on the Snowflake product front — made for a dynamic time in Las Vegas. Here are highlights from the collaborations, integrations and product enhancements that we were proud to dig in to throughout the week.

AWS 103

AWS Amazon Web Services Government Cloud Computing 103

Ascend.io

APRIL 22, 2024

At Ascend, we believe it’s time to rethink data engineering from the ground up. As the world of data continues to evolve at a breakneck pace, we are thrilled to announce the next revolutionary step in our journey – Project Inception. Read More: The State of Data Engineering in 2023: Does Your Data Program Stack Up?

Project 52

Project Metadata Data Pipeline Data Engineering 52

Snowflake

SEPTEMBER 25, 2023

Digital advancements such as smart manufacturing and automation through AI, machine learning (ML), robotics, and IoT require a connected value chain ecosystem with a secure, scalable, and flexible data platform. Easy data sharing and collaboration A key feature of Snowflake is the ability to share and collaborate on data.

Manufacturing 89

Manufacturing Cloud Electronics BI 89

Pinterest Engineering

NOVEMBER 7, 2023

Jeff Xiang | Software Engineer, Logging Platform Vahid Hashemian | Software Engineer, Logging Platform Jesus Zuniga | Software Engineer, Logging Platform At Pinterest, data is ingested and transported at petabyte scale every day, bringing inspiration for our users to create a life they love.

Kafka 99

Kafka Java Software Engineer Software Engineering 99

Snowflake

SEPTEMBER 13, 2023

Users can now easily connect to external network locations, including external LLMs, from their Snowpark code while maintaining high security and governance over their data. With query_hash and query_parameterized_hash, we are now exposing new query-level data to make it easier for customers to group and analyze similar queries.

Python 79

Python SQL Data Pipeline Data Ingestion 79

Databand.ai

AUGUST 30, 2023

DataOps is a collaborative approach to data management that combines the agility of DevOps with the power of data analytics. It aims to streamline data ingestion, processing, and analytics by automating and integrating various data workflows. As a result, they can be slow, inefficient, and prone to errors.

Architecture 52

Architecture Data Ingestion Data Governance Data Cleanse 52

Towards Data Science

DECEMBER 1, 2023

Pet Project for Data/Analytics Engineers: Explore Modern Data Stack Tools — dbt Core, Snowflake, Fivetran, GitHub Actions. This hands-on experience will allow you to develop an end-to-end data lifecycle, from extracting data from your Google Calendar to presenting it in a Snowflake analytics dashboard. See Github repo.

Data Engineering 73

Data Engineering Data Engineer Project Engineering 73

DataKitchen

OCTOBER 2, 2023

Data Teams and Their Types of Data Journeys In the rapidly evolving landscape of data management and analytics, data teams face various challenges ranging from data ingestion to end-to-end observability. It explores why DataKitchen’s ‘Data Journeys’ capability can solve these challenges.

Data Ingestion 40

Data Ingestion Data Government Datasets 40

Ascend.io

DECEMBER 19, 2022

We hope the real-time demonstrations of Ascend automating data pipelines were a real treat—a long with the special edition T-Shirt designed specifically for the show (picture of our founder and CEO rocking the t-shirt below). With this approach, we’re able to augment our uniquely beautiful and intuitive visualization of data pipelines.

Data Ingestion 52

Data Ingestion Data Pipeline Metadata AWS 52