Waitingforcode

JANUARY 15, 2025

A few months ago I wrote a blog post about event skew and how dangerous it is for a stateful streaming job. Since it was a high-level explanation, I didn't cover Apache Spark Structured Streaming deeply at that moment. Now the watermark topic is back to my learning backlog and it's a good opportunity to return to the event skew topic and see the dangers it brings for Structured Streaming stateful jobs.

IT 130

IT 130

Snowflake

JANUARY 15, 2025

Over the last three geospatial-centric blog posts, weve covered the basics of what geospatial data is, how it works in the broader world of data and how it specifically works in Snowflake based on our native support for GEOGRAPHY , GEOMETRY and H3. Those articles are great for dipping your toe in, getting a feel for the water and maybe even wading into the shallow end of the pool.

Transportation 97

Transportation BI Database-centric Metadata 97

Data Engineering Weekly

JANUARY 15, 2025

The Critical Role of AI Data Engineers in a Data-Driven World How does a chatbot seamlessly interpret your questions? How does a self-driving car understand a chaotic street scene? The answer lies in unstructured data processing—a field that powers modern artificial intelligence (AI) systems. Unlike neatly organized rows and columns in spreadsheets, unstructured data—such as text, images, videos, and audio—requires advanced processing techniques to derive meaningful insights.

Data Engineer 62

Data Engineer Data Engineering Unstructured Data Engineering 62

Snowflake

JANUARY 15, 2025

Y Combinator founder Paul Graham advises startup founders to live in the future, then build whats missing. I had the privilege of glimpsing the future through a series of interviews with investors on the bleeding edge of the AI landscape. Insights from these candid conversations laid the foundation for Startup 2025: Building a Business in the Age of AI, the AI startup report that Snowflake is publishing today.

Programming 104

Programming Building Cloud Management 104

Advertisement

In Airflow, DAGs (your data pipelines) support nearly every use case. As these workflows grow in complexity and scale, efficiently identifying and resolving issues becomes a critical skill for every data engineer. This is a comprehensive guide with best practices and examples to debugging Airflow DAGs. You’ll learn how to: Create a standardized process for debugging to quickly diagnose errors in your DAGs Identify common issues with DAGs, tasks, and connections Distinguish between Airflow-relate

Data Pipeline

ThoughtSpot

JANUARY 15, 2025

With the ever-growing focus on GenAI, many legacy BI tools have failed to invest in the analyst. By focusing solely on AI experiences for business teams, theyve alienated data teams, relegating analysts to disjointed tools and data silos. When in reality, businesses still need people who can help decision-makers assess messy data to diagnose and evaluate business problems.

BI 64

BI SQL Data Warehouse Raw Data 64

Cloudyard

JANUARY 15, 2025

Read Time: 2 Minute, 33 Second Snowflakes PARSE_DOCUMENT function revolutionizes how unstructured data, such as PDF files, is processed within the Snowflake ecosystem. Traditionally, this function is used within SQL to extract structured content from documents. However, Ive taken this a step further, leveraging Snowpark to extend its capabilities and build a complete data extraction process.

Data Cleanse 52

Data Cleanse Insurance Raw Data Unstructured Data 52

DareData

JANUARY 15, 2025

DareData will close 2024 with a 5% revenue growth compared to 2023. At first glance, given the rapid growth in our market, one might be tempted to classify this year as underwhelming. However, 2024 has been a transformative year for us. We started the year as a 100% consulting business. Consulting is highly dependent on people, and in small boutique firms like ours, this often means being heavily reliant on the partners.

Consulting 52

Consulting Finance Data Science Project 52

KDnuggets

JANUARY 15, 2025

Read this to understand the most advanced open source multilingual model.

106

106

Snowflake

JANUARY 15, 2025

Over the last three geospatial-centric blog posts, weve covered the basics of what geospatial data is, how it works in the broader world of data and how it specifically works in Snowflake based on our native support for GEOGRAPHY , GEOMETRY and H3. Those articles are great for dipping your toe in, getting a feel for the water and maybe even wading into the shallow end of the pool.

Transportation 81

Transportation BI Database-centric Metadata 81

KDnuggets

JANUARY 15, 2025

In this article, I will go through key elements that will help you answer data questions for your non-technical audience with ease.

Data 87

Data 87

Speaker: Tamara Fingerlin, Developer Advocate

Apache Airflow® 3.0, the most anticipated Airflow release yet, officially launched this April. As the de facto standard for data orchestration, Airflow is trusted by over 77,000 organizations to power everything from advanced analytics to production AI and MLOps. With the 3.0 release, the top-requested features from the community were delivered, including a revamped UI for easier navigation, stronger security, and greater flexibility to run tasks anywhere at any time.

Data Workflow

Uber Engineering

JANUARY 15, 2025

Large-scale computation is a major back end and infrastructure challenge for Uber to solve as we scale. We applied a compute engine called Ray in Ubers marketplace to improve computation efficiency and engineering productivity.

Engineering 58

Engineering 58

Sync Computing

JANUARY 15, 2025

Imagine a world where you could simply tell your data infrastructure what you want it to achieve, rather than meticulously configuring every detail. This is precisely what Jeff Chou, Co-founder and CEO of Sync, discussed in the latest daily.dev webinar. This innovative concept is being made real through Gradient, the AI agent for data infrastructure from Sync.

Data Pipeline 52

Data Pipeline Cloud Computing Machine Learning Cloud 52