Monte Carlo

NOVEMBER 21, 2024

Let’s set the scene: your company collects data, and you need to do something useful with it. Whether it’s customer transactions, IoT sensor readings, or just an endless stream of social media hot takes, you need a reliable way to get that data from point A to point B while doing something clever with it along the way. That’s where data pipeline design patterns come in.

Data Pipeline 52

Data Pipeline Designing Lambda Architecture Kafka 52

Start Data Engineering

NOVEMBER 21, 2024

1. Introduction 2. Analytical databases aggregate large amounts of data 3. Most platforms enable you to do the same thing but have different strengths 3.1. Understand how the platforms process data 3.1.1. A compute engine is a system that transforms data 3.1.2. Metadata catalog stores information about datasets 3.1.3. Data platform support for SQL, Dataframe, and Dataset APIs 3.1.4.

Metadata 130

Metadata Datasets SQL Database 130

Seattle Data Guy

NOVEMBER 19, 2024

Scraping data from PDFs is a right of passage if you work in data. Someone somewhere always needs help getting invoices parsed, contracts read through, or dozens of other use cases. Most of us will turn to Python and our trusty list of Python libraries and start plugging away. Of course, there are many challenges… Read more The post Challenges You Will Face When Parsing PDFs With Python – How To Parse PDFs With Python appeared first on Seattle Data Guy.

Python 130

Python Data Data Science Data Engineer 130

Tweag

NOVEMBER 20, 2024

Two years ago I wrote a blog post to announce that the GHC wasm backend had been merged upstream. I’ve been too lazy to write another blog post about the project since then, but rest assured, the project hasn’t stagnated. A lot of improvements have happened after the initial merge, including but not limited to: Many, many bugfixes in the code generator and runtime, witnessed by the full GHC testsuite for the wasm backend in upstream GHC CI pipelines.

Coding 137

Coding Programming Project Process 137

Advertisement

Whether you’re creating complex dashboards or fine-tuning large language models, your data must be extracted, transformed, and loaded. ETL and ELT pipelines form the foundation of any data product, and Airflow is the open-source data orchestrator specifically designed for moving and transforming data in ETL and ELT pipelines. This eBook covers: An overview of ETL vs.

Architecture

Confessions of a Data Guy

NOVEMBER 18, 2024

In my never-ending quest to plumb the most boring depths of every single data tool on the market, I found myself annoyed when recently using DuckDB for a benchmark that was reading parquet files from s3. What was not clear, or easy, was trying to figure out how DuckDB would LIKE to read default AWS […] The post DuckDB … reading from s3 … with AWS Credentials and more. appeared first on Confessions of a Data Guy.

AWS 113

AWS Data Big Data SQL 113

databricks

NOVEMBER 21, 2024

We’re excited to announce a new integration between Databricks Notebooks and AI/BI Dashboards, enabling you to effortlessly transform insights from your notebooks into.

BI 118

BI 118

KDnuggets

NOVEMBER 20, 2024

Let's take a look at how we can perform NER using that Swiss army knife of NLP and LLM libraries, Hugging Face's Transformers.

120

120

Confluent

NOVEMBER 21, 2024

Confluent’s CwC partner program introduces bidirectional data streaming for SAP Datasphere, powered by Apache Kafka and Apache Flink; CwC Q4 2024 new entrants.

Programming 105

Programming Kafka Data 105

databricks

NOVEMBER 20, 2024

We are thrilled to unveil the finalists for the Databricks Generative AI Startup Challenge , a competition designed to spotlight innovative early-stage startups.

Designing 108

Designing 108

Cloudera

NOVEMBER 21, 2024

As advanced analytics and AI continue to drive enterprise strategy, leaders are tasked with building flexible, resilient data pipelines that accelerate trusted insights. AI pioneer Andrew Ng recently underscored that robust data engineering is foundational to the success of data-centric AI —a strategy that prioritizes data quality over model complexity.

Pipeline-centric 88

Pipeline-centric Data Engineering Data Engineer Engineering 88

Speaker: Tamara Fingerlin, Developer Advocate

In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!

Data

KDnuggets

NOVEMBER 22, 2024

This article examines the challenges of aligning machine behavior with human values, and the role of ethical frameworks in shaping responsible AI.

106

106

Snowflake

NOVEMBER 19, 2024

Today’s business landscape is increasingly competitive — and the right data platform can be the difference between teams that feel empowered or impaired. I love talking with leaders across industries and organizations to hear about what’s top of mind for them as they evaluate various data platforms. In these conversations, there are a number of questions that I hear time and time again: Will my data platform be scalable and reliable enough?

Data Governance 88

Data Governance Government Healthcare Building 88

databricks

NOVEMBER 21, 2024

We’re excited to announce that the Databricks Assistant , now fully hosted and managed within Databricks, is available in public preview! This version.

Management 96

Management 96

Engineering at Meta

NOVEMBER 19, 2024

AI plays a fundamental role in creating valuable connections between people and advertisers within Meta’s family of apps. Meta’s ad recommendation engine, powered by deep learning recommendation models (DLRMs) , has been instrumental in delivering personalized ads to people. Key to this success was incorporating thousands of human-engineered signals or features in the DLRM-based recommendation system.

Architecture 79

Architecture Engineering Deep Learning Systems 79

Advertisement

Many software teams have migrated their testing and production workloads to the cloud, yet development environments often remain tied to outdated local setups, limiting efficiency and growth. This is where Coder comes in. In our 101 Coder webinar, you’ll explore how cloud-based development environments can unlock new levels of productivity. Discover how to transition from local setups to a secure, cloud-powered ecosystem with ease.

Cloud

KDnuggets

NOVEMBER 19, 2024

Interested in data analytics? Here's a list of Python libraries you cannot do without.

Python 132

Python Data Analytics Data 132

Snowflake

NOVEMBER 21, 2024

As travelers increasingly expect personalized experiences, brands in the travel and hospitality industry must find innovative ways to leverage data in their marketing and product experiences. That said, managing vast, complex data sets across multiple brands, loyalty programs and guest touchpoints presents unique challenges for companies in this industry.

Hospitality 81

Hospitality Entertainment Data Governance Government 81

databricks

NOVEMBER 20, 2024

We are excited to introduce the gated Public Preview of Predictive Optimization for statistics. Announced at the Data + AI Summit, Predictive Optimization.

Data 88

Data 88

Precisely

NOVEMBER 18, 2024

Key Takeaways: Data integrity is required for AI initiatives, better decision-making, and more – but data trust is on the decline. Data quality and data governance are the top data integrity challenges, and priorities. A long-term approach to your data strategy is key to success as business environments and technologies continue to evolve. The rapid pace of technological change has made data-driven initiatives more crucial than ever within modern business strategies.

Data Analytics 72

Data Analytics Data Integration Data Governance Government 72

Advertisement

Large enterprises face unique challenges in optimizing their Business Intelligence (BI) output due to the sheer scale and complexity of their operations. Unlike smaller organizations, where basic BI features and simple dashboards might suffice, enterprises must manage vast amounts of data from diverse sources. What are the top modern BI use cases for enterprise businesses to help you get a leg up on the competition?

Business Intelligence

KDnuggets

NOVEMBER 20, 2024

Can SQL be used for advanced data manipulation in data science? It sure can with these seven techniques.

Data Science 114

Data Science SQL Data IT 114

Snowflake

NOVEMBER 19, 2024

On a day-to-day basis, Snowflake teams identify opportunities and help customers implement recommended best practices that ease the migration process from on-premises to the cloud. They also monitor potential challenges and advise on proven patterns to help ensure a successful data migration. This article highlights nine key areas to watch out for and plan around in order to accelerate a smooth transition to the cloud.

Cloud 78

Cloud Data Ingestion Data Validation Data Security 78

databricks

NOVEMBER 21, 2024

While large language models (LLMs) are increasingly adept at solving general tasks, they can often fall short on specific domains that are dissimilar.

Datasets 87

Datasets Building 87

Confluent

NOVEMBER 19, 2024

CDC has evolved to become a key component of data streaming platforms, and is easily enabled by managed connectors such as the Debezium PostgreSQL CDC connector.

PostgreSQL 64

PostgreSQL Database Data Management 64

Speaker: Jay Allardyce, Deepak Vittal, Terrence Sheflin, and Mahyar Ghasemali

As we look ahead to 2025, business intelligence and data analytics are set to play pivotal roles in shaping success. Organizations are already starting to face a host of transformative trends as the year comes to a close, including the integration of AI in data analytics, an increased emphasis on real-time data insights, and the growing importance of user experience in BI solutions.

Data

KDnuggets

NOVEMBER 19, 2024

100% online master’s program with flexible schedules designed for working professionals. Enrolling now for March 3rd.

Programming 103

Programming Data Science Designing Data 103

Snowflake

NOVEMBER 18, 2024

Security has been an integral part of Snowflake’s platform since the company was founded. Through the security capabilities of Snowflake Horizon Catalog , we empower security admins and CISO’s to better protect their environments. As part of our continued efforts to help customers secure their accounts, and in line with our pledge to align with CISA’s Secure By Design principles, we are announcing the general availability of Snowflake Leaked Password Protection (LPP).

Systems 77

Systems Designing Accessible Accessibility 77

databricks

NOVEMBER 18, 2024

As organizations increasingly leverage the Databricks Data Intelligence Platform for data and AI needs, upgrading to Unity Catalog is a key step in.

Data 89

Data 89

Precisely

NOVEMBER 21, 2024

Key Takeaways: Harness automation and data integrity unlock the full potential of your data, powering sustainable digital transformation and growth. Data and processes are deeply interconnected. Successful digital transformation requires you to optimize both so that they work together seamlessly. Simplify complex SAP® processes with automation solutions that drive efficiency, reduce costs, and empower your teams to act quickly.

Data Integration 59

Data Integration High Quality Data Manufacturing Data 59

Speaker: Tamara Fingerlin, Developer Advocate

In this new webinar, Tamara Fingerlin, Developer Advocate, will walk you through many Airflow best practices and advanced features that can help you make your pipelines more manageable, adaptive, and robust. She'll focus on how to write best-in-class Airflow DAGs using the latest Airflow features like dynamic task mapping and data-driven scheduling!

Data

KDnuggets

NOVEMBER 21, 2024

Improving existing software systems, making them more robust and capable of solving complex contemporary problems.

Systems 101

Systems 101

Towards Data Science

NOVEMBER 20, 2024

Data is booming. It comes in vast volumes and variety and this explosion comes with a plethora of job opportunities too. Is it worth switching to a data career now? My honest opinion: absolutely! It is worth mentioning that this article comes from an Electrical and Electronic Engineer graduate who went all the way and spent almost 8 years in academia learning about the Energy sector (and when I say all the way, I mean from a bachelor degree to a PhD and postdoc).

Data Engineering 64

Data Engineering Data Engineer Engineering Electronics 64

databricks

NOVEMBER 21, 2024

We are excited to announce that Azure Private Link is now Generally Available (GA) for Databricks serverless and Mosaic AI Model Serving workloads.

Accessibility 86

Accessibility Accessible Management 86

Precisely

NOVEMBER 18, 2024

Key Takeaways: Data integrity is required for AI initiatives, better decision-making, and more – but data trust is on the decline. Data quality and data governance are the top data integrity challenges, and priorities. A long-term approach to your data strategy is key to success as business environments and technologies continue to evolve. The rapid pace of technological change has made data-driven initiatives more crucial than ever within modern business strategies.

Data Analytics 59

Data Analytics Data Governance Data Integration Government 59

Advertisement

Cloud Development Environments (CDEs) are changing how software teams work by moving development to the cloud. Our Cloud Development Environment Adoption Report gathers insights from 223 developers and business leaders, uncovering key trends in CDE adoption. With 66% of large organizations already using CDEs, these platforms are quickly becoming essential to modern development practices.

Cloud