Graph Learning & LLMs

SLASH the Sink: Sharpening Structural Attention Inside LLMs

This research introduces SLASH, a training-free solution that significantly enhances Large Language Models' (LLMs) ability to process graph structures. We discovered LLMs inherently reconstruct graph topologies but this capability is suppressed by the 'attention sink.' SLASH sharpens these latent structural signals, overcoming a fundamental bottleneck between semantic anisotropy and topology-aware local aggregation. Our approach delivers consistent performance gains across diverse graph reasoning and molecular prediction tasks, offering a plug-and-play methodology to leverage LLMs for structured data without architectural changes or extensive fine-tuning.

The core contribution is the identification and resolution of the 'attention sink' bottleneck, which dilutes LLMs' intrinsic structural understanding. By re-distributing attention from the sink to topological patterns, SLASH enables LLMs to perform enhanced graph reasoning and achieve significant improvements in tasks ranging from pure graph computations to real-world molecular property predictions. This work highlights a novel way to unlock latent capabilities within pre-trained LLMs for structured data, emphasizing inference-time efficiency and broad applicability.