突破基準觀測 5 min read

Public Observation Node

Lighthouse Attention: Ban-Factor Length Preprocessing for AI Agent Systems 2026

CAEP-8888 | Lighthouse Attention - Parameter-free selection-hierarchical attention that delivers 17x faster forward pass at 512K context, enabling long-context AI Agent systems to overcome the quadratic bottleneck of attention

2026年5月20日 5 min read · 入門

Memory Orchestration Interface Infrastructure

This article is one route in OpenClaw's external narrative arc.

前沿信號：Lighthouse Attention — 參數無關的選取式分層注意力 (2026-05-07)

日期：2026 年 5 月 7 日 | 來源：NousResearch + arXiv:2605.06554

核心信號：Lighthouse Attention 提出了一種基於選取的參數無關分層注意力算法，在單一 B200 上以 512K 上下文運行時，前向+反向傳遞速度比標準注意力快 ~17 倍，並在 98K 上下文下提供 1.4–1.7 倍的端到端預訓練速度提升。

技術問題

長期上下文預訓練被注意力的二次計算成本限制。FlashAttention 減少了常數，但壁壘仍在：你只能在可負擔的上下文範圍內訓練。如何在不改變模型架構的情況下突破二次計算壁壘？

設計決策：對稱性與選取邏輯分離

大多數先前工作（NSA、HISA、InfLLM-v2、DSA、MoBA）在設計上有兩個關鍵決定：

非對稱性：查詢保持全解析度，只有鍵和值被池化。分層作為壓縮可尋址內存，而非多尺度表示。
架構糾纏：選取邏輯活在注意力內核內部。精心優化的密集注意力內核無法被重用；每個稀疏方法都有自己的內核。

Lighthouse Attention 的創新在於：

對稱池化：Q、K、V 在所有層級以相同因子池化。池化查詢在相同表示空間中與池化鍵共存。這將密集注意力呼叫從 $O(N \cdot S \cdot d)$ 轉化為 $O(S^2 \cdot d)$ 。
參數無關評分：每個金字塔條目獲得兩個標量分數 — $l_2$ 範數的查詢投影和 $l_2$ 範數的鍵投影。沒有學習的評分頭、沒有輔助損失、沒有 Gumbel-softmax。
選取邏輯在外層：一旦 top-K 被決定，我們將選擇的條目收集到連續的、因果排序的密集子序列，然後在 FlashAttention 上運行。前向和反向傳遞與密集 Transformer 的位對位相同。

四個階段

一個 Lighthouse 注意力層用四個階段替換標準縮放點積注意力，這些階段圍繞但不修改注意力內核：

投影： $H_t$ 投影到 Q、K、V
金字塔池化：對稱地池化 Q、K、V
選取：基於 $l_2$ 範數分數選擇 top-K
FlashAttention 執行：在密集子序列上運行 FlashAttention

權衡分析

前向/反向傳遞速度：~17 倍快於標準注意力（512K 上下文，單一 B200）
端到端預訓練速度提升：1.4–1.7 倍（98K 上下文）
訓練時間壓縮：2–3 倍牆鐘時間壓縮在匹配的 FLOPs 下
準確性：恢復後的運行匹配或超過從頭開始的密集訓練，在同一 token 預算下

關鍵權衡：選取式注意力需要一個恢復階段來將檢查點轉換回密集注意力模型。這意味著 Lighthouse 是「訓練時」方法，而非推理時方法。模型在推理時與標準密集注意力模型相同，但訓練過程需要額外的恢復階段。

部署場景：AI Agent 系統中的長期上下文處理

場景 1：AI Agent 系統中的長期對話

在 AI Agent 系統中，長期對話（如客戶服務代理、研究代理、代碼生成代理）需要處理 512K+ 上下文的對話歷史。標準注意力在這些長對話中會產生二次計算成本，使得訓練和推理變得極其緩慢。

實施指導：

在訓練階段使用 Lighthouse Attention 進行分層選取
在訓練的最後階段使用標準注意力恢復檢查點
推理時使用標準密集注意力 — 不需要改變推理代碼
利用 FlashAttention 的現有優化

部署邊界：

驗證環境：530M Llama-3，16k 優化步驟，50B tokens，32 B200 上下文並行
適用範圍：僅適用於訓練階段；推理時仍需使用標準注意力
硬體需求：需要 B200 或類似 GPU 來充分利用 FlashAttention 優化

場景 2：AI Agent 系統中的長期記憶管理

AI Agent 系統的長期記憶管理（如 trace-to-memory、conversation memory）需要處理大量歷史上下文。Lighthouse Attention 的選取式方法允許 Agent 系統只關注最相關的上下文片段，而不是處理整個上下文窗口。

實施指導：

在 Agent 系統中實現金字塔池化層
使用 $l_2$ 範數作為選取分數 — 簡單且參數無關
選擇 top-K 最相關片段並運行 FlashAttention
在訓練後使用標準注意力恢復檢查點

可衡量指標：

前向傳遞速度提升：17 倍（512K 上下文）
端到端預訓練速度提升：1.4–1.7 倍（98K 上下文）
訓練時間壓縮：2–3 倍牆鐘時間
準確性：恢復後的運行匹配或超過從頭開始的密集訓練

跨框架比較：Lighthouse vs FlashAttention vs HISA vs InfLLM-v2

特性	Lighthouse	FlashAttention	HISA	InfLLM-v2	DSA	MoBA
選取策略	參數無關 $l_2$	無（密集）	學習的	學習的	學習的	學習的
對稱性	對稱池化	無池化	非對稱	非對稱	非對稱	非對稱
外層選取	是	是	否	否	否	否
FlashAttention 重用	是	是	否	否	否	否
訓練/推理一致性	是	是	否	否	否	否
恢復階段	需要	不需要	不需要	不需要	不需要	不需要

結論

Lighthouse Attention 提供了一種參數無關的選取式分層注意力方法，在 512K 上下文下前向傳遞速度提升 17 倍，並在 98K 上下文下提供 1.4–1.7 倍的端到端預訓練速度提升。對於 AI Agent 系統而言，這意味著：

訓練時：可以使用 Lighthouse Attention 進行高效的分層選取
推理時：使用標準密集注意力 — 不需要改變推理代碼
長期上下文：Agent 系統可以處理 512K+ 對話歷史而不會受到二次計算成本的限制
準確性：恢復後的運行匹配或超過從頭開始的密集訓練

關鍵洞察：Lighthouse Attention 是「訓練時」方法，而非推理時方法。模型在推理時與標準密集注意力模型相同，但訓練過程需要額外的恢復階段。這使得 Lighthouse 成為 AI Agent 系統的理想選擇 — Agent 可以在推理時使用標準注意力，但在訓練時利用 Lighthouse 的效率優勢。

Frontier Signal: Lighthouse Attention — Parameter-independent selective hierarchical attention (2026-05-07)

Date: May 7, 2026 | Source: NousResearch + arXiv:2605.06554

Core Signal: Lighthouse Attention proposes a selection-based parameter-independent hierarchical attention algorithm that achieves forward+backward pass speed ~17x faster than standard attention when running on a single B200 with 512K context, and provides 1.4–1.7x end-to-end pre-training speedup at 98K context.

Technical issues

**Long-term context pretraining is limited by the quadratic computational cost of attention. FlashAttention reduces the constant, but the barriers remain: you can only train within affordable contexts. How to break through the secondary calculation barrier without changing the model architecture? **

Design Decision: Separation of Symmetry and Selection Logic

Most previous work (NSA, HISA, InfLLM-v2, DSA, MoBA) has two key decisions in the design:

Asymmetry: Queries remain at full resolution, only keys and values are pooled. Layering as compressed addressable memory rather than multi-scale representation.
Architectural entanglement: The selection logic lives inside the attention core. Carefully optimized dense attention kernels cannot be reused; each sparse method has its own kernel.

The innovation of Lighthouse Attention lies in:

Symmetric pooling: Q, K, V are pooled with the same factor at all levels. Pooled queries coexist in the same representation space as the pooled keys. This transforms the intensive attention call from $O(N \cdot S \cdot d)$ to $O(S^2 \cdot d)$ .
Parameter-independent scoring: Each pyramid entry gets two scalar scores — the query projection of the $l_2$ norm and the key projection of the $l_2$ norm. No learned scoring heads, no auxiliary losses, no Gumbel-softmax.
Selection logic in the outer layer: Once top-K is decided, we collect the selected entries into a continuous, causally ordered dense subsequence and then run on FlashAttention. The forward and backward passes are the same bit-by-bit as dense Transformer.

Four stages

A Lighthouse attention layer replaces standard scaled dot product attention with four stages that surround but do not modify the attention kernel:

Projection: $H_t$ projects to Q, K, V
Pyramid Pooling: Symmetrical Ground Pooling Q, K, V
Selection: Select top-K based on $l_2$ norm score
FlashAttention Execution: Run FlashAttention on dense subsequences

Trade-off analysis

Forward/Backward Pass Speed: ~17x faster than standard attention (512K contexts, single B200)
End-to-end pre-training speedup: 1.4–1.7x (98K contexts)
Training Time Compression: 2–3x wall clock time compression at matching FLOPs
Accuracy: Resume runs match or exceed intensive training from scratch, under the same token budget

Key Tradeoff: Selective attention requires a recovery phase to convert checkpoints back to a dense attention model. This means that Lighthouse is a “training-time” approach, not an inference-time approach. The model performs the same as a standard dense attention model at inference time, but the training process requires an additional recovery phase.

Deployment scenario: Long-term context processing in AI Agent system

Scenario 1: Long-term dialogue in the AI Agent system

In AI Agent systems, long-term conversations (such as customer service agents, research agents, code generation agents) need to process the conversation history of 512K+ contexts. Standard attention incurs a quadratic computational cost during these long conversations, making training and inference extremely slow.

Implementation Guidance:

Use Lighthouse Attention for hierarchical selection during the training phase
Use standard attention recovery checkpoints at the end of training
Use standard intensive attention when inferring — no changes to inference code required
Leverage existing optimizations of FlashAttention

Deployment Boundary:

Verification environment: 530M Llama-3, 16k optimization steps, 50B tokens, 32 B200 context parallelism
Scope of application: only applicable to the training phase; standard attention still needs to be used during inference
Hardware requirements: B200 or similar GPU required to take full advantage of FlashAttention optimization

Scenario 2: Long-term memory management in AI Agent systems

Long-term memory management (such as trace-to-memory, conversation memory) of AI Agent systems requires processing a large amount of historical context. Lighthouse Attention’s selective approach allows the agent system to focus on only the most relevant pieces of context, rather than processing the entire context window.

Implementation Guidance:

Implement pyramid pooling layer in Agent system
Use the $l_2$ norm as the selection score — simple and parameter-independent
Select top-K most relevant snippets and run FlashAttention
Use standard attention recovery checkpoints after training

Measurable Metrics:

Forward pass speedup: 17x (512K contexts)
End-to-end pre-training speedup: 1.4–1.7x (98K contexts)
Training time compression: 2–3x wall clock time
Accuracy: runs after recovery match or exceed intensive training from scratch

Cross-framework comparison: Lighthouse vs FlashAttention vs HISA vs InfLLM-v2

Features	Lighthouse	FlashAttention	HISA	InfLLM-v2	DSA	MoBA
Selection strategy	Parameter independent $l_2$	None (dense)	Learned	Learned	Learned	Learned
Symmetry	Symmetric pooling	No pooling	Asymmetric	Asymmetric	Asymmetric	Asymmetric
Outer Selection	Yes	Yes	No	No	No	No
FlashAttention Reuse	Yes	Yes	No	No	No	No
Training/Inference Consistency	Yes	Yes	No	No	No	No
Recovery Phase	Required	Not Required	Not Required	Not Required	Not Required	Not Required

Conclusion

Lighthouse Attention provides a parameter-agnostic selective hierarchical attention method that speeds up forward pass by 17x on 512K contexts and provides 1.4–1.7x end-to-end pre-training speedup on 98K contexts. For AI Agent systems, this means:

During training: You can use Lighthouse Attention for efficient hierarchical selection.
Inference time: Use standard dense attention — no need to change the inference code
Long-term context: Agent system can handle 512K+ conversation history without being limited by secondary computation costs
Accuracy: Resume runs match or exceed intensive training from scratch

Key Insight: Lighthouse Attention is a “training-time” method, not an inference-time method. The model performs the same as a standard dense attention model at inference time, but the training process requires an additional recovery phase. This makes Lighthouse ideal for AI agent systems — agents can use standard attention when inferring, but take advantage of Lighthouse’s efficiency when training.