Enterprise AI Analysis

NABAOS: Practical Epistemic Verification

NABAOS addresses the semantic integrity problem: are the claims in the output grounded in evidence? Inspired by the Nyāya school of Indian philosophy, it classifies every LLM claim by its epistemic source (pramāṇa). This provides nuanced trust signals: Pratyaksa (direct tool output), Anumāna (inference), Śabda (external testimony), Abhāva (absence), or Ungrounded (no evidence).

The core mechanism is HMAC-signed tool execution receipts. When an agent requests a tool invocation, the runtime executes it in a sandboxed environment, generates a cryptographically signed receipt containing tool name, input/output hashes, result count, and extracted facts, then stores it. The LLM receives the tool output and receipt identifier. These receipts are unforgeable, detect tampering (e.g., mismatched counts), and provide a complete audit trail of tool interactions.

The NABAOS Verification Workflow

The NABAOS system employs a six-stage verification protocol:

1. User Request: User initiates a query.
2. Tool Execution: Agent runtime executes tool calls, generates HMAC-signed receipts, and stores them.
3. LLM Call: LLM is prompted with user request, tool outputs, receipt IDs, and a VERIFICATION PROMPT to self-classify claims.
4. LLM Response with Self-Tags: LLM generates a response, tagging each factual claim with its pramāṇa category and cited receipt ID.
5. Verification Engine: Processes each tagged claim:
   • Pratyaksa: Verifies receipt signature, claimed count, and facts against the receipt.
   • Anumāna: Checks if cited premises exist in receipt facts.
   • Abhāva: Verifies the tool returned an empty result set.
   • Śabda: Confirms web fetch tool invocation for cited source.
   • Ungrounded: Flags claims with no evidence.
6. Trust-Annotated Output: User receives the response augmented with trust levels (Fully Verified, Mostly Verified, Partial, Unreliable, Ungrounded).

For autonomous agents performing multi-step web tasks where intermediate tool calls aren't controlled, NABAOS employs a Deep Agent Cross-Checking protocol, including schema validation, URL re-fetching, computation replay, temporal consistency checks, and cross-source verification.

Superior Detection & Calibrated Trust

Evaluated on NYAYAVERIFYBENCH, a new multilingual benchmark of 1,800 agent response scenarios with six hallucination types, NABAOS achieved the highest detection rate at 91% with the lowest false positive rate of 4% and a negligible verification overhead of <15 ms per response. This significantly outperforms baselines like Self-Consistency (45% detection, 12% FPR, 3-5s latency) and RAG-Grounding (52% detection, 18% FPR, 1-2s latency).

NABAOS excels particularly in detecting fabricated tool calls (94.2%) and false absence claims (91.3%), which are deterministically verifiable. Its performance is stable across languages (88.7% to 92.8%), thanks to receipt-based verification operating on language-independent structured data.

The system's trust levels are well-calibrated: "Fully Verified" responses are correct 98.7% of the time, while "Unreliable" responses are correct only 23.4% of the time, providing users with actionable and accurate trust signals.

Identified Limitations & Future Directions

While effective, NABAOS has limitations:

• Self-tagging reliance: Relies on LLM compliance with the self-tagging prompt (92% for Claude, 85% for open models).
• Limited content verification: Receipts verify structural properties (counts, facts, hashes), but cannot verify subtle misphrasing or misleading paraphrases of tool output content.
• Cross-checking latency: Deep agent cross-checking adds 200-500 ms, suitable for autonomous agents but not interactive use.
• Self-reporting dependency: Inference-as-fact detection depends on the LLM honestly reporting inferences.
• Adversarial robustness: An adversarially fine-tuned LLM might learn to game self-tagging. However, fabricated tool call and count mismatch detection remain effective.
• Synthetic benchmark: NYAYAVERIFYBENCH uses injected hallucinations; real-world patterns may differ.

NABAOS does not protect against compromised tools (which return incorrect data) or reasoning errors, only that claims are grounded in evidence. It provides one layer of defense in a defense-in-depth strategy.