Methodology

Claims are extracted from LLM responses using our ClaimExtractor, then verified against ground-truth evidence using a cross-encoder NLI model (DeBERTa-v3). Each claim is classified as entailment, contradiction, or neutral. The factual accuracy score is the ratio of entailed claims to total verifiable claims.

Numerical claims (interest rates, amounts, percentages) are extracted and compared against known correct values with domain-specific tolerance thresholds. Financial calculations are independently verified.

Our EvidenceMatcher uses sentence-transformers to compute semantic similarity between each claim and the source evidence documents. Higher relevance scores indicate better grounding in factual sources.

Each model is queried 3 times with the same prompt (temperature=0.3). Pairwise cosine similarity between responses measures self-consistency. Inconsistent models are more likely to hallucinate unpredictably.

When available, token-level logprobs are used to compute entropy-based uncertainty scores. Well-calibrated models express higher uncertainty on questions they are likely to get wrong. For models without logprobs support, text-based heuristic analysis is used as fallback.