The field stops treating fact-checking as a single classification problem and starts treating it as a controlled evidence-gathering process.
Every paper fixes a different mismatch between benchmark fact-checking and real fact-checking: realism, evidence evaluation, cost, multi-hop decomposition, and source coverage.
The field stops treating fact-checking as a single classification problem and starts treating it as a controlled evidence-gathering process.
These datasets typically rely on information modified or extracted from Wikipedia.
| Dataset | Size | Labels | Task / Speciality |
|---|---|---|---|
| FEVER | ~185,400 claims | Supported, Refuted, Not Enough Info | Standard single-hop fact verification against Wikipedia; claims are human-written. |
| HOVER | ~26,171 claims; 18,171 train, 4,000 dev, 4,000 test | Supported, Not-Supported | Multi-hop reasoning across 2-4 Wikipedia articles. |
| VitaminC | ~400,000 pairs | Supports, Refutes, Neutral | Robust verification with contrastive evidence pairs and subtle factual changes. |
| FEVEROUS | N/A | N/A | Verification over unstructured text and structured Wikipedia tables. |
| WikiFactCheck | N/A | N/A | Claims checked specifically against Wikipedia as a source. |
These use claims from news organizations, social media, or political speech, often with open-web retrieval.
| Dataset | Size | Labels | Task / Speciality |
|---|---|---|---|
| AVeriTeC | 4,568 claims; 3,068 train, 500 dev, 1,000 test | Supported, Refuted, Conflicting Evidence / Cherry-picking, Not Enough Evidence | Real-world claims verified through QA decomposition and web evidence; avoids temporal leakage. |
| LIAR | 12,836 statements | 6 fine-grained truthfulness levels | Short PolitiFact statements for fake-news detection. |
| LIAR-PLUS | ~12,791 total | 6 fine-grained labels | LIAR extension with automatically extracted justifications and evidence. |
| LIAR-New | N/A | Binarized Supported / Refuted | Updated post-2021 PolitiFact claims to reduce LLM pretraining leakage. |
| RAWFC | 2,012 total | 3 classification categories | Fake-news detection with weakly structured evidence. |
| PolitiHop | 933 total | 3 classification categories | Multi-hop fact-checking focused on political claims. |
| Factcheck-Bench | 631 claims | Supported, Partially Supported, Not Supported, Refuted | Fine-grained fact-checker evaluation at claim, sentence, and document levels. |
| FacTool-QA | 233 instances | True, False | FacTool subset for claims requiring world knowledge. |
| FELM-WK | 184 instances | True, False | LLM factuality benchmark for world knowledge, including some ungrounded claims. |
| BingCheck | 142 sampled instances | Binary, binarized from original 4 labels | Checks factuality of LLM responses using web-search tools. |
| MultiFC | N/A | N/A | Large multi-domain dataset from 26 fact-checking organizations. |
| Covid-Fact | N/A | N/A | COVID-19 fact extraction and verification sourced from Reddit. |
These focus on specific formats, such as charts and graphs, or domains, such as science and health.
| Dataset | Size | Labels | Task / Speciality |
|---|---|---|---|
| PubHealth | N/A | Binarized Supported / Refuted | Public-health claims paired with expert-curated evidence paragraphs. |
| SciFact | N/A | N/A | Scientific claims verified against research abstracts. |
| ChartCheck | N/A | N/A | Explainable fact-checking over real-world chart images. |
| FactKG | N/A | N/A | Verification over knowledge graphs, assuming evidence exists inside the KG. |
| ClaimPKG | N/A | N/A | Graph-based reasoning with pseudo-subgraph generation. |
| HealthVer | N/A | N/A | Evidence-based fact-checking for health-related claims. |
| XFact | N/A | N/A | Multilingual fact-checking benchmark. |
| Model | Q only | Q + A | Veracity @ (.2/.25/.3) | Justifications @ (.2/.25/.3) | ||||
|---|---|---|---|---|---|---|---|---|
| No search | 0.19 | 0.11 | 0.03 | 0.02 | 0.01 | 0.02 | 0.01 | 0.01 |
| Gold evidence | 1.00 | 1.00 | 0.49 | 0.49 | 0.49 | 0.28 | 0.28 | 0.28 |
| AVeriTeC-BLOOM-7b | 0.26 | 0.21 | 0.23 | 0.15 | 0.00 | 0.11 | 0.07 | 0.05 |
| gpt-3.5-turbo | 0.29 | 0.16 | 0.17 | 0.10 | 0.06 | 0.06 | 0.04 | 0.02 |
| Model | S | R | C | N | Macro |
|---|---|---|---|---|---|
| No evidence | .30 | .22 | .00 | .16 | .17 |
| Gold evidence | .48 | .74 | .15 | .59 | .49 |
| AVeriTeC | .41 | .69 | .10 | .16 | .23 |
| gpt-3.5-turbo | .62 | .71 | .02 | .20 | .39 |
S = supported, R = refuted, C = conflicting evidence / cherry-picking, N = not enough evidence.
| Framework | LLM | FacTool-QA | FELM-WK | BingCheck | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Label = True | Label = False | Label = True | Label = False | Label = True | Label = False | ||||||||||||||
| Prec | Recall | F1 | Prec | Recall | F1 | Prec | Recall | F1 | Prec | Recall | F1 | Prec | Recall | F1 | Prec | Recall | F1 | ||
| Random | - | 0.81 | 0.47 | 0.59 | 0.28 | 0.64 | 0.39 | 0.75 | 0.49 | 0.59 | 0.30 | 0.57 | 0.39 | 0.77 | 0.67 | 0.72 | 0.40 | 0.52 | 0.45 |
| Always True | - | 0.76 | 1.0 | 0.86 | 0 | 0 | 0 | 0.72 | 1.0 | 0.84 | 0 | 0 | 0 | 0.70 | 1.0 | 0.83 | 0 | 0 | 0 |
| Always False | - | 0 | 0 | 0 | 0.24 | 1.0 | 0.39 | 0 | 0 | 0 | 0.28 | 1.0 | 0.44 | 0 | 0 | 0 | 0.30 | 1.0 | 0.46 |
| FacTool | GPT-4o | 0.88 | 0.81 | 0.84 | 0.52 | 0.66 | 0.58 | 0.69 | 0.53 | 0.60 | 0.57 | 0.73 | 0.64 | 0.86 | 0.57 | 0.68 | 0.43 | 0.79 | 0.56 |
| FacTool | GPT-4o-mini | 0.92 | 0.68 | 0.78 | 0.45 | 0.82 | 0.58 | 0.67 | 0.37 | 0.48 | 0.51 | 0.78 | 0.62 | 0.92 | 0.55 | 0.69 | 0.45 | 0.88 | 0.60 |
| Factcheck-GPT | GPT-4o | 0.90 | 0.79 | 0.84 | 0.52 | 0.71 | 0.60 | 0.67 | 0.68 | 0.67 | 0.61 | 0.61 | 0.61 | 0.85 | 0.70 | 0.77 | 0.50 | 0.71 | 0.59 |
| Factcheck-GPT | GPT-4o-mini | 0.85 | 0.80 | 0.82 | 0.47 | 0.56 | 0.51 | 0.61 | 0.50 | 0.55 | 0.51 | 0.62 | 0.56 | 0.88 | 0.78 | 0.83 | 0.60 | 0.76 | 0.67 |
| Safe | GPT-4o | 0.92 | 0.88 | 0.90 | 0.66 | 0.77 | 0.71 | 0.70 | 0.80 | 0.75 | 0.72 | 0.60 | 0.65 | 0.84 | 0.90 | 0.87 | 0.71 | 0.60 | 0.65 |
| Safe | GPT-4o-mini | 0.92 | 0.82 | 0.87 | 0.58 | 0.79 | 0.67 | 0.61 | 0.76 | 0.68 | 0.61 | 0.44 | 0.51 | 0.86 | 0.81 | 0.84 | 0.60 | 0.69 | 0.64 |
| FIRE | GPT-4o | 0.92 | 0.88 | 0.90 | 0.65 | 0.71 | 0.68 | 0.70 | 0.86 | 0.77 | 0.77 | 0.54 | 0.63 | 0.86 | 0.88 | 0.87 | 0.70 | 0.67 | 0.68 |
| FIRE | GPT-4o-mini | 0.87 | 0.88 | 0.87 | 0.60 | 0.59 | 0.59 | 0.63 | 0.82 | 0.71 | 0.67 | 0.44 | 0.53 | 0.87 | 0.91 | 0.88 | 0.74 | 0.67 | 0.70 |
| FIRE (No Reason) | GPT-4o | 0.88 | 0.86 | 0.87 | 0.60 | 0.64 | 0.62 | 0.70 | 0.85 | 0.77 | 0.77 | 0.58 | 0.66 | 0.85 | 0.89 | 0.87 | 0.70 | 0.62 | 0.66 |
| FIRE (No Reason) | GPT-4o-mini | 0.87 | 0.84 | 0.86 | 0.55 | 0.61 | 0.58 | 0.65 | 0.84 | 0.73 | 0.71 | 0.47 | 0.57 | 0.84 | 0.87 | 0.85 | 0.66 | 0.6 | 0.62 |
| FIRE (No Search) | GPT-4o | 0.86 | 0.87 | 0.88 | 0.61 | 0.54 | 0.57 | 0.69 | 0.86 | 0.77 | 0.77 | 0.55 | 0.65 | 0.86 | 0.91 | 0.88 | 0.79 | 0.64 | 0.71 |
| FIRE (No Search) | GPT-4o-mini | 0.84 | 0.84 | 0.84 | 0.49 | 0.48 | 0.49 | 0.61 | 0.86 | 0.72 | 0.7 | 0.36 | 0.48 | 0.83 | 0.9 | 0.87 | 0.71 | 0.57 | 0.63 |
| Framework | LLM | LLM cost | Search cost | Time |
|---|---|---|---|---|
| FacTool | GPT-4o | 24.76 | 3.67 | 2.92 |
| FacTool | GPT-4o-mini | 1.49 | 3.67 | 2.34 |
| Factcheck-GPT | GPT-4o | 21.41 | - | 4.25 |
| Factcheck-GPT | GPT-4o-mini | 1.28 | - | 4.09 |
| Safe | GPT-4o | 6.34 | 2.93 | 4.62 |
| Safe | GPT-4o-mini | 0.43 | 2.93 | 4.25 |
| FIRE | GPT-4o | 3.35 | 0.60 | 1.31 |
| FIRE | GPT-4o-mini | 0.14 | 0.20 | 1.25 |
| FIRE (No Reason) | GPT-4o | 1.65 | 0.68 | 0.57 |
| FIRE (No Reason) | GPT-4o-mini | 0.07 | 0.59 | 0.54 |
| FIRE (No Search) | GPT-4o | 1.70 | - | 1.03 |
| FIRE (No Search) | GPT-4o-mini | 0.11 | - | 1.34 |
Cost is in USD and time is in hours for evaluating 559 atomic claims in FacTool-QA, FELM-WK, and BingCheck.
| Scorer | VitaminC | FEVER | AVeriTeC | Avg | ||||
|---|---|---|---|---|---|---|---|---|
| ρ | r | ρ | r | ρ | r | |ρ| | |r| | |
| Reference-based (Precision) | ||||||||
| GPT4o | .203 | .200 | .140 | .129 | .278 | .256 | .207 | .195 |
| Gemini-Pro | .222 | .216 | .139 | .118 | .237 | .177 | .199 | .170 |
| Gemini-Flash | .202 | .198 | .106 | .079 | −.290 | −.205 | .199 | .161 |
| Llama 3.1 | −.037 | −.036 | .110 | .109 | .216 | .210 | .121 | .118 |
| Reference-based (Recall) | ||||||||
| GPT4o | .229 | .228 | .146 | .114 | .314 | .284 | .230 | .209 |
| Gemini-Pro | .253 | .246 | .176 | .143 | .285 | .278 | .238 | .222 |
| Gemini-Flash | .232 | .227 | .032 | .000 | −.190 | −.039 | .151 | .089 |
| Llama 3.1 | .020 | .025 | −.011 | −.012 | .128 | .121 | .053 | .053 |
| Reference-based (F1) | ||||||||
| GPT4o | .215 | .213 | .143 | .121 | .295 | .269 | .218 | .202 |
| Gemini-Pro | .237 | .230 | .155 | .129 | .259 | .216 | .217 | .193 |
| Gemini-Flash | .216 | .212 | .049 | .000 | −.230 | −.066 | .172 | .115 |
| Llama 3.1 | .087 | .164 | −.024 | −.027 | .161 | .154 | .074 | .073 |
| Proxy-reference | ||||||||
| Proxy-ref | .456 | .479 | .290 | .480 | .487 | .459 | .411 | .473 |
Reference-based scores use different LLM backbones; proxy-reference uses a DeBERTa-based component.
| Metric | Ref-less baselines | Proxy-ref | Ref-based baselines | Ev2R | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-4o | Gem-Pro | Gem-Flash | Llama | DeBERTa | RougeL | BLEU | Meteor | H-Met | GPT-4o | Gem-Pro | Gem-Flash | Llama | |
| COV (ρ) | .237 | .287 | .275 | .297 | .338 | .150 | .236 | .229 | .005 | .321 | .341 | .221 | .328 |
| COV (r) | .261 | .296 | .287 | .286 | .348 | .169 | .184 | .240 | −.024 | .323 | .326 | .299 | .336 |
| REL (ρ) | .292 | .203 | .287 | .227 | .298 | .086 | .107 | .062 | .008 | .297 | .278 | .403 | .224 |
| REL (r) | .360 | .263 | .333 | .250 | .374 | .099 | .079 | .076 | .003 | .332 | .315 | .404 | .277 |
| Dataset / Test | Reference-less baselines | Ev2R | Proxy-ref | ||||||
|---|---|---|---|---|---|---|---|---|---|
| METEOR | ROUGE | BLEU | BLEURT | GPT4o | Prec | Rec | Proxy-comp. | DeBERTa | |
| Semantics-altering tests | |||||||||
| Completeness | −30.56 | −53.19 | −77.87 | −143.70 | −91.76 | −21.2 | −59.5 | −39.35 | −82.68 |
| Random shuffle | −32.32 | −89.94 | −94.20 | −1.02 | 0.55 | −6.8 | −6.0 | −32.22 | −92.32 |
| Average | −31.44 | −71.57 | −86.04 | −72.36 | −45.61 | −14.0 | −32.75 | −35.79 | −87.50 |
| Semantics-preserving tests | |||||||||
| Invariance contraction | −0.53 | −0.97 | −1.35 | −1.65 | 0 | −0.8 | −0.7 | −0.85 | 0.02 |
| Invariance num2text | −8.34 | −10.24 | −14.10 | −22.13 | −3.85 | 0.1 | 0 | −4.02 | −35.64 |
| Invariance text2num | −0.04 | −0.90 | −1.15 | −0.81 | −4.40 | 0 | −0.3 | −1.59 | 0.01 |
| Invariance synonyms | −8.22 | −36.64 | −51.83 | −51.00 | −2.75 | −11.6 | −11.3 | −15.35 | −85.71 |
| Redundancy sent | −26.16 | −46.52 | −56.15 | −80.45 | 3.30 | 0 | −0.1 | −5.04 | −67.95 |
| Redundancy words | −5.73 | −10.68 | −32.69 | −31.90 | −2.75 | −1.0 | 0.2 | −2.89 | −52.55 |
| Fluency | −11.52 | −23.33 | −29.26 | −41.69 | −2.20 | −4.3 | −3.9 | −2.6 | −57.12 |
| Noise | −31.82 | −31.81 | −31.81 | −21.20 | −1.65 | −28.7 | −2.2 | 2.89 | −62.42 |
| Argument structure | −0.01 | 0 | 0 | 0 | 1.65 | 0 | 0 | −0.74 | −28.93 |
| Average | −10.33 | −18.43 | −23.94 | −27.87 | −15.04 | −5.16 | −2.9 | −3.78 | −39.79 |
Values are score differences, in percent, between initial evidence and manipulated evidence. The evaluated Ev2R reference-based scorer uses GPT4o.
| System | Supp F1 | Ref F1 | NEI F1 | Conf F1 | Acc | AVeriTeC 0.25 |
|---|---|---|---|---|---|---|
| AllAtOnce | .591 | .813 | 0 | 0 | .705 | .340 |
| LLM+LLM | .644 | .821 | 0 | 0 | .720 | .385 |
| Seq+Seq | .638 | .816 | 0 | 0 | .715 | .370 |
| 4 class | .486 | .593 | .148 | .069 | .415 | .245 |
| No late verdict | .643 | .811 | 0 | 0 | .705 | .450 |
| No long doc | .577 | .819 | 0 | 0 | .705 | .465 |
| Multi-doc | .673 | .837 | 0 | 0 | .735 | .460 |
| No metadata | .575 | .810 | 0 | 0 | .700 | .410 |
| No paraphrase | .701 | .839 | 0 | 0 | .745 | .225 |
| Repeat not para | .624 | .813 | 0 | 0 | .710 | .340 |
| Algorithm 1 | .716 | .841 | 0 | 0 | .750 | .495 |
| Data | Submission | Supp F1 | Ref F1 | NEI F1 | Conf F1 | Acc | AVeriTeC 0.25 |
|---|---|---|---|---|---|---|---|
| Dev | Algorithm 1 | .698 | .853 | 0 | 0 | .754 | .486 |
| Dev | Inflated to 10 | .698 | .853 | 0 | 0 | .754 | .510 |
| Test | Algorithm 1 | — | — | — | — | — | .445 |
| Test | Inflated to 10 | — | — | — | — | — | .477 |
Best results are bold; second-best results are underlined. Grey-highlighted columns in the original table denote models that use self-retrieved rather than gold evidence.
| Method | Wikipedia sourced | Web sourced | With gold evidence | Overall Avg. (%) | |||||
|---|---|---|---|---|---|---|---|---|---|
| FEVER | HOVER | LIAR-New | AVeriTeC | SummEval | AFact-CNN | PubHealth | Avg. (%) | ||
| Large-scale LLMs | |||||||||
| Gemini-2.5-flash | 54.8 | 62.2 | 59.1 | 61.0 | 69.9 | 64.6 | 70.8 | 68.4 | 63.2 |
| GPT-4 | 51.4 | 62.5 | 58.6 | 60.2 | 69.7 | 60.7 | 73.2 | 67.9 | 62.3 |
| GPT-4o | 55.3 | 63.8 | 60.1 | 61.5 | 76.3 | 66.8 | 67.0 | 70.0 | 64.4 |
| Claude 3.5-Sonnet | 57.1 | 64.4 | 61.3 | 63.0 | 77.3 | 68.8 | 73.8 | 73.3 | 66.5 |
| DeepSeek-V3 67B | 53.5 | 61.7 | 58.8 | 60.5 | 68.3 | 63.2 | 72.9 | 68.1 | 62.7 |
| Small-scale LLMs | |||||||||
| Llama3 8B [7] | 48.2 | 55.1 | 50.3 | 54.7 | 51.7 | 51.3 | 70.7 | 57.9 | 54.6 |
| Qwen2.5 7B [46] | 50.1 | 57.6 | 52.9 | 56.3 | 58.5 | 45.3 | 59.1 | 54.3 | 54.3 |
| Llama3.3 70B [7] | 57.4 | 61.2 | 60.5 | 62.1 | 57.6 | 53.5 | 76.9 | 62.7 | 61.3 |
| Qwen2.5 72B [46] | 58.1 | 62.0 | 59.7 | 61.4 | 53.4 | 49.9 | 76.7 | 60.0 | 60.2 |
| Specialized methods | |||||||||
| HerO [49] | 67.5 | 58.1 | 70.2 | 71.4 | 65.8 | 62.4 | 70.3 | 66.2 | 66.5 |
| GraphRAG (GPT-4o) [8] | - | - | - | - | 68.2 | 60.4 | 74.2 | 67.6 | - |
| GraphCheck [5] | - | - | - | - | 66.3 | 61.9 | 70.9 | 66.4 | - |
| FIRE [45] | 90.6 | 67.0 | 72.8 | 72.8 | 58.2 | 56.9 | 64.1 | 59.7 | 68.9 |
| WKGFC(Ours) | 91.9 | 72.8 | 81.3 | 73.2 | 69.8 | 58.4 | 72.8 | 67.0 | 74.3 |
Benchmarks often used artificial claims, closed Wikipedia evidence, weak intermediate reasoning, and sometimes evidence published after the claim.
Journalistic claims need context, temporal discipline, conflicting evidence handling, and evidence that can explain a verdict.
Normalize the claim, then annotate question-answer evidence with textual justifications. Verification becomes a research path.
English-only. The metric can check evidence usefulness, but cannot fully guarantee source-answer consistency.
| AVeriTeC label | Meaning |
|---|---|
| Supported | Evidence backs the normalized claim. |
| Refuted | Evidence contradicts it. |
| Conflicting Evidence / Cherry-picking | Both support and refutation appear, or the claim selectively uses evidence. |
| Not Enough Evidence | Available evidence is insufficient. |
AVeriTeC changes the unit of work. The system is no longer rewarded for only choosing a label. It has to recover the questions a fact-checker would ask.
Systems were judged by final verdict accuracy or evidence overlap with reference text. That works better in closed-source settings than on the web.
Two systems can retrieve different evidence and both be right. Exact text matching can punish valid evidence.
Ev2R combines reference-based, proxy-reference, and reference-less evidence scorers.
Evaluated on AVeriTeC, FEVER, and VitaminC.
The contribution is not a new fact-checker. It is a better ruler for measuring evidence retrieval.
| Scorer type | What it checks |
|---|---|
| Reference-based | Does retrieved evidence match the atomic facts in gold evidence? |
| Proxy-reference | Does evidence support the predicted verdict label? |
| Reference-less | Is evidence relevant to the input claim without needing a gold passage? |
Evaluation on AVeriTeC, FEVER, and VitaminC showed high correlation with human judgments on criteria such as verdict agreement.
Ev2R matters because open-web fact-checking creates multiple acceptable trails. The metric moves the field away from brittle text overlap and toward evidence sufficiency.
Atomic claims were checked with a fixed retrieval budget, then verified. The same pipeline ran even when the model already had enough information.
At scale, fact-checking becomes expensive. Fixed retrieval also does not imitate how humans search, pause, and search again only when needed.
FIRE integrates retrieval and verification in one loop. The agent decides whether to answer or issue another search query based on confidence.
FacTool-QA, FELM-WK, Factcheck-Bench, and BingCheck.
The confidence decision is compressed into the agent loop for cost reasons. Splitting confidence, retrieval, and verification could make behavior more interpretable, but may cost more.
FIRE is the efficiency chapter. It asks a practical question: why retrieve ten things for every claim if the system can decide when retrieval is actually necessary?
Complex claims were often decomposed statically or checked through coarse evidence retrieval. This can cause noisy evidence and accumulated reasoning errors.
AFEV extracts one atomic fact, retrieves and reranks evidence, verifies it, then uses the result to decide the next atomic fact.
On HOVER, the model follows the multi-hop chain by verifying fragments sequentially across articles instead of treating the whole claim as one retrieval query.
Five retrieval candidates, two reranked evidence pieces, and one dynamic demonstration per atomic fact.
| Dataset | Split / size used | Reported result |
|---|---|---|
| LIAR-PLUS | 10,240 / 1,284 / 1,267 | LA 83.73, Macro-F1 83.12 |
| HOVER | 18,171 / 4,000 / 4,000 | LA 78.87, Macro-F1 78.76 |
| PolitiHop | 592 / 141 / 200 | LA 74.14, Macro-F1 57.69 |
| LIAR + RAWFC | LIAR 10,269 train; RAWFC 1,612 train | Reported SOTA over LLM baselines such as RAFTS on both. |
AFEV is the “do not verify the whole thing at once” paper. It turns a complex claim into a chain of smaller verified commitments.
RAG systems relied heavily on semantic similarity over text. KG systems assumed that the right evidence existed inside the graph.
Real claims operate under open-world evidence conditions. A fact may be missing from a KG, while web evidence may be relevant but unstructured.
WKGFC frames fact-checking as a partially observable decision process. The agent expands a KG, searches the web when needed, aligns passages into triplets, then decides.
It uses self-reflection and TextGrad-style prompt optimization, keeping the base LLM frozen.
| Evaluation category | Datasets |
|---|---|
| Wikipedia-based | FEVER, HOVER |
| Web-sourced | LIAR-New, AVeriTeC |
| Provided evidence / summaries | SummEval-series, AggreFact-CNN, PubHealth |
Insufficient KG coverage, maximum-step exhaustion, and over-confident early verdicts.
WKGFC brings the story into agentic retrieval. It does not choose between structured and unstructured evidence. It uses a graph as the anchor and the web as the expansion layer.
The final truth label for the claim: supported, refuted, conflicting evidence / cherry-picking, or not enough evidence. In AVeriTeC, the veracity score is only awarded when retrieval clears the evidence cutoff.
The free-text explanation for the verdict. It shows how the evidence leads to the label, and AVeriTeC reports justification quality separately from veracity.
A text-similarity score for comparing model output with a reference answer. It gives partial credit for close wording or meaning when valid evidence is phrased differently.
In plain language: veracity answers “what is the truth status?” Justification answers “why is that the verdict?” METEOR gives partial credit when the model’s evidence or explanation is close to the gold reference.
| Paper | Main question | Technical move | Limit left open |
|---|---|---|---|
| AVeriTeC | Does the benchmark resemble real fact-checking? | Real claims, QA evidence, four labels. | English scope, imperfect evidence-answer consistency checks. |
| Ev2R | How do we judge evidence on the open web? | Evidence scoring beyond exact gold overlap. | Metric quality still depends on language and scorer reliability. |
| FIRE | How do we make LLM fact-checking cheaper? | Iterative retrieve-or-answer loop. | Confidence and stopping behavior remain hard to inspect. |
| AFEV | How do we verify complex claims? | Iterative atomic fact extraction, reranking, dynamic demonstrations. | More steps increase compute and can still inherit evidence quality issues. |
| WKGFC | How do we combine structured and open evidence? | KG-first agent with web augmentation and prompt-level policy optimization. | KG incompleteness, max-step failures, over-confident stopping. |
The conclusion is simple: modern AFC is less about one smarter classifier and more about when to search, where to search, how to decompose the claim, and when the evidence is enough.
Sources used: AVeriTeC, Schlichtkrull et al., 2023, arXiv:2305.13117 / NeurIPS Datasets and Benchmarks 2023.
Ev2R, Akhtar et al., arXiv:2411.05375, 2024; TACL article, 2026.
FIRE, Xie et al., Findings of NAACL 2025, DOI 10.18653/v1/2025.findings-naacl.158.
Fact in Fragments / AFEV, Zheng et al., arXiv:2506.07446, 2025.
WKGFC, Multi-Sourced, Multi-Agent Evidence Retrieval for Fact-Checking, arXiv:2603.00267, 2026.
Dataset details also reflect the provided notes for LIAR, HOVER, PolitiHop, RAWFC, PubHealth, Factcheck-Bench, FacTool-QA, FELM-WK, BingCheck, SummEval, AggreFact-CNN, XFact, ChartCheck, DanFEVER, FakeCovid, and SciFact.