Factual reliability remains a critical bottleneck for LLMs, whose tendency to hallucinate undermines user trust. To mitigate this problem, we need methods to scientifically measure what a model knows and why it makes factual mistakes. However, many fundamental research questions in this space are not trivial to test, demanding the design of dedicated, controlled experiments precisely tailored to isolate the relevant phenomenon. Using three recent papers as case studies, we will explore the full research cycle: from defining the research question and formulating a hypothesis, to designing a controlled study to test it.
The papers we will discuss are:
[1] Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? (EMNLP 2024)
[2] Inside-Out: Hidden Factual Knowledge in LLMs (COLM 2025)
[3] Thinking to Recall: How Reasoning Unlocks Parametric Knowledge in LLMs (COLM 2026)