Home
Google Scholar · Nov 2026
Sher Badshah
PhD Candidate in Computer Science
HyperMatrix Lab, Dalhousie University
Advisor: Prof. Hassan Sajjad
Email: sh545346@dal.ca
Research Interests
- LLM reliability
- Uncertainty quantification
- Conformal prediction
- Weight-space programming
- RL and post-training
Google Scholar · LinkedIn · CV
Welcome
I am a PhD candidate in Computer Science at Dalhousie University, where I work in the HyperMatrix Lab with Prof. Hassan Sajjad. My research is about making large language models reliable. A model that is right most of the time is still hard to deploy if we cannot tell when it is wrong. I study how to measure what a model does not know and how to act on that signal.
Much of my work is in uncertainty quantification. I develop uncertainty estimates for LLM outputs and combine them with conformal prediction, so a system can answer, seek more evidence, or abstain while keeping its error rate below a level the user sets. I have applied these ideas to LLM evaluation, where they come with formal guarantees on the decisions a model accepts.
I am also interested in weight-space programming, a new area that reads, edits, and transfers what a network has learned by working on its weights directly. My current project explores whether a skill learned by one model can be found in its weights and moved into another. Another interest is reinforcement learning and post-training of language models. Before NLP, I worked on empirical software engineering, with papers on DevOps, software processes, and AI ethics.
Recent Publications, Talks & News
- November 2026PhD thesis defence, Improving the Reliability of Large Language Models as Judges, Dalhousie University (Nov 25).
- October 2026REFRAME: A Test Suite for the Knowledge–Deployment Gap in LLMs - EMNLP 2026 main conference.
- 2026SCOPE: Selective Conformal Optimized Pairwise LLM Judging - accepted at ICML 2026, Seoul. Certified error control for pairwise LLM judges.
- 2026Judge, Retrieve, or Abstain - accepted at COLM 2026. Judges that retrieve or abstain with provable risk guarantees.
- 2026SAGE: A Search-Augmented Evaluation of LLMs on Free-Form QA - accepted at ACL 2026 (main conference).
- July 2026Invited talk, Uncertainty-Guided LLM Judging, at a workshop in Seoul, Korea, held alongside ICML 2026.
- 2025CLEV: Lightweight Efficient Voting for Free-Form QA - Findings of IJCNLP-AACL 2025.
- 2025Reference-Guided Verdict: LLMs-as-Judges in Automatic Evaluation of Free-Form QA - Widening NLP Workshop 2025.
- September 2025Started as Research Assistant with Dr. Deena Abul-Fottouh, Department of Sociology and Social Anthropology, Dalhousie.
- 2024Quantifying the Capabilities of LLMs Across Scale and Precision - arXiv preprint.
- 2023Best Poster Award, Dalhousie University.