Machine learning models are increasingly deployed in high-stakes applications where reliable uncertainty quantification is as important as predictive accuracy. In such settings, learning algorithms must not only generate accurate predictions but also communicate their uncertainty in a statistically principled manner.
The first part of the research investigates predictive inference under imperfect training data. It first analyzes the robustness of conformal prediction and risk-controlling procedures to noisy labels, identifying conditions under which these methods construct reliable estimates with theoretical guarantees despite corrupted observations. Building upon these theoretical insights, the research studies predictive inference where additional features, referred to as privileged information, are available during training but unavailable at deployment. We develop calibration methods that account for the distribution shifts induced by missing or corrupted data: one based on distribution re-weighting and one relying on uncertainty-preserving imputation. We analyze the robustness of weighting-based calibration approaches to inaccuracies in the estimated weights and propose a triply robust calibration strategy that provides statistically valid prediction sets under complementary assumptions.
The second part considers uncertainty quantification for modern interactive AI systems, with a particular focus on large language models (LLMs). We formulate the evaluation of model safety and utility of agentic systems or LLMs as a time-to-event prediction problem and develop conformal survival methods that construct statistically valid predictive bounds on the number of interactions required before events of interest occur, such as an unsafe response of an LLM or a successful task completion of an agent. To improve the statistical efficiency of these procedures under limited computational resources, we introduce a theoretically valid dynamic budget allocation framework for sequential evaluation, enabling adaptive allocation of computational resources while preserving finite-sample distribution-free guarantees of the predictive bounds.
Collectively, the contributions presented in this research extend the scope of reliable predictive inference from classical supervised learning to increasingly practical machine learning settings.
