Expert Disagreement Annotation: Capturing Genuine Ambiguity in High-Stakes LLM Training Data
LLMs are increasingly being implemented in high-stakes fields, such as medicine, law, finance, and government, where errors or overconfidence can have serious consequences. The reliability of such systems depends not only on the architecture of the models and training methods, but also on the quality of the data used