Home / AI Research / AltRA-Test: An Alternative Annotator Test for LLM Judges Without Shared Annotators
ALAI Research
AltRA-Test: An Alternative Annotator Test for LLM Judges Without Shared Annotators
by Technion · Nagham Omar, Rotem Dror, Almog Gueta, Avigdor Gal, Roi Reichart, Nitay Calderon
Large language models (LLMs) are increasingly used as judges and graders to replace human annotators, raising the question of when such replacement is statistically justified. To support such decisions, the Alternative Annotator Test (Alt-Test) was introduced for dense settings where all items are annotated by the same annotators. However, in many large-scale annotation settings, items are annotated by different subsets of annotators. We therefore propose the Alternative Annotator Test with Resampled Annotations (AltRA-Test), a statistical test that does not require that all items are annotated by the same annotators. AltRA-Test uses a bootstrap that resamples items, draws k human…
Upvotes23
Comments3
Published18 Nov 2026