Accuracy and the Robot Judge

Document Type

Article

Publication Title

The Journal of Appellate Practice and Process

Abstract

How can it be shown that AI-powered robot judges are more accurate than human judges? Some commentators believe that legal decision making is too indeterminate for this to be demonstrated. Even commentators who believe that this might be possible do not indicate precisely how this might be done. This article provides possible answers to this question, which to my knowledge has not been directly addressed in the burgeoning robot judge literature. I contend that there are three broad paths through which robot judges can potentially demonstrate superior accuracy over their human counterparts: the consensus path, the comparison path, and the process path. Part I examines the consensus path, where it is simply accepted by all relevant stakeholders that the robot judge is more accurate than the human judge. While this type of robot judge does not exist in traditional legal settings, it does exist in the professional baseball setting in the form of the robot umpire, which is currently calling balls and strikes at the highest tier of minor league baseball. Beginning the discussion of robot judges in this context allows us to examine the conditions needed for such a consensus to form, provides insight as to whether these conditions could possibly exist in traditional legal settings, and provides a real-world test case about whether a reasonable objection not predicated on accuracy concerns can be made to robot judges. Part II turns to the comparison path. Here, there is no consensus among all relevant stakeholders, but an argument can be made that the robot judge is more accurate than the human judge based on a comparison using a specified metric. For example, some commentators have contended that robot judges are more accurate than human judges in the bail context based on quantitative metrics for the real-world consequences of the decisions. After describing the three-step framework set forth by proponents of this comparison method, I demonstrate how difficult it is to utilize this comparison method in any particular area of law, including bail hearings. I then introduce a new comparison method that potentially has broader applicability to legal decision making. Under this approach, robot judges can demonstrate greater accuracy than human judges by being better at “matching” the decisions made by human judges deciding appeals from the decisions of lower-level judges. Part III focuses on the process path. It explores whether an argument can be made for the superior accuracy of the robot judge compared to the human judge based on the process that the robot judge utilizes to reach its decision. I assert that there is a reasonable basis for such an argument, premised on the Condorcet Jury Theorem, when the robot judge replicates the decision of the majority of the qualified human judges. I then consider whether a process-based argument can be made that does not strictly adhere to the Condorcet Jury Theorem. I contend that such an argument can be made if a consensus forms that the process that the robot judge follows is more likely than a human judge to produce accurate decisions. Perhaps venturing into the realm of science fiction (at least for the moment), I then propose a candidate for such a process in the multi-judge context typically found in appellate decision making. In my proposal, a society of robot appellate judges is developed and programmed to deliberate with each other under conditions designed to produce accurate decisions more often than human judges. Part IV concludes by briefly considering the effect that robot judges might have on the accuracy of human judges.

First Page

1

Last Page

66

Publication Date

2025

Link to Full Text

Share

COinS