A recently published evaluation of Utah’s self-built ODR platform for small claims cases has guidance relevant to other courts and ODR developers. The evaluation, conducted by the Innovation for Justice Program at the University of Arizona, examined the usability of the platform as well as the affidavit and summons. It found serious issues at several points in the process that pointed to the need for better design, more information and greater functionality in order for litigants to be able to confidently pursue their case from beginning to end.
In Utah, the vast majority of small claims cases are debt claims. These largely end in default judgment because the defendants don’t appear for their hearing. According to this study, emerging data from the pilot sites indicated that only 36% of defendants registered on the ODR platform. Of those 36% of cases, 50% achieved a settlement or voluntary dismissal. In anticipation of a statewide launch of the platform, the evaluation was undertaken to determine what changes could be made to improve the platform, potentially increasing usage and get better outcomes.
The research team first interviewed stakeholders, including potential defendants, about the platform before designing the evaluation. It then assessed usability by observing eight people who were demographically similar to debt claim defendants in Utah. Those eight people were observed doing specific tasks from the receipt of the affidavit and summons through the end of the ODR process. They were asked to think aloud while they performed tasks that were based on a script written by the research team. The team also video-recorded them as they performed the tasks to further observe their expressions and movements to identify confusion, confidence, frustration, etc.
Once the baseline testing was completed, the research team conducted a series of workshops with low income members of the community in Pima County, Arizona, to identify areas for improvement in the design of both the affidavit and summons and the ODR platform. With the information from both baseline testing and the workshops, the research team then redesigned the documents and platform, then tested the redesigns on another group of eight people. With this third group, the research team did the same type of assessment as it did with the baseline group to determine whether their redesign improved the usability of the platform.
The initial group of eight were stepped through 11 tasks. In five of these, a majority of test participants were unable to complete the task without help due to lack of information, design issues or functionality problems. The problems began at the very beginning, with participants not knowing what to do with the summons and continued with problems finding help on the platform, registering on the platform, sharing documents and reviewing the written agreement.
The first critical issue identified was that the test participants could not fully understand the affidavit and summons. Three of the eight didn’t understand they could register for ODR and a fourth didn’t understand that the URL provided enabled them to participate in ODR. Further, only one of the eight identified all of the options available to them (participate in ODR, ask to be excused from ODR, right to a jury trial, and so forth). People also had difficulty typing the URL into their cellphones.
The second critical issue the participants encountered was failing to find help information on the platform. Only one of the eight participants was able to find the “Help” PDF. Further, the test participants wanted more information than they were provided. They wanted to be told what ODR was on the home page, wanted legal terms to be defined and more information on how to use the platform.
The next failure occurred when participants attempted to register. Only one test participant was able to complete the process without help. Common errors included typing in their name or case number incorrectly, failing to notice the system requirements for a password and not understanding the meaning of the terms plaintiff and defendant. In addition, several participants couldn’t find the corresponding information on the affidavit and summons that they needed in order to register.
The final critical issues the test participants faced were when they attempted to upload documents and review and sign the agreement. The problems in those instances were with the functionality of the technology, however, participants also had trouble figuring out where they could access the document upload function.
Interestingly, the test participants were best able to use the chat function to negotiate and come up with a payment plan. Despite this, they needed more information to do this well, including a better understanding of the ODR facilitator’s role and how to interact with the facilitator. They wanted a way to chat with the facilitator individually (without the other party) and wanted the ODR facilitator to start the chat.
At the end of their testing and redesign, the research team made the following recommendations:
- Employ best practices for URL formation, website naming, user interface design, and highlight key information on the affidavit and summons.
- Streamline the registration process.
- Simplify document sharing and review, as well as allow users to confirm settlement details and download and print the agreement.
- Improve ODR information and help – include an FAQ button on the home page, include a quick guide and include a welcome video outlining how ODR works. Include closed captioning of the video.
- Clarify legal information and user options.
The research team also included recommendations for further study, which are relevant to anyone designing an ODR platform:
- To increase accessibility, the platform design should comply with World Wide Web Consortium (W3W) and Web Accessibility Initiative (WAI) standards.
- Create informational videos. The videos should be close-captioned and in multiple languages.
- Develop an auto-responses bank containing common chat responses that parties can select as they negotiate with the other party.
- Create a paper ODR quick guide that can be sent with the affidavit and summons. The guide should describe the ODR process, provide legal information, and explain in general terms how the ODR platform works.
- Provide a link or button on each page that provides help for that particular page.
- Provide an integrated interest calculator to help parties check the amount in controversy and submit calculations to the ODR facilitator and other party.
- Integrate a calendar function so the parties and ODR facilitator can identify a mutually agreeable time to chat synchronously.
- Provide the option for each party to chat individually with the ODR facilitator.
- Integrate an AI chatbot to answer questions and alert the ODR facilitator when they are needed.
- Integrate a video-conference feature.
- Employ an integrated exit survey to provide ongoing feedback.
The take-home message from this evaluation is that parties need a lot of information provided in accessible formats. They also require document and platform design that makes it easy to find that information and a platform that is not only easy to use but also flexible.
