Resources / Study / Innovation for Court ADR

Just Court ADR

The blog of Resolution Systems Institute

Author Archive

« Older Entries
Newer Entries »

Usability Evaluation of Utah’s ODR Platform Provides Insights for Developing Accessible Tools for All

Jennifer Shack, September 28th, 2020

A recently published evaluation of Utah’s self-built ODR platform for small claims cases has guidance relevant to other courts and ODR developers. The evaluation, conducted by the Innovation for Justice Program at the University of Arizona, examined the usability of the platform as well as the affidavit and summons. It found serious issues at several points in the process that pointed to the need for better design, more information and greater functionality in order for litigants to be able to confidently pursue their case from beginning to end.

In Utah, the vast majority of small claims cases are debt claims. These largely end in default judgment because the defendants don’t appear for their hearing. According to this study, emerging data from the pilot sites indicated that only 36% of defendants registered on the ODR platform. Of those 36% of cases, 50% achieved a settlement or voluntary dismissal. In anticipation of a statewide launch of the platform, the evaluation was undertaken to determine what changes could be made to improve the platform, potentially increasing usage and get better outcomes. 

The research team first interviewed stakeholders, including potential defendants, about the platform before designing the evaluation. It then assessed usability by observing eight people who were demographically similar to debt claim defendants in Utah. Those eight people were observed doing specific tasks from the receipt of the affidavit and summons through the end of the ODR process. They were asked to think aloud while they performed tasks that were based on a script written by the research team. The team also video-recorded them as they performed the tasks to further observe their expressions and movements to identify confusion, confidence, frustration, etc.

Once the baseline testing was completed, the research team conducted a series of workshops with low income members of the community in Pima County, Arizona, to identify areas for improvement in the design of both the affidavit and summons and the ODR platform. With the information from both baseline testing and the workshops, the research team then redesigned the documents and platform, then tested the redesigns on another group of eight people. With this third group, the research team did the same type of assessment as it did with the baseline group to determine whether their redesign improved the usability of the platform.

The initial group of eight were stepped through 11 tasks. In five of these, a majority of test participants were unable to complete the task without help due to lack of information, design issues or functionality problems. The problems began at the very beginning, with participants not knowing what to do with the summons and continued with problems finding help on the platform, registering on the platform, sharing documents and reviewing the written agreement.

The first critical issue identified was that the test participants could not fully understand the affidavit and summons. Three of the eight didn’t understand they could register for ODR and a fourth didn’t understand that the URL provided enabled them to participate in ODR. Further, only one of the eight identified all of the options available to them (participate in ODR, ask to be excused from ODR, right to a jury trial, and so forth). People also had difficulty typing the URL into their cellphones.

The second critical issue the participants encountered was failing to find help information on the platform. Only one of the eight participants was able to find the “Help” PDF. Further, the test participants wanted more information than they were provided. They wanted to be told what ODR was on the home page, wanted legal terms to be defined and more information on how to use the platform.

The next failure occurred when participants attempted to register. Only one test participant was able to complete the process without help. Common errors included typing in their name or case number incorrectly, failing to notice the system requirements for a password and not understanding the meaning of the terms plaintiff and defendant. In addition, several participants couldn’t find the corresponding information on the affidavit and summons that they needed in order to register.

The final critical issues the test participants faced were when they attempted to upload documents and review and sign the agreement. The problems in those instances were with the functionality of the technology, however, participants also had trouble figuring out where they could access the document upload function.

Interestingly, the test participants were best able to use the chat function to negotiate and come up with a payment plan. Despite this, they needed more information to do this well, including a better understanding of the ODR facilitator’s role and how to interact with the facilitator. They wanted a way to chat with the facilitator individually (without the other party) and wanted the ODR facilitator to start the chat.

At the end of their testing and redesign, the research team made the following recommendations:

  • Employ best practices for URL formation, website naming, user interface design, and highlight key information on the affidavit and summons.
  • Streamline the registration process.
  • Simplify document sharing and review, as well as allow users to confirm settlement details and download and print the agreement.
  • Improve ODR information and help – include an FAQ button on the home page, include a quick guide and include a welcome video outlining how ODR works. Include closed captioning of the video.
  • Clarify legal information and user options.

The research team also included recommendations for further study, which are relevant to anyone designing an ODR platform:

  • To increase accessibility, the platform design should comply with World Wide Web Consortium (W3W) and Web Accessibility Initiative (WAI) standards.
  • Create informational videos. The videos should be close-captioned and in multiple languages.
  • Develop an auto-responses bank containing common chat responses that parties can select as they negotiate with the other party.
  • Create a paper ODR quick guide that can be sent with the affidavit and summons. The guide should describe the ODR process, provide legal information, and explain in general terms how the ODR platform works.
  • Provide a link or button on each page that provides help for that particular page.
  • Provide an integrated interest calculator to help parties check the amount in controversy and submit calculations to the ODR facilitator and other party.
  • Integrate a calendar function so the parties and ODR facilitator can identify a mutually agreeable time to chat synchronously.
  • Provide the option for each party to chat individually with the ODR facilitator.
  • Integrate an AI chatbot to answer questions and alert the ODR facilitator when they are needed.
  • Integrate a video-conference feature.
  • Employ an integrated exit survey to provide ongoing feedback.

The take-home message from this evaluation is that parties need a lot of information provided in accessible formats. They also require document and platform design that makes it easy to find that information and a platform that is not only easy to use but also flexible. 

Posted in Evaluation, Online Dispute Resolution | No Comments »

Resources for Courts Considering and Developing ODR Programs

Jennifer Shack, August 31st, 2020

Back in March, as courts put a hold on in-person activities, I provided Court ADR Connection readers with a list of resources on online dispute resolution with an idea that courts were beginning to think about how to provide remote services. Now, six months later, with courts continuing these remote services, I thought it might be time to share the list again in hopes that the resources might provide guidance.

Considerations and Concerns in ODR Program Design

Online Dispute Resolution Special Topic

Resolution Systems Institute, 2019

RSI has written a guide for courts who are considering, have started developing or already have ODR programs. It discusses important considerations for ODR implementation, from goal setting to costs to ethical concerns.

Read RSI’s advice about ODR on our website.

Considerations in Implementing Court ODR Systems

Doug Van Epps and Michelle Hilliker. Michigan Supreme Court State Court Administrators Office of Dispute Resolution. Jan. 6, 2020

Van Epps and Hilliker share their insights and the knowledge gained from their development and implementation of ODR in the Michigan courts in this guide. Based on both the issues they encountered and their discussions with others involved in implementing ODR systems, their considerations are meant to assist courts to determine how to design, implement and evaluate an online dispute resolution (ODR) system.

The considerations span a variety of topics including leadership and court staff; prospective users and stakeholders; goals; implementing authority and legal implications administration; platform attributes and functions; mediators; non-court dispute resolution service staff; costs, fees and funding sources; confidentiality; protections; vendor selection; data collection and evaluation; and marketing plans. The guide also includes a list of recent ODR publications and resources.

Access the Considerations document on Michigan’s Supreme Court State Court Administrator’s Office website.

Case Studies in ODR for Courts

Joint Technology Committee, 2020

This paper presents seven case studies of ODR implementation in the courts. The case studies are short, but include key takeaways about what worked and what didn’t. The ODR programs include two outside the US, and deal with small claims, family, tax and traffic cases.

Read the case studies.

Online Dispute Resolution: A Digital Door to Justice or Pandora’s Box? Parts I-III

Doug McQuiston and Sharon Sturges, Colorado Lawyer, February and March, 2020

McQuiston and Sturges are in the midst of publishing a three-part series on ODR in the courts that examines the use of videoconferenced mediation. They note that videoconferencing may be appropriate for family cases and those involving intimate partner violence. The main obstacle to providing this service is limited or poor internet connectivity.

Part II focuses on the use of artificial intelligence in ODR. McQuiston and Sturges cite the many benefits of AI-assisted ODR for small claims and family cases, such as the ability to negotiate asynchronously, which eliminates the need to coordinate schedules. Self-represented litigants who may be reluctant to attend mediation without an attorney may be more inclined to use this technology. Further, in doing so, they can save money. McQuiston and Sturges note some drawbacks, however. These include AI’s inability to understand and address human emotions and its tendency to deviate to the mean, without reference to shades of gray in disputes or situational fairness. To help readers understand how AI in ODR would work, they end by describing systems already in place around the world.

In Part III, McQuiston and Sturgis explore ethical considerations of ODR from the attorney perspective. They also discuss relevant standards, including those of the International Counsel for ODR – that ODR programs be accessible, accountable, competent, confidential, equal, fair, legal, secure and transparent. To courts that plan to implement ODR, McQuiston and Sturgis advise that they establish an ethical framework that “incorporates the underlying purposes of mediator standards.”

Read Part I, Part II and Part III of the series.

Designing and Implementing a State Court ODR System: From Disappointment to Celebration

David Allen Larson, Journal of Dispute Resolution, Vol. 2019, No. 2, Jun. 5, 2019

This article chronicles the author’s work to develop an online dispute resolution (ODR) system to handle credit card debt collection in New York State courts. The author worked with the New York State Unified Court system for a little over two years to design and implement their ODR platform. The article discusses the issues related to dispute system design in this setting, explains how the project was derailed and ends with lessons learned. The four lessons discussed are (1) anticipate conflicts and resistance, (2) obtain support from judges and court staff at the beginning, (3) figure out the technology while also ensuring a fair vendor bidding process and (4) pick your case type carefully.

The article also touches on some issues specific to ODR such as how ODR relates to a court system that is not fully digitized and how long to retain records of online communications. It also offers general advice, such as recommending that ODR processes should “balance efficiencies, neutrality and self-determination.”

Read the full article on SSRN.

Pouring a Little Psychological Cold Water on ODR

Jean Sternlight, Journal of Dispute Resolution, 2020

This article explores online dispute resolution (ODR) from a psychological lens to examine the strengths and weakness of ODR. The article examines the psychology of dispute resolution by focusing on four different areas: the psychology of perception and memory, the psychology of human wants, the psychology of communication, and judgment and decision making. Sternlight’s article suggests that ODR may not be the best tool to assist individuals in creatively working things out with a fellow disputant and may be better employed for small and predictable disputes, like small online purchases. The article also posits that computers may not be the best forum for communication and argues that human mediators, lawyers or friends are more effective than computers in helping humans deal with their emotions and other judgement and decision-making issues. Sternlight ends by calling for empirical research for both online and in-person dispute resolution.

Read the full article on SSRN.

Studies of Online Dispute Resolution Programs

So far, there have been few published studies of online dispute resolution programs in the courts. Below are two conducted a while back.

Getting Divorced Online: Procedural and Outcome Justice in Online Divorce Mediation

Martin Gramatikov and Laura Klaming, Journal of Law & Family Studies, Jan. 1, 2012

This study of a Dutch experiment with ODR for divorcing couples found that the participants perceived the process to be fair, with procedural fairness, interpersonal justice and informational justice all given high marks. On a scale of 1 to 5, they had averages of 4.27, 4.5 and 4.19, respectively. The participants’ perception of the outcome was also positive, though to a lesser extent than for the procedure. They gave an average of 3.91 for distributive justice, 3.37 for restorative justice, 3.18 for functionality and 3.0 for transparency. The ratings were similar for both men and women. Other findings included men reporting higher out of pocket costs and time spent in mediation than women, and women reporting higher levels of frustration and anger than men.

The participants were referred to ODR, which was provided free of charge, if both parties had an email account and the issues were not complex. Once referred, the parties completed an intake questionnaire to provide the mediator with some details about the dispute. The parties could communicate with the mediator and each other via text message or email. The mediator moderated all communications. Each party was required to respond to the other within 48 hours as a condition of the agreement to mediate. Once all issues in dispute were finalized, the parties completed an evaluation of the procedure before the agreement could be finalized.

Read the complete abstract and access the full study in RSI’s Research Library.

Evaluation of the Small Claims Online Dispute Resolution Pilot

Marc Mason, Avrom Sherr. Sep. 1, 2008

Two courts in England tested online mediation to resolve 25 small claims cases. Those parties who were willing to try mediation were given the option of mediating face-to-face, by telephone, or online. Two mediators were responsible for all online mediations, which were conducted using TheMediationRoom.com.

The online mediations resulted in settlement in 48% of the cases, which was similar to the settlement rate for the face-to-face and telephone mediations, but lower than other small claims mediation programs have reported. Mediators and parties were surveyed post-mediation about their experiences with the process. Mediators reported using more than one method of communication outside TheMediationRoom.com platform – generally email or telephone – to complete the mediation in most cases, and as many settlements were completed outside the platform as within it. The mediators attributed this to difficulties in getting responses from the defendants, as well as to technical difficulties. Because of this and because they lacked the ability to judge non-verbal cues, the mediators said they would have preferred using telephone or email in all but four cases.

The 18 parties who responded to the questionnaire were less frustrated with their experience than the mediators. They expressed fewer issues with the technology, with 47% saying the technology was easy to use. However, they were not overwhelmingly satisfied with the process or the fairness of the outcome. Only 53% were satisfied with their experience and only 23% felt the outcome was fair. Responses to both satisfaction and fairness of outcome were more positive for those who settled their case. The small number of responses limits the reliability of these findings.

Read the full study on SSRN.

Studies Regarding Particular Issues Related to ODR

Shuttle and Online Mediation: A Review of Available Research and Implications for Separating Couples Reporting Intimate Partner Violence or Abuse

Fernanda S. Rossi, Amy Holtzworth-Munroe, Amy G. Applegate, Connie J. Beck, Jeannie M. Adams, Darrell F. Hale. Family Court Review (Association of Family and Conciliation Courts), Aug. 17, 2017

This article examines the published research on shuttle mediation, online audio-visual mediation, and online text-based mediation to discuss the applicability of these mediation methods to family law cases with a history of intimate partner violence and/or abuse (IPV/A). It first presents potential advantages and disadvantages of each mediation method in cases with IPV/A history. The authors suggest that mediators on IPV/A cases must carefully consider a variety of potential issues including the parties’ suspicion of mediator bias, confidentiality concerns and victim-perpetrator power dynamics. The authors also note the need for more empirical research comparing different effects of various mediation methods.

This article is behind a paywall on the Wiley Online Library.

Building Trust Online: The Realities of Telepresence for Mediators Engaged in Online Dispute Resolution

Susan Nauss Exon and Soomi Lee. Stetson Law Review, Vol 49, No. 1, 2019

Nauss Exon and Lee found that trust in an experienced mediator is the same whether a mediation participant interacts with that mediator via video or face-to-face. In their experiment, a single experienced mediator conducted 31 simulated mediations with one party in the room with him and the other interacting via telepresence. Telepresence is sophisticated video conferencing, in which sensitive microphones and special cameras that pan and zoom are used to help participants follow the flow of the conversation.

During the experiment, the participants were asked to complete a questionnaire before the mediation began that measured their level of interpersonal trust. They then completed a second questionnaire after mediation that asked them about their interactions with the mediator and their perceptions of him in order to determine how much they trusted him and found him to be trustworthy. In all, 59 participants provided usable data.

Nauss Exon and Lee found that although the participants’ questionnaire responses before mediation indicated they were on average more likely to distrust others than trust them, all participants agreed mildly or strongly that they could trust the mediator and that the mediator was trustworthy. Further, they found no difference in the level of response (mild or strong) between those who were in the same room as the mediator and those who participated via telepresence, with one exception. They found that those who had a lower predisposition to trust were more likely to see the mediator as trustworthy.

Read the full study on SSRN.

Studies Regarding Topics Related to ODR

ADR Empirical Research Studies

James Coben and Donna Steinstra. Mitchell Hamline Dispute Resolution Institute, Jun. 1, 2018

This compilation of abstracted studies includes a number on topics that are related to the use of ODR. These include:

  • A study of compliance with emailed requests
  • Team decision-making in a virtual environment
  • A qualitative analysis of email negotiation
  • Honesty in face-to-face communication as compared to through an intermediary

Find the abstracted studies on the Mitchell Hamline website.

Call for Research on ODR and Access to Justice

Measuring “Access to Justice” in the Rush to Digitize

Amy Schmitz, Fordham Law Review, 2020

As online dispute resolution (ODR) gains rapidly in popularity among courts in the US, Amy Schmitz provides the field with a rationale and guide for researching its provision of access to justice, or A2J. In this article, she outlines ODR’s promise for increasing A2J for those who are involved in the court system and reviews the current status of research regarding A2J before discussing in detail what research should be done and how it can be undertaken. Schmitz calls for research about how people deal with their legal disputes, with comparisons over time among differing social groups and demographics. In sum, research should look into who can access the courts, how they do it and what resolutions they obtain.

Access the article on SSRN.

Posted in Online Dispute Resolution, Research | No Comments »

A Possible Way Forward to Addressing the Coming Eviction Crisis

Jennifer Shack, August 5th, 2020

A few changes to a housing court in St. Paul, Minnesota, appear to have reaped dividends in term of fewer evictions and more settlements. The housing court instituted a housing clinic, bringing together financial services, legal services and mediation at the same place to help those coming to their eviction hearings. Along with changes to court rules and forms, the clinic has had a number of positive outcomes for both landlords and tenants. Although only one component to the changes to housing court was to increase access to mediation, the overall concept is instructive to anyone involved in ADR and housing courts.

In “Justice Served, Housing Preserved: The Ramsey County Housing Court” (Mitchell Hamline Law Journal of Public Policy and Practice, 2020), Colleen Ebinger and Elizabeth Clysdale discuss the impetus for reform, the process for identifying and instituting needed changes and the results of those changes. The Chief Judge saw a need to make changes that would improve access to justice and bring together resources for tenants that would address the root causes of eviction. To that end, he sought the assistance of the McKnight Foundation and Family Housing Fund. They, in turn, turned to the National Center for State Courts to facilitate the planning process. Other stakeholders who were included in the planning process included legal services, the local dispute resolution center, a lawyer who represented landlords, the county’s financial assistance program, the city’s housing department, as well as judges and court administrators.

The group agreed on three areas of action: implement a number of procedural changes, improve coordination among government entities, and expand access to mediation and legal services. Procedural changes included changes to forms, such as including information in the summons tenants receive about the eviction hearing that details the financial, legal and dispute resolution services available to them. In addition, the settlement form allowed the parties to check that they had agreed to an expungement, which keeps the eviction from showing up in their credit history, and the court order was changed to include the possibility of immediate expungement. Further, if expungement was contingent on the tenant making payments, both parties were now allowed to file a notice of compliance with the payments, rather than just the landlord. This meant that the tenant had more control over whether the expungement was carried through.

Coordination among government entities was improved by providing office space in the courthouse for financial assistance workers representing two different funding agencies. This allowed them to work together and allowed tenants to apply to both at the same time rather than having to wait to be denied from one to apply to the other. In addition, the court began providing all partners with information on all litigants on the calendar, which allows them to be more prepared to assist the litigants when they come to court.

To expand access to legal services and mediation, the court and partners agreed to  have attorneys available for consultation at all hearings , as well as mediators, who would be particularly helpful in dealing with disputes that were not legal in nature. Further, the judge began promoting these services from the bench to ensure that all litigants knew about their right to access these resources.

After a year and a half, the court’s numbers appear to show an improvement in outcomes. The court has a goal of reducing evictions by 50% in five years. In the first 18 months, evictions declined by 8%, to the lowest eviction rate in 10 years. Settlements increased by 5%, to the highest rate in five years. The impact was highest on expungements, which doubled. On the other end, fears of increased trial numbers and longer court calls didn’t come true. The number of trials as a proportion of cases declined and court call length increased by 10 minutes on average.

Anecdotally, the response has been positive. Judges report that tenants are more prepared for trial, with a better understanding of the process and when and how to raise their legal defenses. Landlords, too, see benefits from the changes. They have said they appreciate having financial services at the courthouse. Financial assistance staff will spend time with landlords and landlord attorneys, developing relationships with them that, Ebinger and Clysdale note, bear fruit outside of the courthouse as well. For example, one of the services reported an increase in inquiries from landlords before they file an eviction, wanting to know if their tenants are eligible for emergency assistance.

Ebinger and Clysdale outline six lessons learned from the program:

  • a collaborative attitude between partners is critical to success
  • small changes, such as a new check box on a settlement form, can provide big dividends
  • state law matters and can have its own impact regardless of changes made at the court level
  • financial service providers are better situated to solving emergencies than individuals left on their own to navigate social services
  • different circumstances require different interventions – some litigants will need legal assistance, some mediation and some financial assistance, thus each partner is necessary for the success of the program
  • as settlements increased, so did settlement failures (e.g., tenants failing to pay arrearages as agreed to in the settlement) – along with a higher rate of settlement agreements was a greater number of affidavits of non-compliance

This approach to eviction cases is similar to the one taken by many foreclosure courts in response to the housing crisis, in which homeowners are offered an array of services (albeit not at the same time and not all at the courthouse) to help guide them through the court process and stave off foreclosure if possible. Evaluation of those programs demonstrated their effectiveness. While the data looks promising for this program, it is still early and more can be learned. It would be wonderful to know more from the tenants about their experience with the process and whether they feel they are being well-served.

Posted in Foreclosure Mediation, Research | No Comments »

The Effect of Attendance on Litigant Perceptions of Fairness

Jennifer Shack, June 30th, 2020

Litigant attendance at a dispute resolution process impacts their assessment of the fairness of that process, according to research conducted by Donna Shestowsky. The impact of litigant attendance differs for settlement and adjudicative procedures. As she notes in her article, “Great Expectations? Comparing Litigants’ Attitudes Before and After Using Legal Procedures” (Law and Human Behavior, June 2020) [sub. req.], this has implications for how lawyers shape their clients’ perceptions of individual procedures. 

This is the fifth in a series of articles presenting different aspects of her research into the decision-making of litigants in civil cases. Previous articles reported that litigants prefer mediation, looked at what they wanted from a dispute resolution process, discussed their lack of awareness of what options were available to them and noted that litigants were most likely to select a process based on their lawyer’s advice. The research is based on surveys of litigants in three jurisdictions (in California, Oregon and Utah), that offered both mediation and arbitration options to the surveyed litigants. 

For this aspect of the research, 335 litigants completed surveys soon after their case was filed and were interviewed within three weeks after their case ended. Among the questions asked at the outset of their case was how attractive different procedures were to them. These included all the options available to them for their case, which Shestowsky divided into settlement procedures (mediation and negotiation) and adjudicative procedures (trial and arbitration) when analyzing the results. After their case ended, they were asked how fair they thought the procedure was and whether they attended that procedure. 

Shestowsky found that when litigants attended a settlement procedure used to resolve their case, they rated that procedure as fairer than those litigants who attended an adjudicative procedure. However, when litigants did not attend the procedure used to resolve their case, they saw settlement and adjudicative procedures as similarly fair. When comparing attendance within procedures, she found that attendance did not affect fairness ratings for settlement procedures, but that those who attended an adjudicative procedure rated the procedure as less fair than those who did not attend the procedure.

Attendance also interacted with the litigants’ initial perceptions of the dispute resolution processes. Litigants who rated the procedure they used more negatively at the outset of their case saw that procedure as fairer if they attended the procedure than if they did not. On the other hand, Shestowsky found that those who were attracted to a procedure at the outset of their case rated that procedure as less fair if they attended the procedure than if they did not. For litigants who did not attend the procedure, their initial rating of the procedure did not affect their perception of its fairness at the end of the case. 

Shestowsky states that these findings raise questions about the role lawyers play in shaping their clients’ perceptions of individual procedures before they are used, as well as their perceptions after the case ends if their clients do not attend, particularly since litigants said their lawyer’s advice was the most important factor in their selection of the process. She notes that it is possible that some lawyers set up expectations in such a way that litigants who attend a dispute resolution process are let down by the reality of their experience. Alternatively, lawyers who discuss what happened in a procedure that their clients did not attend may paint a picture that is highly satisfactory to those clients. 

Posted in Research | No Comments »

« Older Entries
Newer Entries »
Verified by ExactMetrics