Resources / Study / Innovation for Court ADR

Just Court ADR

The blog of Resolution Systems Institute

Archive for the ‘Evaluation’ Category

What You Need to Know in Order to Know More About Your Program

Jennifer Shack, June 1st, 2021

I thought I’d do something a little different this month and point out a few resources to those of you who are interested in either starting to examine your ADR programs or are thinking about how to expand or improve current efforts to evaluate program effectiveness. 


There has been a push lately to have courts collect demographic information from parties, particularly race and ethnicity, so that courts can better understand and address inequities in service provision. In that vein, the National Center for State Courts has published “Collecting Race and Ethnicity Data.” This is a short report that provides helpful information on the standards for collecting such information, things to think about when planning to collect it and how you may want to customize race and ethnicity categories to best fit the community you serve. 

Model Surveys

Demographics are also included in the Model Surveys created by RSI and the American Bar Association Section of Dispute Resolution. But that’s only one part of what you’ll find in our packet. The Model Surveys include questions you should ask on any mediation program survey, as well as instructions about how to customize the surveys for your particular program.

Guide to Program Success

If you’re looking for more extensive information on how to monitor and evaluate your program, RSI has included two chapters in our Guide to Program Success that step you through tracking your program and conducting evaluations. In Chapter 11, “Design a System to Track Your Program,” you’ll learn how to decide what to track, what data will be needed from what sources in order to do so, and more.  Chapter 15, “Evaluate Your Program,” dives into everything you need to know about how to do a full program evaluation.

Research Year in Review

Jennifer Shack, December 18th, 2020

The past year we focused on research that related to the times we’ve been experiencing. With courts going online and an expected surge in evictions on the horizon, I turned my attention to those topics, summarizing research on online dispute resolution (ODR) and presenting outcomes from housing mediation programs. 

Online Dispute Resolution

In March, I rounded up the research to date on ODR. A study in the Netherlands found that participants in ODR for divorcing couples perceived the process to be fair, with procedural fairness, interpersonal justice and informational justice all given high marks. On a scale of 1 to 5, they had averages of 4.27, 4.5 and 4.19, respectively. The participants’ perception of the outcome was also positive, though to a lesser extent than for the procedure. They gave an average of 3.91 for distributive justice, 3.37 for restorative justice, 3.18 for functionality and 3.0 for transparency.

A small study of a pilot small claims ODR program had less positive results. It found that 47% of cases reached agreement. The 18 parties who responded to a survey had some issues with the technology, with only 47% saying the technology was easy to use. In addition, only 53% were satisfied with their experience and only 23% felt the outcome was fair.

In the round up, I also summarized research about the potential advantages and disadvantages of using video-based and text-based ODR in cases with a history of intimate partner violence or abuse (IPV/A). The researchers suggest that mediators on IPV/A cases must carefully consider a variety of potential issues including the parties’ suspicion of mediator bias, confidentiality concerns, and victim-perpetrator power dynamics. 

While others in 2020 wrote about the possibilities for ODR, Jean Sternlight examined some of its weaknesses. Her article explored online dispute resolution (ODR) through the lens of the psychology of dispute resolution, focusing on four different areas: the psychology of perception and memory, the psychology of human wants, the psychology of communication, and judgment and decision making. She concluded that ODR may not be the best tool to assist individuals in creatively working things out with a fellow disputant and may be better employed for small and predictable disputes, like small online purchases.

An RSI survey found that the COVID pandemic has led most states to adopt video mediation for family cases. Others are moving forward with formal ODR platforms. Despite the increased availability of online services, almost half of the states that responded to the survey said there was an unmet need for family ODR and that funding was the main requirement for meeting that need. 

During the past year, we also learned about how to design ODR platforms from a study of Utah’s small claims platform, and were given tips on researching the impact of ODR on access to justice. 

Eviction Mediation

Two articles published in this year discussed programs in Minnesota and Missouri to help landlords and tenants avoid eviction. The results of these programs indicate that they help keep evictions off tenant credit histories and reduce forcible evictions. 

In St. Paul, Minnesota, the court instituted multiple changes to its housing court, including expanding access to mediation and making it, along with financial and legal resources, available at the court during eviction hearings. After a year and a half, the court’s numbers appear to show an improvement in outcomes. The court has a goal of reducing evictions by 50% in five years. In the first 18 months, evictions declined by 8%, to the lowest eviction rate in 10 years. Settlements increased by 5%, to the highest rate in five years. The impact was highest on expungements, which doubled. On the other end, fears of increased trial numbers and longer court calls didn’t come true. The number of trials as a proportion of cases declined and court call length increased by 10 minutes on average.

In St. Louis, in a voluntary program for cases in which neither landlord nor tenant is represented, 71% of mediated cases resulted in a settlement in 2018. The terms of more than half of these agreements were completed, resulting in a dismissal. One-third of agreements resulted in a consent judgment for eviction against the tenant and 25% resulted in the sheriff executing the judgment through forcible removal of the tenant. Cases that went to trial, on the other hand, were significantly more likely to end in eviction. Consent judgments were entered against tenants in 92% of these cases and resulted in forcible removal in 40%. The authors extrapolate from that data that 279 families avoided eviction in 2018 by settling in mediation and completing the terms of their agreement rather than going to trial.

Family ADR

Studies of family ADR programs continue to demonstrate the benefits of helping parents to resolve their issues outside of court. 

In Anchorage, Alaska, an Early Resolution Program (ERP) for family cases reduced time to resolution, reduced staff time spent on cases and had no impact on the number of post-disposition motions to modify, according to a recently completed evaluation. The study found that 80% of the parties who participated in ERP reached agreement in a three-hour hearing. Unsurprisingly, ERP cases reached disposition more quickly, with a median of 42 days as compared to a median of 104 for cases in the control group. The program also led to significant time savings for staff. For cases undergoing ERP, there were 28 to 30 processing steps, taking a total of 240 minutes (4 hours). The number of steps for the average non-ERP case was 49, taking a total of 1,047 minutes (17.45 hours).

study of parenting time mediation in Massachusetts found multiple benefits for parents and families. In surveys, parents said that conflict between them and the other parent was diminished in about 2/3 of the mediations. This benefit appeared to last for weeks after mediation for many parents, as 53% of those who were interviewed said that conflict continued to be reduced. Similarly, more than 2/3 of surveyed parents reported greater civility between them and the other parent. Again, this benefit remained over time, with 50% saying that they and the other parent treated each other with greater civility. Most parents also said that their communication had improved, with 72% of those surveyed saying so and 54% of those interviewed weeks later agreeing.  

Litigant Perception Research

Litigant attendance at a dispute resolution process impacts their assessment of the fairness of that process, according to research conducted by Donna Shestowsky. Shestowsky found that when litigants attended a settlement procedure used to resolve their case, they rated that procedure as fairer than those litigants who attended an adjudicative procedure. However, when litigants did not attend the procedure used to resolve their case, they saw settlement and adjudicative procedures as similarly fair. When comparing attendance within procedures, she found that attendance did not affect fairness ratings for settlement procedures, but that those who attended an adjudicative procedure rated the procedure as less fair than those who did not attend the procedure.

I wish you all a happy, safe and healthy holiday season!

Eviction Mediation in St. Louis Significantly Reduces Evictions

Jennifer Shack, November 23rd, 2020

As the eviction crisis looms, a number of courts around the country are implementing mediation programs. Data collected from a decade-old program in St. Louis County provide more evidence that these new programs are likely going to be effective. Mediation there was found to have a positive effect on outcomes and compliance, helping both landlords and tenants to maintain stability in income and housing. 

In a recent article, “Addressing the Housing Crisis Through Mediation” (Washington University Journal of Law and Policy, 2020), Karen Tokarz, et al, discuss how the program works and the benefits that have accrued to participants. The Washington University School of Law Civil Rights & Mediation Clinic developed the program in partnership with Metropolitan St. Louis Equal Housing and Opportunity Council more than a decade ago. In 2012, mediators affiliated with United States Arbitration & Mediation joined clinic students in providing free mediation services for landlord-tenant cases in which neither side has a lawyer. Originally opt-in, the program was made opt-out in 2018. 

The mediators for the program – lawyers and students alike – attend a training that includes an overview of housing law in St. Louis County, mediator ethics, mediation strategies and agreement drafting. The mediators must observe at least two mediations, co-mediate at least two mediations, and be shadowed for at least two mediations before they begin mediating independently. Mediations are conducted on the first court date for the case, which is generally the trial date.

The program uses two agreement forms that are completed as a part of each mediation agreement. The first, the conditional continuance, lays out the settlement terms. This document continues the case while the parties comply with the terms and notes that if the terms are satisfied, the case will be dismissed. It also notes that if a party breaches the terms of the agreement, the other party may file a consent judgment. The consent judgment is the second form that is completed during the mediation.  It typically grants possession and the full rent owed to the landlord. Should the case come back before the judge to sign the consent judgment, the judge uses both documents to determine whether to do so. The judge may decline to sign if, for example, the landlord has not made repairs agreed to in the conditional continuance. 

The program has been successful. In 2018, 71% of mediated cases resulted in a settlement. The terms of more than half of these agreements were completed, resulting in a dismissal. One-third of agreements resulted in a consent judgment for eviction against the tenant and 25% resulted in the sheriff executing the judgment through forcible removal of the tenant. Cases that went to trial, on the other hand, were significantly more likely to end in eviction. Consent judgments were entered against tenants in 92% of these cases and resulted in forcible removal in 40%. The authors extrapolate from that data that 279 families avoided eviction in 2018 by settling in mediation and completing the terms of their agreement rather than going to trial. It must be noted, however, that the two groups of cases – those that mediated and those that did not – are not similar. Mediated cases, as mentioned above, were limited to those in which neither side had an attorney. Those cases that went to trial included those in which at least one party (generally the landlord) had an attorney. 

The authors note that the impact of the eviction mediation program is limited due to its focus on cases in which neither party is represented and the day-of-trial mediation format. Further, growth is difficult due to the limited number of mediators available. They point to four directions the program can take to widen its impact. The first direction is to offer mediation prior to the first court date, or even before the eviction is filed. This would require greater outreach to landlords, tenants and government agencies to ensure that landlords are on board, tenants know about the program and agencies can urge its use. The second direction is to fund the program so that it can be sustained at a broader scale. Third, the program could be expanded to Municipal Court, where housing and building code enforcements are handled. Landlords and tenants are often unrepresented in this court and mediation in this context could lead to housing improvements and stability. The fourth direction would be to adopt online dispute resolution, allowing mediations to occur during the pandemic.  

The St. Louis County eviction mediation program is one of many recent programs that have been implemented around the country. The data indicating its effectiveness adds to the increasing evidence that such programs are successful at reducing evictions, thus providing stability to landlords, tenants and communities. 

Usability Evaluation of Utah’s ODR Platform Provides Insights for Developing Accessible Tools for All

Jennifer Shack, September 28th, 2020

A recently published evaluation of Utah’s self-built ODR platform for small claims cases has guidance relevant to other courts and ODR developers. The evaluation, conducted by the Innovation for Justice Program at the University of Arizona, examined the usability of the platform as well as the affidavit and summons. It found serious issues at several points in the process that pointed to the need for better design, more information and greater functionality in order for litigants to be able to confidently pursue their case from beginning to end.

In Utah, the vast majority of small claims cases are debt claims. These largely end in default judgment because the defendants don’t appear for their hearing. According to this study, emerging data from the pilot sites indicated that only 36% of defendants registered on the ODR platform. Of those 36% of cases, 50% achieved a settlement or voluntary dismissal. In anticipation of a statewide launch of the platform, the evaluation was undertaken to determine what changes could be made to improve the platform, potentially increasing usage and get better outcomes. 

The research team first interviewed stakeholders, including potential defendants, about the platform before designing the evaluation. It then assessed usability by observing eight people who were demographically similar to debt claim defendants in Utah. Those eight people were observed doing specific tasks from the receipt of the affidavit and summons through the end of the ODR process. They were asked to think aloud while they performed tasks that were based on a script written by the research team. The team also video-recorded them as they performed the tasks to further observe their expressions and movements to identify confusion, confidence, frustration, etc.

Once the baseline testing was completed, the research team conducted a series of workshops with low income members of the community in Pima County, Arizona, to identify areas for improvement in the design of both the affidavit and summons and the ODR platform. With the information from both baseline testing and the workshops, the research team then redesigned the documents and platform, then tested the redesigns on another group of eight people. With this third group, the research team did the same type of assessment as it did with the baseline group to determine whether their redesign improved the usability of the platform.

The initial group of eight were stepped through 11 tasks. In five of these, a majority of test participants were unable to complete the task without help due to lack of information, design issues or functionality problems. The problems began at the very beginning, with participants not knowing what to do with the summons and continued with problems finding help on the platform, registering on the platform, sharing documents and reviewing the written agreement.

The first critical issue identified was that the test participants could not fully understand the affidavit and summons. Three of the eight didn’t understand they could register for ODR and a fourth didn’t understand that the URL provided enabled them to participate in ODR. Further, only one of the eight identified all of the options available to them (participate in ODR, ask to be excused from ODR, right to a jury trial, and so forth). People also had difficulty typing the URL into their cellphones.

The second critical issue the participants encountered was failing to find help information on the platform. Only one of the eight participants was able to find the “Help” PDF. Further, the test participants wanted more information than they were provided. They wanted to be told what ODR was on the home page, wanted legal terms to be defined and more information on how to use the platform.

The next failure occurred when participants attempted to register. Only one test participant was able to complete the process without help. Common errors included typing in their name or case number incorrectly, failing to notice the system requirements for a password and not understanding the meaning of the terms plaintiff and defendant. In addition, several participants couldn’t find the corresponding information on the affidavit and summons that they needed in order to register.

The final critical issues the test participants faced were when they attempted to upload documents and review and sign the agreement. The problems in those instances were with the functionality of the technology, however, participants also had trouble figuring out where they could access the document upload function.

Interestingly, the test participants were best able to use the chat function to negotiate and come up with a payment plan. Despite this, they needed more information to do this well, including a better understanding of the ODR facilitator’s role and how to interact with the facilitator. They wanted a way to chat with the facilitator individually (without the other party) and wanted the ODR facilitator to start the chat.

At the end of their testing and redesign, the research team made the following recommendations:

  • Employ best practices for URL formation, website naming, user interface design, and highlight key information on the affidavit and summons.
  • Streamline the registration process.
  • Simplify document sharing and review, as well as allow users to confirm settlement details and download and print the agreement.
  • Improve ODR information and help – include an FAQ button on the home page, include a quick guide and include a welcome video outlining how ODR works. Include closed captioning of the video.
  • Clarify legal information and user options.

The research team also included recommendations for further study, which are relevant to anyone designing an ODR platform:

  • To increase accessibility, the platform design should comply with World Wide Web Consortium (W3W) and Web Accessibility Initiative (WAI) standards.
  • Create informational videos. The videos should be close-captioned and in multiple languages.
  • Develop an auto-responses bank containing common chat responses that parties can select as they negotiate with the other party.
  • Create a paper ODR quick guide that can be sent with the affidavit and summons. The guide should describe the ODR process, provide legal information, and explain in general terms how the ODR platform works.
  • Provide a link or button on each page that provides help for that particular page.
  • Provide an integrated interest calculator to help parties check the amount in controversy and submit calculations to the ODR facilitator and other party.
  • Integrate a calendar function so the parties and ODR facilitator can identify a mutually agreeable time to chat synchronously.
  • Provide the option for each party to chat individually with the ODR facilitator.
  • Integrate an AI chatbot to answer questions and alert the ODR facilitator when they are needed.
  • Integrate a video-conference feature.
  • Employ an integrated exit survey to provide ongoing feedback.

The take-home message from this evaluation is that parties need a lot of information provided in accessible formats. They also require document and platform design that makes it easy to find that information and a platform that is not only easy to use but also flexible.