Resources / Study / Innovation for Court ADR

Just Court ADR

The blog of Resolution Systems Institute

Archive for the ‘Evaluation’ Category

Limited Participation Reduces Success of Otherwise Promising Texas Pilot ODR Program, Evaluation Finds

Just Court ADR, May 31st, 2022

A newly published study conducted by RSI Director of Research Jennifer Shack and University of California, Davis, Professor Donna Shestowsky highlights both the potential of online dispute resolution (ODR) and the importance of appropriate outreach and education on ODR to maximize participation and, thus, program impact.

Jen and Donna evaluated a pilot program in Collin County, Texas, that used a text-based ODR platform to resolve debt and small claims cases in a single court. By adopting ODR, the court sought to reduce the burden of a growing caseload while providing access to justice through a process that did not require the parties to travel or miss work to resolve their case.

The evaluation produced evidence that ODR can be an effective method of dispute resolution, especially for debt cases. In 73% of cases where both parties used the ODR platform, participants resolved their dispute and avoided trial. The rate is similar to that of cases that had the opportunity to use in-person mediation. Unfortunately, however, the program’s goals were not met, because both sides used ODR in only 24% of cases uploaded to the platform. Findings and recommendations related to these outcomes are discussed later in this summary.

The Pilot Program

The pilot ran from September 2019 through August 2020; thus it was concurrent with the COVID-19 pandemic from March to August 2020. Additionally, because ODR is a rather new practice in this context, some details of the program and the platform’s application were being worked out even as the evaluation took place.

The Modria ODR platform allowed parties to communicate by text one-on-one or with the help of a mediator. Parties were ostensibly required to use the platform before their first hearing. If parties reached agreement, their case was dismissed without a trial. During the 12-month pilot period, 1,874 debt and 274 small claims cases were filed, for a total of 2,148 cases.

When a defendant filed an answer, the civil clerk determined whether the case was eligible for ODR. It was ineligible if one side had multiple parties, if a party was not equipped to use ODR, or (until the second quarter of 2020) if the court did not have email addresses for both parties or their attorneys. If the case was eligible, the clerk uploaded it to the ODR platform, which in turn rejected any cases that contained errors, such as missing information, and sent an error report to the IT department so the errors could be fixed. When email addresses and phone numbers were available, the platform sent an automated email (and after April 2020, also a text) to the parties, instructing them to use ODR. The clerk also set the case for trial and mailed the parties, or their attorneys, a notice of their trial date and informed them they were required to use ODR prior to that date. The notice included a link to the platform.

Once a case was uploaded to ODR, participants had 45 days to negotiate one-on-one via the platform’s chat function. At any time during this window, either side could ask for a mediator. Mediation cost each party $40 and had to be completed within 30 days.

If participants reached agreement on the platform, they were given the opportunity to sign an online agreed judgment form, which was automatically sent to the case management system, and the trial was cancelled. If the participants did not reach agreement, the parties continued to trial unless they otherwise reached agreement before the trial date.

Key Findings

Below are some of Jen and Donna’s main findings and top recommendations from their evaluation. For more details and complete recommendations, read the full report here.

Litigant Use of ODR

  • 49% of cases with answers filed were uploaded to ODR. During the pilot period, answers were filed in 698 cases. These 698 cases form the subset that could potentially have been uploaded to ODR. Of these, 341 cases (49%) were eligible and did not contain errors that barred their upload. These were ultimately offered ODR. According to court staff, the most common reason that cases with answers filed were not uploaded to ODR appears to be that the court lacked email addresses for at least one side of the case.
  • One party completed at least one activity online in 50% of cases uploaded to ODR. In 170 of 341 cases (50%), at least one case participant performed at least one activity on the ODR platform, such as asserting a claim, uploading a file, or using the chat function to communicate with the other side.
  • Both sides completed at least one activity on the ODR platform in about one-fourth of eligible cases. In 81 cases (24%) uploaded to ODR, both sides used the platform. Parties in small claims cases were more likely to use ODR (76%) than parties in debt claim cases (45%).
  • Litigants appeared to be unaware of the ODR program. Litigant survey responses suggested that parties were generally not aware of the ODR program, despite participation being required. Only one survey respondent out of ten indicated having received information about the program. When asked what would make them more likely to use ODR for a similar case in the future, half said more information.
  • Litigants had limited access to information about the ODR program. According to court staff, the only ways litigants received information from the court about the ODR program was through the notice the court mailed to them (or their lawyers) about their court date and through an email or text from the platform when the court uploaded their case, if their side had an email address or cellphone number on file with the court. Both of these events occurred only after the defendant filed an answer.
  • Litigants appear open to online options. Among survey respondents, none of whom had participated in ODR, two out of three indicated that the option to use it in future similar cases was attractive. Similarly, when asked to consider using video mediation to resolve future similar cases, 60% responded favorably. 

Outcomes and Time to Disposition

  • 73% of cases in which both parties used ODR resolved before trial. The percentage of ODR cases that resolved before trial was similar to that of cases that did not use ODR, both before and during the ODR program.
  • Debt claim cases were significantly more likely than small claims cases to resolve before trial. Additionally, debt claim cases in which defendants were represented were significantly more likely to resolve before trial than debt claim cases in which defendants were unrepresented.
  • Time to resolution was, on average, 4.6 months for cases that used ODR. This figure includes cases delayed either because of the court’s closure amid the COVID-19 pandemic or because of an upload error on a court server.

Program Costs

It is important to note that workload and cost conclusions are derived from self-reports made during interviews and are inherently subjective.

  • Direct costs to the court to implement ODR were covered by a filing fee. Litigants covered the costs through an extra $5 filing fee the court instituted for all civil cases filed in Collin County except eviction and mental health cases.
  • There were significant indirect costs to the court. Court personnel indicated that they devoted a significant amount of time to ODR prior to its launch. The project manager estimated that the cost in staff time approached six figures and was largely due, in his opinion, to the numerous meetings that involved many court personnel as well as the high percentage of time that he and the responsible IT staff member spent on the project in this phase. Some of this effort laid the groundwork for an anticipated county-wide rollout of ODR.
  • Costs to administer ODR were minimal. After the program’s launch, the time that personnel spent on ODR appeared to drop considerably. No one interviewed reported spending more than a couple of hours per week on the project.
  • ODR did not appreciably change administrative workload. The court administrator and the civil clerk did not perceive an appreciable increase or decrease in their workload. However, it is hard to determine what their workload may have been in the absence of the COVID-19 pandemic, or how much it would have been had greater effort been expended on promoting litigants’ awareness of the program and otherwise attempting to increase ODR use.

Recommendations

Based on the findings of this evaluation, the following recommendations may be relevant for any court considering implementing ODR:

  • Expect to spend significant time and resources to get the program up and running.
  • Notify parties and lawyers about the ODR program early in the process.
  • Educate litigants and lawyers more fully about the program.
  • Conduct outreach to raise awareness of, and promote interest in, the ODR program.
  • Explore video mediation as a dispute resolution option.

Lessons Learned from the Implementation of a Video Eviction Mediation Program in Uncertain Times

Jennifer Shack, May 12th, 2022

This article is part of a series of perspectives on eviction mediation program development that is being supported by the American Arbitration Association-International Centre for Dispute Resolution Foundation. The AAA-ICDR’s grant is enabling RSI to expand our outreach to court ADR colleagues working in the fast-evolving eviction field, and we are tremendously grateful to the Foundation for their support.

I recently had the pleasure of writing a report on the development and implementation of the eviction mediation program in Kane County, Illinois. While the State of Illinois had an eviction moratorium in place, and in anticipation of a surge of cases when the moratorium was lifted, the court asked RSI to develop and administer the program. By all accounts, the program had a successful rollout. During the program’s first six months, referred parties and attorneys displayed an openness to mediation, the vast majority of parties and lawyers who took the post-mediation survey indicated they had a positive experience, and 54 of 81 mediations (67%) resulted in an agreement.

For the report, I interviewed individuals who played a large role in the development and implementation of the program, including the judge, the program partners and the program coordinator. We discussed their aspirations for the program, the steps taken to develop and implement the program, the program process, and program challenges and keys to success.

Universally, the interviewees pointed to five keys to the program’s success:

  • Strong court interest in the project
  • The support of the eviction judge
  • Good communication among the program partners
  • Good administration
  • Buy-in from landlords and their attorneys

The Program

The mediation program was conceived as a point of contact for multiple services to helptenants and landlords navigate the court process, obtain financial assistance, and address housing issues. It was designed within the following context:

  • The courthouse was closed due to the Covid-19 pandemic
  • The widely held expectation that evictions would surge when the moratorium on evictions ended, with the demand for mediation rising significantly as well
  • Significant rental relief (funds to help tenants pay their back and future rent) was available
  • Tenants and landlords needed to know whether rental relief would be provided to them in order to reach an agreement
  • The eviction process needed to keep moving forward while rental relief and mediation options were being sought
  • Almost all tenants in eviction cases are self-represented, as are some landlords
  • There was the potential for external funding for mediation

This context meant that the program needed to be a remote process, with cases triaged quickly and referred for other services prior to mediation. Anticipating a large number of cases, the program required a large number of mediators. RSI and the court therefore designed the program as a free, multi-step process with a full-time coordinator, legal aid and financial counseling partners, and paid mediators, all of which was possible due to the availability of multiple sources of external funding.

Lessons Learned

The individuals I interviewed outlined the multiple challenges they confronted in developing and implementing the program. The lessons they learned from working through those challenges are outlined below.

Flexibility is required, particularly when confronting uncertainty.

The program was planned while three main factors affecting that planning were very uncertain: the number of cases that would be filed, what level of funding would be available and when the predicted surge in cases would begin. These circumstances required the program partners to remain flexible during the planning phase in terms of when to ramp up their services, and it required RSI to react to the changing landscape of cases after the program launched. The ability to change procedures and to increase capacity have been essential to the continued provision of mediation to all parties interested in participating.

Communication among the stakeholders is essential.

During both the planning phase and after program launch, continued communication allowed program partners to plan and to set up efficient mechanisms for referrals. It essentially helped them to be flexible. It also served as a point of exchange of information regarding other types of services available to tenants and landlords, which helped program partners open more doors for their clients.

Judicial support is key.

The program can only function if the judges support it both by educating the parties about the resources available to them and by either strongly encouraging or requiring the parties to attend mediation. When tenants are educated about the benefits of mediation, they are more likely to want to participate.

Landlord and/or landlord attorney buy-in is required.

It is important to get the perspective of the landlords during the planning phase and to address their concerns. If the landlords and/or their attorneys do not see the value of mediation to them, they will not participate or, if ordered to, will not participate fully. Note that it is also essential to obtain the perspective of the tenants; their concerns and interests were presented by Prairie State Legal Services.

Provision of services is time-intensive.

The program was originally designed with the program coordinator (PC) conducting an intake with each party who came to her during the court hearings, letting them know about the services available and making referrals on the spot. This became untenable when the number of cases per hearing date rose to 40 or 50 and RSI found that information exchange with parties took longer than expected. To provide this kind of service would require more than one or two people. The PC, therefore, shifted to obtaining contact information from each interested party and then following up after court.

RSI’s program partners had similar challenges keeping up with demand. According to the director of program partner the Aurora Financial Empower Center (FEC), the FEC’s three counselors would not be able to assist all tenants who required help if the number of cases surged too high. Legal aid program partner Prairie State Legal Services similarly did not have the staff required to help all eligible tenants seeking their services. All of this suggests that further resources are required to provide the optimal level of service for all those who need it.

Good program administration is important.

The program coordinator’s skillful management of the program has been a key to the program’s success. Her organizational skills and development of efficient processes have made the program run smoothly.


Many thanks to the American Arbitration Association-International Centre for Dispute Resolution Foundation for its support of the evaluation of the eviction mediation program, of which the implementation report is a part. Many thanks as well, to the Illinois Equal Justice Foundation for its support of the eviction mediation program.

Go to RSI’s website to read the full report.

Survey Data Indicates Eviction Mediation Program Offers Procedural Justice

Jennifer Shack, March 9th, 2022

This article is part of a series of perspectives on eviction mediation program development that is being supported by the American Arbitration Association-International Centre for Dispute Resolution Foundation. The AAA-ICDR’s grant is enabling RSI to expand our outreach to court ADR colleagues working in the fast-evolving eviction field, and we are tremendously grateful to the Foundation for their support.

Last year, RSI and the 16th Judicial Circuit Court in Kane County, Illinois, launched a new mediation program to address eviction cases. In this program, parties are informed of the program when they receive their summons and are invited to participate when they arrive for their initial hearing. If they decide to participate, they meet with the program coordinator, who informs them of their options, including meeting with a financial counselor and/or a legal services representative. The coordinator then schedules the mediation on a date prior to the next hearing. Mediation takes place via Zoom. Between June 2021 and early January 2022, the program mediated 81 disputes.

After each mediation, parties and attorneys are invited by email to complete a survey online about their experience. From the first 81 cases, 28 participants responded, including 6 tenants, 2 landlords and 20 attorneys. While this is a small sample size from which to draw definitive conclusions, their responses indicate the program is offering participants a positive experience in mediation.

The tenants, landlords and attorneys all gave favorable ratings to their experience in mediation. 89% indicated they would recommend mediation to a friend or colleague. 93% rated fairness of the process highly. All but one said they could express what was important to them during the mediation. When commenting about what they liked about the mediation, they most often said something positive about the mediator. A few also commented on the convenience of the process.

RSI has published these findings in a brief report available on our website. We are grateful to the American Arbitration Association-International Centre for Dispute Resolution Foundation’s support for our ongoing evaluation of the program and the dissemination of the findings, and to the Illinois Equal Justice Foundation, whose support has enabled us to operate this mediation program. We look forward to sharing more information with you as the program evolves.

What You Need to Know in Order to Know More About Your Program

Jennifer Shack, June 1st, 2021

I thought I’d do something a little different this month and point out a few resources to those of you who are interested in either starting to examine your ADR programs or are thinking about how to expand or improve current efforts to evaluate program effectiveness. 

Demographics

There has been a push lately to have courts collect demographic information from parties, particularly race and ethnicity, so that courts can better understand and address inequities in service provision. In that vein, the National Center for State Courts has published “Collecting Race and Ethnicity Data.” This is a short report that provides helpful information on the standards for collecting such information, things to think about when planning to collect it and how you may want to customize race and ethnicity categories to best fit the community you serve. 

Model Surveys

Demographics are also included in the Model Surveys created by RSI and the American Bar Association Section of Dispute Resolution. But that’s only one part of what you’ll find in our packet. The Model Surveys include questions you should ask on any mediation program survey, as well as instructions about how to customize the surveys for your particular program.

Guide to Program Success

If you’re looking for more extensive information on how to monitor and evaluate your program, RSI has included two chapters in our Guide to Program Success that step you through tracking your program and conducting evaluations. In Chapter 11, “Design a System to Track Your Program,” you’ll learn how to decide what to track, what data will be needed from what sources in order to do so, and more.  Chapter 15, “Evaluate Your Program,” dives into everything you need to know about how to do a full program evaluation.

Research Year in Review

Jennifer Shack, December 18th, 2020

The past year we focused on research that related to the times we’ve been experiencing. With courts going online and an expected surge in evictions on the horizon, I turned my attention to those topics, summarizing research on online dispute resolution (ODR) and presenting outcomes from housing mediation programs. 

Online Dispute Resolution

In March, I rounded up the research to date on ODR. A study in the Netherlands found that participants in ODR for divorcing couples perceived the process to be fair, with procedural fairness, interpersonal justice and informational justice all given high marks. On a scale of 1 to 5, they had averages of 4.27, 4.5 and 4.19, respectively. The participants’ perception of the outcome was also positive, though to a lesser extent than for the procedure. They gave an average of 3.91 for distributive justice, 3.37 for restorative justice, 3.18 for functionality and 3.0 for transparency.

A small study of a pilot small claims ODR program had less positive results. It found that 47% of cases reached agreement. The 18 parties who responded to a survey had some issues with the technology, with only 47% saying the technology was easy to use. In addition, only 53% were satisfied with their experience and only 23% felt the outcome was fair.

In the round up, I also summarized research about the potential advantages and disadvantages of using video-based and text-based ODR in cases with a history of intimate partner violence or abuse (IPV/A). The researchers suggest that mediators on IPV/A cases must carefully consider a variety of potential issues including the parties’ suspicion of mediator bias, confidentiality concerns, and victim-perpetrator power dynamics. 

While others in 2020 wrote about the possibilities for ODR, Jean Sternlight examined some of its weaknesses. Her article explored online dispute resolution (ODR) through the lens of the psychology of dispute resolution, focusing on four different areas: the psychology of perception and memory, the psychology of human wants, the psychology of communication, and judgment and decision making. She concluded that ODR may not be the best tool to assist individuals in creatively working things out with a fellow disputant and may be better employed for small and predictable disputes, like small online purchases.

An RSI survey found that the COVID pandemic has led most states to adopt video mediation for family cases. Others are moving forward with formal ODR platforms. Despite the increased availability of online services, almost half of the states that responded to the survey said there was an unmet need for family ODR and that funding was the main requirement for meeting that need. 

During the past year, we also learned about how to design ODR platforms from a study of Utah’s small claims platform, and were given tips on researching the impact of ODR on access to justice. 

Eviction Mediation

Two articles published in this year discussed programs in Minnesota and Missouri to help landlords and tenants avoid eviction. The results of these programs indicate that they help keep evictions off tenant credit histories and reduce forcible evictions. 

In St. Paul, Minnesota, the court instituted multiple changes to its housing court, including expanding access to mediation and making it, along with financial and legal resources, available at the court during eviction hearings. After a year and a half, the court’s numbers appear to show an improvement in outcomes. The court has a goal of reducing evictions by 50% in five years. In the first 18 months, evictions declined by 8%, to the lowest eviction rate in 10 years. Settlements increased by 5%, to the highest rate in five years. The impact was highest on expungements, which doubled. On the other end, fears of increased trial numbers and longer court calls didn’t come true. The number of trials as a proportion of cases declined and court call length increased by 10 minutes on average.

In St. Louis, in a voluntary program for cases in which neither landlord nor tenant is represented, 71% of mediated cases resulted in a settlement in 2018. The terms of more than half of these agreements were completed, resulting in a dismissal. One-third of agreements resulted in a consent judgment for eviction against the tenant and 25% resulted in the sheriff executing the judgment through forcible removal of the tenant. Cases that went to trial, on the other hand, were significantly more likely to end in eviction. Consent judgments were entered against tenants in 92% of these cases and resulted in forcible removal in 40%. The authors extrapolate from that data that 279 families avoided eviction in 2018 by settling in mediation and completing the terms of their agreement rather than going to trial.

Family ADR

Studies of family ADR programs continue to demonstrate the benefits of helping parents to resolve their issues outside of court. 

In Anchorage, Alaska, an Early Resolution Program (ERP) for family cases reduced time to resolution, reduced staff time spent on cases and had no impact on the number of post-disposition motions to modify, according to a recently completed evaluation. The study found that 80% of the parties who participated in ERP reached agreement in a three-hour hearing. Unsurprisingly, ERP cases reached disposition more quickly, with a median of 42 days as compared to a median of 104 for cases in the control group. The program also led to significant time savings for staff. For cases undergoing ERP, there were 28 to 30 processing steps, taking a total of 240 minutes (4 hours). The number of steps for the average non-ERP case was 49, taking a total of 1,047 minutes (17.45 hours).

study of parenting time mediation in Massachusetts found multiple benefits for parents and families. In surveys, parents said that conflict between them and the other parent was diminished in about 2/3 of the mediations. This benefit appeared to last for weeks after mediation for many parents, as 53% of those who were interviewed said that conflict continued to be reduced. Similarly, more than 2/3 of surveyed parents reported greater civility between them and the other parent. Again, this benefit remained over time, with 50% saying that they and the other parent treated each other with greater civility. Most parents also said that their communication had improved, with 72% of those surveyed saying so and 54% of those interviewed weeks later agreeing.  

Litigant Perception Research

Litigant attendance at a dispute resolution process impacts their assessment of the fairness of that process, according to research conducted by Donna Shestowsky. Shestowsky found that when litigants attended a settlement procedure used to resolve their case, they rated that procedure as fairer than those litigants who attended an adjudicative procedure. However, when litigants did not attend the procedure used to resolve their case, they saw settlement and adjudicative procedures as similarly fair. When comparing attendance within procedures, she found that attendance did not affect fairness ratings for settlement procedures, but that those who attended an adjudicative procedure rated the procedure as less fair than those who did not attend the procedure.

I wish you all a happy, safe and healthy holiday season!