Resources / Study / Innovation for Court ADR

Just Court ADR

The blog of Resolution Systems Institute

Archive for the ‘Online Dispute Resolution’ Category

Ten Tips for Evaluating Your ODR Program

Jennifer Shack, May 18th, 2023

After Donna Shestowsky and I completed two of the nation’s first neutral evaluations of state court online dispute resolution (ODR) programs, we had some thoughts to share with courts about how they could best ensure the evaluations of their ODR programs were useful and of high quality, so we wrote an article, “Ten Tips for Getting the Most Out of an Evaluation of Your ODR Program,” which was recently published in Court Review, the journal of the American Judges Association. The following is a summary of what we wrote. For more complete guidance, see the complete article.

Tip 1: Negotiate data access when contracting with an ODR provider
RSI Director of Research Jennifer Shack, second from left, and University of California Davis Professor Donna Shestowsky, second from right, are the authors of “Ten Tips for Getting the Most Out of an Evaluation of Your ODR Program.” They joined Nick White, Research & Evaluation Director of the Maryland Judiciary’s Mediation and Conflict Resolution Office, and Dr. Deborah Goldfarb, Assistant Professor at Florida International University, on a panel at the American Bar Association Section of Dispute Resolution Spring Conference.

The best time to ensure you will have the data needed for a future evaluation, and to monitor program activity in general, is when you negotiate your contract with an ODR provider. As you screen providers, learn how each ensures data security and confidentiality. This is also the ideal time to negotiate data access for your evaluation. To keep your evaluation options open, you will want to secure terms that obligate the provider to share data not only with you, but with external evaluators you might hire later. And whether you hire a provider or develop an ODR platform in-house, be mindful of any data sharing or confidentiality rules or policies in your jurisdiction for the types of cases you plan to include in your evaluation.

Tip 2: Determine when to evaluate

Ideally, you would plan the evaluation of your ODR program as you design your program. But evaluation planning can happen at any time, as can the evaluation itself.

If you plan your evaluation before launching your program, you can increase the probability that the evaluation will accurately reflect your program’s use and effectiveness if you begin your evaluation after 1) the provider has addressed technology glitches that may emerge during early testing of your platform and 2) after you conduct outreach to ensure parties know about your program. If your program is already up and running, you should avoid scheduling an evaluation for a time frame when major changes are planned for your court or the ODR program itself. Significant changes while data are being collected may introduce noise into the data.

Tip 3: Find a neutral evaluator

Selecting a neutral evaluator is important for enhancing the quality, usefulness and credibility of your evaluation. In choosing your evaluator, consider whether they have experience evaluating court alternative dispute resolution (ADR) and/or ODR programs. If you hire someone who is not knowledgeable about ADR, be prepared to spend a lot of time explaining how ADR works, the theory behind it, and the specific issues involved.

Tip 4: Ensure that key personnel are involved in the evaluation planning process

Include court personnel who have knowledge that can assist with evaluation planning. These individuals are, at minimum, judges hearing the cases served by the ODR program and court staff who understand the processes involved and the underlying technology. They can help you determine the questions to answer, identify what data are needed, or work out how to access relevant existing data.

You will also want to decide who should serve as the point of contact for your evaluators. This person will answer the evaluator’s questions and help to obtain data. Staff members should be clear on their role in the evaluation.

Tip 5: Prepare to use data from a variety of sources

To best understand your ODR program, you should obtain information from multiple sources, such as your case management system, the ODR platform and participant surveys. To collect systematic feedback from parties (or other stakeholders, such as lawyers), your evaluator will need your help to facilitate distribution of surveys. They will work with you to determine which parties to survey and the best method for contacting the parties. They should also ask you to review the survey questions.

Tip 6: Expect to spend time with the evaluator

To conduct an effective evaluation, your evaluator will need to thoroughly understand your ODR program, how it fits with your overall process for handling cases, and how the platform interfaces with your case management system. Your evaluator will want to spend time with you to discuss your program processes and get answers to any questions throughout the evaluation process.

Tip 7: Facilitate the participation of court personnel and other program partners in the evaluation

Give court personnel and other program partners (e.g., mediators) who were not a part of the evaluation planning process a heads up about the evaluation and ask for their cooperation. Introduce your evaluator to relevant personnel and partners. These efforts should pave the way for your evaluator to reach out to them to get their perspectives on the ODR program and its impact on their work. When asking court personnel and program partners for their cooperation, reassure them that the evaluation’s objective is to improve the program, not to find fault with it or with them.

Tip 8: Help your evaluator to pilot test their survey materials

Before the evaluation period, your evaluator should obtain feedback on their surveys from individuals similar to those who will be surveyed for the evaluation—typically, parties to similar cases. Your evaluator will need your help to gain access to those individuals.

In addition, every court has unique terminology that should be reflected in how questions are worded. Your evaluator should work with your staff to ensure that the correct terminology is used so that it is more likely to be understood by those who will be asked to complete the survey.

Tip 9: Be flexible about the length of time set aside for data collection

The time allocated for data collection needs to be long enough to get the data necessary for analysis, but measured enough to provide timely results. Your evaluator will work with you during the evaluation planning phase to determine the time frame. But be prepared to be flexible. Your evaluator may recommend extending the data collection period if the level of program use and/or survey participation is lower than expected and they need more time to collect data to deliver a useful evaluation.

Tip 10: Survey those who use ODR as well as those who do not

Our evaluations have shown that the motto “If you build it, they will come” does not always apply to ODR. Surveying eligible parties who did not use ODR could help you identify issues that might be driving lower-than-expected usage. Surveys can point to marketing or party education problems, or in the case of voluntary ODR programs, uncover program attributes that parties find unattractive. You can ask parties whether they knew about your program, how and when they learned about it, and whether they knew they were eligible.

In the end, when a court invests resources to establish an ODR program, a major goal is to have it be used. It is imperative to commit resources to effectively market the program, which should include efforts to educate parties and ensure they know they are eligible or required to use it.

Conclusion

Courts that have their ODR programs objectively evaluated should be applauded for their efforts. Evaluations can facilitate program design that is data-driven and evidence-based, rather than guided by anecdotes or hunches. This grounding in data is especially important when making decisions geared toward satisfying the interests of litigants, since understanding their unique perspectives requires collecting data directly from them. Ideally, ODR evaluations will be conducted by neutral third parties who have no stake in the results and meet high research standards. Neutral evaluations are uniquely situated to offer an outside perspective on what works well about a program and to suggest how it might be improved. In addition, constituents, including lawyers, are more likely to accept the findings of a neutral, outside evaluation that concludes that a program delivers beneficial outcomes.

Does ADR + Tech = Better Access to Justice? RSI Spent Much of 2022 Trying to Find Out

Sandy Wiegand, May 2nd, 2023

RSI spends a lot of time and energy studying the conditions under which court-based alternative dispute resolution (ADR) can best improve access to justice. In recent years, that has often meant using new technologies and/or assessing their impact.

As is often the case with innovations, ADR options that employ new technology are sometimes hailed as the solution to longstanding challenges. For example, online dispute resolution (ODR) is celebrated for its potential to increase access to justice by allowing parties to engage on their own schedules, in their own spaces. Unfortunately, however, technological innovations can also bring challenges and create their own barriers to justice.

RSI’s 2022 annual report asks the question: Does ADR + Tech = Better Access to Justice? Our staff spent much of last year examining that premise. We published two landmark evaluations of court programs that used ODR-specific platforms; completed an in-depth report on the potential for ODR to serve thinly resourced parents, courts and communities; and used video mediation to serve hundreds of clients in northern Illinois. We also evaluated how those programs were operating and how participants viewed them.

Our annual report outlines these efforts and summarizes some of our findings. Not surprisingly, we found both promising signs and causes for concern when it came to technology’s impact on access to justice. We also discovered a lot more questions that need to be answered and problems that need to be addressed.

We hope you will take the time to read the Resolution Systems Institute 2022 Annual Report and review what we have learned so far. The role of technology is, of course, just one of many aspects of court-based ADR that RSI is examining. Please join us as we continue exploring what technology can and can’t solve, as well as other keys to providing cost-effective, timely and fair conflict resolution.

Rethinking Party Safety in Online Mediation

Dee Williams, January 19th, 2023

The COVID-19 pandemic has led to online mediation becoming far more common in family cases than it was previously. This shift from in-person to video mediation has both benefits and potential pitfalls when it comes to participant safety, as discussed in a recent article by Erin R. Archerd.

In her Winter 2022 Stetson Law Review article, “Online Mediation and the Opportunity to Rethink Safety in Mediation,” Archerd describes some of the security benefits and challenges of mediating online, recommends steps mediators can take to enhance party security in online mediation, and calls for a more expansive conception of safety for mediations in general.

Photo by Liza Summer via Pexels

Some observers argue that online mediation can be safer than mediating in person because of the physical distance between the parties. Archerd acknowledges this benefit, but also sees a downside. She notes that when mediating in person, a mediator can personally ensure that the room has safe exit routes for all parties in case of a confrontation and that the mediation is not observed or interrupted by an unauthorized party. Such assurances are more difficult online. Additionally, Archerd states that interacting via camera also entails the loss of some of the nonverbal cues that mediators might normally use to assess parties’ senses of safety. To make up for this, she suggests that — once screening for impediments has been completed and the mediator and parties decide to go forward with­ mediation — mediators hold private pre-mediation sessions with each party. During such a meeting, the mediator can go over the security of the parties’ mediation locations, make sure they will be in a safe and appropriately private environment during the mediation, and establish ways to communicate if the party is being watched or intimidated from off-screen. Mediators can do something similar on the day of mediation by holding a private session with each party prior to joint session to ask them to describe their space and ask whether they feel they can safely complete the mediation process.

Maintaining confidentiality in an online mediation also requires more work, since mediators are not able to monitor all aspects of the space in the same way. Archerd recommends that mediation agreements make it clear that unauthorized parties should not be present at the mediation. In addition, mediators should communicate with parties in advance about how to ensure privacy in their mediation locations. At the start of the mediation session, mediators should confirm with parties that they are not recording and that no unacknowledged parties are present. Another aspect of safety is the long-term well-being of participants: Mediators conducting mediations online need to be sure they are well connected to “wraparound services” such as domestic violence or special education resources. Archerd notes that lack of access to in-person meetings can hamper feedback that would otherwise be received about the overall well-being of parties, and greater effort to connect parties to required services may be beneficial in online mediation environments.

New RSI Report Sheds Light on Family ODR for Thinly Resourced Parents, Courts and Communities

Susan M. Yates, October 26th, 2022

Do you have a project that you started before the pandemic that you had to put on the back burner in the face of many urgent tasks? I did, but not anymore! I am thrilled to say that RSI’s report, “Family Court Online Dispute Resolution for Thinly Resourced Parents, Courts and Communities: Impediment, Improvement or Impossible Dream?” is now available online.

RSI is very grateful to the JAMS Foundation, whose generous funding made this project possible. We are also thankful to many others who contributed to the project, who you can learn about in the report.

Why RSI Did this Project

Having worked with court mediation in its early years, in recent years I have been witnessing similar responses to court online dispute resolution (ODR). There are proponents who see ODR as a great way to make court systems more accessible, less expensive and quicker. However, some also have significant concerns about issues such as whether ODR will be fair and accessible, who will pay for ODR and what might be lost by relying on technology.

RSI wanted to sort out whether family ODR could improve access to justice for thinly resourced parents who were in court over child-related issues (e.g., parenting time and decision-making), which we know is an area of great need in many jurisdictions. We were especially interested in how family ODR might work in jurisdictions and communities that were also thinly resourced.

Structure of the Project

We created a framework for the project. It is a series of steps – each building on the previous steps – that walks through a process of considering what it would take for family ODR to be accessible, ethical, effective, feasible and sustainable.

To work our way through that framework, we:

  • Conducted research on the literature and the state of court ODR
  • Surveyed state alternative dispute resolution (ADR) and ODR leaders
  • Surveyed 37 national experts working in academia, ADR, court ADR, court administration, family law, funding, intimate partner violence, judging, legal aid, legal technology design, ODR, self-represented litigants and technology
  • Convened those 37 experts three times
  • Drew on RSI’s organizational experience

The data used in the report reflects the status of ODR in 2020. Because the project was already delayed by the pandemic, we decided not to continue to update the report as new programs were developed and new resources became available.

Tensions

Through the above work, we identified three tensions that must be resolved in order for family ODR to serve thinly resourced parents, courts and communities.

1. The desire to serve all parents is in tension with the limitations imposed by the thinly resourced environment explored in the project.

Courts have a responsibility to protect potentially vulnerable parents and ensure ODR is accessible and ethical. However, courts that are thinly resourced are unlikely to be able to provide the full range of services recommended by some experts to ensure ODR is accessible for all parents. The services include, for example, individualized education for each parent about their rights, personalized counseling for each parent about their best options, and one-on-one assistance while using ODR. Indeed, in our experience working with courts, it is likely that these thinly resourced courts would be looking for ways to reduce their costs by implementing ODR, not to increase costs because of a need for additional services to supplement ODR.

To address this tension, a safe tradeoff can be constructed by drawing on a long-established requirement of in-person family mediation. Prior to mediation, each parent must be screened individually to determine if a party has experienced intimate partner violence or other coercive behavior in the relationship that would make participation in a traditional mediation unwise. This need for screening is also true for family ODR.

This screening can be expanded to address the concerns specific to ODR, such as issues related to language, disability or access to the internet. The screener would assist the parents in finding ways to access ODR (e.g., how to involve a translator), would work with them to determine if mandatory participation in ODR is appropriate (e.g., in the case of an insurmountable barrier due to a serious illness or a violent relationship), and would help them access other suitable services when needed.

Screening some parents out of ODR will reduce the number of families that can benefit from ODR. However, it will also help to ensure that ODR is accessible and ethical for the parents who do participate.

2. There is a tension between the need for voluntary decision-making (to help make ODR ethical) and the need for participation (to help make ODR effective).

Neither the literature nor the gathered experts agree on whether mandatory or voluntary participation is inherently better. There is, however, a way to address this tension.

A safe tradeoff can be accomplished — as is sometimes the case with in-person family mediation — by requiring that parents who are not screened out of ODR try an initial ODR step. Because this comes after screening, it avoids requiring parents to use ODR if they are unable to participate in ODR or if they should not participate in ODR for any of a variety of reasons. It also increases the likelihood that a court ODR program will serve enough parents to make it effective by requiring that parents at least try ODR.

3. There is a tension between the cost of accessible, ethical, effective family court ODR and the ability of thinly resourced parents, courts and communities to pay for it.

The project pondered ways to resolve that tension, i.e., how to pay for quality court ODR. In the end, this tension could not be resolved. The project was unable to identify a feasible, sustainable path by which family court ODR could be provided nationwide to parents who need it via courts that cannot afford it.

Recommendations

The report resulted in nine recommendations.

1. Support family ODR
There is a need for family ODR despite the growth in family ODR and the availability of family ADR in some areas. There should be nationwide support for providing family ODR to thinly resourced parents, courts and communities.

2. Develop national standards for family court ODR
National standards for family court ODR should be developed and promoted. They should provide definitions; descriptions; guidance and, potentially, specific measurable criteria. The standards should articulate how to ensure family ODR is accessible, ethical and effective.

3. Consider how to assess whether family court ODR meets the standards
During the development of the standards, the question of how to assess whether court programs and vendors meet the standards should be addressed. For example, who would conduct the assessments? What would be the impact of any finding by the assessment?

4. Ensure every participant has a live conversation with a screener prior to ODR
There are situations in which some parents should not participate in ODR; therefore, every parent should engage in a live telephone or video conversation with a screener prior to using ODR. Together, they should explore whether: there was or is any intimate partner violence in the relationship; they have access to ODR; they are comfortable communicating in a language in which ODR is offered; they are comfortable with ODR technology; they are experiencing any mental illness or substance use issues that prevent them from participating in ODR; and they might need any accommodations as a result of disability.

5. Investigate the potential for a national program to conduct screenings
In many places across the country, parents are not routinely screened prior to family mediation. We see the same practice developing with family ODR. A national program is needed to offer screening that is affordable for thinly resourced parents and courts that cannot afford to pay screeners for ODR.

6. Require every parent who is not screened out of ODR to make an initial attempt to use ODR to identify areas of agreement with the other parent
Requiring parents to attempt to use ODR after screening will provide an ethical combination of screening parents out of, and mandating them into, ODR. It will encourage the maximum number of parents to try ODR, thereby increasing the opportunity for effectiveness, but not require parents who are unsuited to ODR to use it. Parents who do use ODR should not be required to reach agreement using it, but the experience of trying the initial step can also encourage parents to keep using ODR if they find it to be easy to use and helpful.

7. Provide guidance and model materials to courts developing ODR projects
Reliable, curated resources presented in an accessible format can help prevent courts from having to reinvent the ODR wheel. These resources could include, for example, guidance on how to determine what ODR processes and platforms to use, what standards to apply, how to select a vendor and what best practices are. These materials should also include model outreach and educational materials such as text for summonses, websites and communications with parents, as well as videos to which local court information could be added.

Courts also need assistance from experienced, knowledgeable experts to put those resources to work. Courts and communities with the least resources should be actively contacted, made aware of the resources, helped to assess whether there is a need for family ODR in their jurisdiction and, if there is a need, supported as they implement family ODR.

8. Enable courts to assess and improve their family ODR services
ODR platforms generally can provide regular statistical information on how ODR is functioning. Courts may need assistance determining what data they need, working with their vendor to obtain the data, and learning how to draw useful information for reports. Video mediation apps, such as Zoom, do not have built-in reporting mechanisms. Courts using video mediation will therefore need to devise other ways to collect critical data.

Courts also need to ensure parents are experiencing procedural justice when they participate in ODR. For courts using ODR platforms, this will likely require the insertion of surveys into the ODR system or the adaptation of surveys provided as part of the ODR platform. Courts using video mediation will need to survey parties about their mediation experience another way, e.g., by email or text.

Additionally, courts should participate in comprehensive program evaluations when possible. They should share results of these evaluations with other courts and with ODR providers to inform other ODR programs.

9. Investigate the potential for a national family court ODR provider
Although the project did not identify an entity that would be able to establish and sustain a national provider of family ODR, it is still possible that a resource-rich home for family ODR exists somewhere. Individuals and entities that are concerned with services to thinly resourced parents, courts and communities should explore whether there is a deep-pocketed funder who would commit to a multi-year national program.

Conclusion

This project investigated the study question, “How might family court online dispute resolution serve thinly resourced parents, courts and communities?” It found that family court ODR can be an impediment to access to justice if not provided in an appropriate manner. However, if it is provided in a manner that is accessible, ethical and effective, family court ODR can improve access to justice. Doing so will require standards for family court ODR, as well as resources to support the provision and evaluation of ODR. It will also necessitate comprehensive screening conversations with all parents prior to ODR, which will enable courts to require that all parents who are not screened out attempt at least an initial stage of ODR.

In the end, whether family ODR that is accessible, ethical, effective and feasible can be provided nationwide to parents who need it, despite limited family, court and community resources, remains an unanswered question and potentially an impossible dream. There is no clear path to determining how to sustain family court ODR services.

Significant Participation, Agreement Levels Highlight Potential of ODR for Family Cases

Jennifer Shack, August 31st, 2022

University of California, Davis, Professor Donna Shestowsky and I recently had the pleasure of conducting the first neutral evaluation of any family law court online dispute resolution (ODR) program in the United States. The program was launched by the 20th Circuit of Michigan’s Friend of the Court (FOC) in August 2020 with the goal of providing parties with post-judgment family law disputes a simpler, more convenient and cost-effective way to reach agreements related to child custody, parenting time and child support. It also aimed to increase efficiency in the disposition of these matters. By and large, we found that the program was providing the benefits the FOC hoped it would.

What We Studied

The FOC used Matterhorn’s text-based ODR platform, which allows participants to communicate with each other and their caseworker via asynchronous text messages and document exchanges. We used case data, ODR data, pre- and post-process party surveys, and staff interviews to gain insight into:

  • Parties’ expectations for the ODR process at the time it was offered to them, and their views on a video mediation alternative
  • ODR access, including the percentage of parties who participated and opted out, information about ODR available to parties, and parties’ capacity to use ODR
  • Participants’ evaluation of their experience of ODR in terms of procedural justice, satisfaction, fairness of the process, and ability to control the outcome of their matter
  • Parties’ impressions of the FOC and the other party
  • The agreement rates, hearing rates, and efficiency (time to disposition, caseworker time spent on matters) associated with ODR use
  • Direct costs and the FOC’s staff members’ perceptions of the effect of ODR on their work

Excitement and Anxiety

We found that before ODR, parties tended to be confident they could reach agreement, but the majority did not believe the other party would be truthful. They expressed high levels of excitement about using ODR, but also high levels of anxiety. Those who were planning on using it were twice as likely to report a high level of fear of the other party as those who weren’t sure they would, or who were not going to, use ODR.

Because of the platform’s technological limitations, caseworkers did not offer ODR to parties who had attorneys, those who had limited English proficiency, or to those who were blind or visually impaired. In addition, the FOC had decided that those who had a history of a high level of conflict would not benefit from ODR, and they were therefore not offered the opportunity.

Uptake Higher than for Other ODR

Almost half of the parties who were offered the chance to use ODR did so, which was high compared with other early ODR programs, which have had participation rates of 21% to 36%. In half the cases in which ODR was offered but not used, at least one of the parties simply did not register to use it despite the FOC’s intent to require its use. We noted that the emails sent to the parties informing them to register for ODR used language that could have confused the parties both about what the program was and their requirement to participate.  We also found that parties lacked an understanding of the main features of the program. This, too could have reduced program use.

A surprising finding was that almost all parties who used ODR accessed the platform with their mobile phone at least part of the time. Only 8% exclusively accessed the platform with a computer, while 71% only used their mobile phone. This indicates that the platform and any auxiliary activities, such as communications to the parties and agreement forms, must be optimized for phones.

When parties used ODR, they were four times as likely to give high ratings for fairness of the process as those who did not use ODR (50% vs. 12.5%), and twice as likely to give high ratings for satisfaction (50% vs. 25%). They were also much more likely to reach agreement than those who were offered ODR but didn’t use it (59% vs. 11%). In cases involving child support, parties using the online platform reached resolution nearly twice as fast as those who did not.

Positive, with Room for Improvement

Our evaluation indicates that the FOC’s ODR program has provided parties with a positive experience, improves agreement rates and reduces time to resolution for some cases. Program use is high in comparison with other programs.

However, our analysis suggests that the FOC could do more to educate parties about the program, direct parties to use it, and increase access to parties with disabilities as well as those who need an interpreter’s assistance to use the platform. The FOC should also explore ways to reduce access barriers for those identified by caseworkers as less likely to use or benefit from the program because they have lawyer representation or have high-conflict relationships. Attempts to reduce access barriers should also be directed at those who lack digital literacy and those who would use their mobile phone to access the platform. Our evaluation was limited by a small survey and case data sample size, which may have obscured statistical significance of some findings and did not allow us to conduct more detailed analyses of participant experience. We are looking forward to seeing more evaluations of ODR programs conducted, to build on our findings.