Program Design - Just Court ADR

Archive for the ‘Program Design’ Category

New Board Member Nancy Welsh Discusses Fairness, Self-Determination in ADR

Just Court ADR, May 10th, 2024

In February, Resolution Systems Institute welcomed two new Board of Directors members to their first RSI board meeting. Recently we wrote a blog introducing you to one of them, University of Denver Law Professor Oladeji M. Tiamiyu. For this edition, we spoke with the other, Texas A&M Law Professor Nancy A. Welsh.

Nancy Welsh is a member of the Board of Directors of Resolution Systems Institute

Nancy Welsh is the Frank W. Elliott, Jr. University Professor, Professor of Law and Director of the Dispute Resolution Program at Texas A&M University School of Law. She is a leading scholar and teacher of dispute resolution and procedural law. Prof. Welsh examines negotiation, mediation, arbitration, judicial settlement and dispute resolution in US and international contexts, focusing on self-determination, procedural justice, due process and institutionalization dynamics. Read more about Prof. Welsh’s background and find links to her work in her RSI bio.

When/where were you first introduced to alternative dispute resolution?

I first learned about alternative dispute resolution when I was in law school. Frank Sander — one of the founders of the ADR movement (including the contemporary mediation movement) — was one of my law professors. He taught a course that I believe was called Alternatives to Litigation. That was where I first learned about mediation. I’m sure we also covered arbitration, but negotiation principles and mediation are what I remember best. I actually enjoyed the exam in that course — an unusual experience! — because we needed to think about what the parties wanted and what their underlying interests likely were. We also needed to think practically when different alternatives existed that might be responsive to their underlying interests. I found the problem-solving aspect of the course to be really exciting.

I also had the opportunity to take a mediation training when I was in law school and then to mediate in a small claims court.

What are some of the big questions related to ADR that interest you or that you are currently focusing on?

I’ve always been interested in the intersection between negotiation, mediation, arbitration and the courts, which, of course, is where RSI largely is located. I have been a big advocate of negotiation, mediation and arbitration when the parties have actually selected these processes, and when the processes have been managed in a manner that helps to ensure that people really have the opportunity to think and share what they care about, to think about what options might exist for resolution, to be fully informed, and to freely make their choices.

One thing that really excited me about negotiation and mediation was that it seemed as though — especially once you started asking about and looking at underlying interests — a whole new path to resolution opened up. I had gotten to a point where it seemed to me that when we were talking about the law, there was no path; there were only positions and legal arguments.

So when people are choosing negotiation or mediation, when they really have the opportunity to be informed and to explore what other options exist that can be responsive to their needs, I am entirely in favor of these processes.

When the mediation process is one in which mediators or lawyers decide that the parties are never going to be in direct contact with each other — putting them in separate rooms and with the mediator just shuttling back and forth — and when the focus of mediation is primarily on getting the parties to be more realistic in the way that the mediators and lawyers want them to be, I am less enthusiastic about the processes. Some reality-testing is almost inevitable, but the mediation process also should provide the opportunity for the parties to express what is really important to them, to be heard in a dignified setting, and to explore options that meet their needs. I care a lot about procedural justice and self-determination and have written extensively on both.

Importantly, we really don’t know what happens in most mediations. The courts don’t regularly collect or publish such data. Again, this is a world in which RSI operates; RSI has been involved in much more research and evaluation than a lot of organizations and encourages court-connected programs to evaluate and collect data. That is really important. How can you know what is going on if you don’t have any information?

Meanwhile, I know that a lot of the data we do have indicates that people are satisfied with the mediation process. So that’s important. I just think the process can be one that enables people to fully exercise self-determination consistent with the American ideal of democracy, that each of us is a thinking human being who can be educated and make good decisions. And then, of course, we also need data regarding the other dispute resolution processes.

What in your current academic work, if anything, relates to the work of RSI?

My academic work relates to RSI’s work because RSI does so much with data and evaluation, and a lot of my writing has been about mediation and court-connected processes, which are obviously core areas for RSI.

I wrote a series of articles (one of which is “But Is it Good: The Need to Measure, Assess, and Report on Court-Connected ADR”) that focus largely on the need for more data, for regular reporting by the courts regarding their use of dispute resolution processes. How many cases were eligible? How many cases actually went to these dispute resolution processes? Did they settle? On what terms? What were parties’ perceptions of the procedures and outcomes? And then I also have urged that courts have some responsibility to ensure substantive fairness in the aggregate, or at least some responsibility to ensure that there is not a systemic pattern of unfairness in outcomes.

What attracted you to/made you want to join the RSI board?

I have thought for a long time that RSI is a really wonderful organization and it’s doing important work. When I identify who is out there focusing on court-connected dispute resolution, helping to ensure that research is being done and that courts are getting the kind of assistance they need to provide good court-connected dispute resolution processes, RSI has been at the center of it. Jen Shack is a wonder. Susan Yates is a wonder.

What are you most looking forward to during your time on the RSI board?

I think there are amazing people who are involved with RSI, so I’m looking forward to getting to know the staff and the board. I’m excited to be working with people on the board — some of whom I’ve known and respected for a long time, and others whom I’ve known by reputation but have never before met. It’s an honor to join them.

Tags: access to justice, ADR, court adr, fairness, mediation, self-determination
Posted in Program Design, Program Management, RSI Board of Directors, RSI News, Welcome | No Comments »

Want Your Court Communications to Be Accessible? RSI Focus Groups Offer Insights

Rachel Feinstein, October 16th, 2023

RSI’s research has shown that self-represented parties in small claims cases often don’t understand what online dispute resolution (ODR) is or how to use it, even when courts require their participation. To learn what self-represented parties need when a small claims case is filed against them, RSI’s OPEN Project is going to the source — conducting focus groups with people similar to these parties and asking what works for them.

**Participants in an RSI focus group in Texas provide feedback on sample court documents in October 2023.**

Director of Research Jennifer Shack and I led two focus groups in rural New Hampshire in August, followed by two groups in Texas in early October. We will finish our data collection for the ODR Party Engagement Project in Maryland this month. In the meantime, we want to share some of the initial insights we have gained.

Hearing from 26 participants so far, we have learned about many of the barriers people experience when faced with examples of court documents, a court website and instructional court videos. Groups also shared their recommendations for how the material could be improved and their preferences for receiving court notifications and instructions. The majority of participants have a maximum of high school education. Most, if not all, participants earn less than $50,000/year. These income and education characteristics parallel the backgrounds typical of self-represented litigants, making their insights regarding the comprehensibility and usability of court material invaluable as we aim to develop recommendations for accessible court resources.

Notification Preferences Vary Widely

We are excited to share some preliminary findings from our focus groups. First, we have learned that providing court resources in a variety of formats is essential to addressing the public’s needs and preferences. Focus group participants expressed minimal consensus about the ideal way to learn about their involvement in a lawsuit or how to proceed with online dispute resolution. For example, only half of the 26 participants said they would prefer to receive an initial notice about their lawsuit through the mail. Six people would prefer to receive notice about their case over the phone, while five would prefer text message, and only one person wants to learn of their case via email.

“There are times where the form of a video works wonders in comparison to throwing a chapter out of a book at me or something.”
— Focus group participant

Further reflecting this need for variety, participants in two of our groups were enthusiastic about using instructional videos to learn about registering for ODR. One participant in New Hampshire shared, “I think a video would be good. Where they could break it down and explain it a little bit more in depth.” Another person agreed, “Yeah, I think so. I mean, I’m a visual learner … If this was on YouTube … everything would be fine. It’d be perfect.”

A third participant added, “There are times where the form of a video works wonders in comparison to throwing a chapter out of a book at me or something.”

In contrast, most participants in the Texas focus groups did not express a need or interest in viewing videos to get this information. But several people did agree that, as one said, “options are good,” when attempting to meet the potential variety of needs, learning styles and preferences among self-represented litigants.

Participants Wary of Possible Scams

**RSI focus group participants in Texas shared their recommendations for how court informational materials could be improved and their preferences for receiving court notifications and instructions.**

One topic where focus group participants were largely in agreement was their concern about being scammed. During the focus groups, we asked all participants to look at one of two ODR websites on a laptop or tablet that we provided. The first step many participants took was to assess the credibility of the website. For instance, the first reactions routinely included comments about whether the site was legitimate or a scam. Some participants also expressed apprehension regarding receiving the mailed Notice to Defendants, wanting to contact the court to check that it was legitimately a lawsuit against them before following the instructions on the document.)

This initial step of assessing documents and websites for legitimacy may be crucial for courts to be aware of when developing their communications and other resources, since apprehension about whether the material is trustworthy could inhibit people from beginning the process.

Simple, Organized Info Is Desired

One of the most consistent themes among the participants so far has been the desire for court resources to be simple and quick to use. For instance, we heard from many individuals who want courts to use simple language, concise instructions and well-organized documents or videos. Some participants specifically requested more spacing around paragraphs, and people found sections with bullet points or short fill-in-the blank questions easy to understand.

We anticipate delving more deeply into strategies for making court resources simpler to use and comprehend. Additionally, participants have been identifying key information that is missing from the material and sharing their emotional responses to the court resources. We look forward to examining these and other themes in more detail after we conclude our focus groups later this month.

Check back soon for a summary of our findings and a guide for courts, which we will provide on a new RSI webpage this spring!

As always, RSI is grateful to the AAA-ICDR Foundation for supporting this important work.

Tags: court programs, online dispute resolution (ODR)
Posted in Accessibility, Diversity and Inclusion, Online Dispute Resolution, Program Design, Research, RSI News | No Comments »

RSI Guide Will Help Courts Make ODR Communications More Accessible

Rachel Feinstein, September 14th, 2023

In 2022, RSI and the University of California, Davis, published two evaluations of court-related online dispute resolution (ODR) programs. A main finding of those evaluations was that parties did not have sufficient information about the programs they were being required to use. We went on to look at programs in other courts and found that the information available to parties about ODR varied, and that courts, despite making significant efforts to provide this information, needed help to do so. This was especially true for communicating with self-represented litigants. Our findings have led RSI to conduct new research that will result in a guide for courts that need to communicate with parties about their ODR programs.

The Case for Accessible Court Communications

**RSI Researcher Rachel Feinstein, pictured, and Director of Research Jennifer Shack facilitated focus groups for RSI’s ODR Party Engagement (OPEN) Project in Berlin, NH, in late August.**

Across the US, 72% of family law cases and 76% of civil cases involve at least one self-represented litigant. As more people are handling civil cases without a lawyer, it is increasingly important that everyone can understand and use court communications, regardless of their educational background. Accessible court communications can reduce default rates and increase access to court services and programs, including ODR programs.

Half of the adult population in the US struggles to read lengthy, dense texts to complete tasks and accurately answer questions. Developing court resources with this in mind can significantly improve people’s understanding of how to access court resources, and can increase participation rates in various programs. One recent study found that simplifying the text used in court forms improved participants’ understanding of the purpose of a subpoena from 23% to 70%. Studies have also found important patterns in the reading styles and strategies of people who have low literacy, such as avoiding dense blocks of text and ignoring information on the left and right sides of the web page. Effective court communications take these types of reading patterns into account.

Digital Literacy is Another Challenge

But language and writing styles are not the only issues. Many people rely on websites and online resources provided by courts to gather the information they need, provide information to courts, and participate in online court programs. Despite widespread use of the internet, a large percentage of US adults struggle with digital literacy, or the ability to use digital technology to find information, complete tasks or communicate.

When designing websites or other digital resources, courts can benefit from recognizing common patterns among people who have low digital literacy skills. For example, people with low digital literacy will often avoid the use of search boxes, opting instead to link surf. And it is common to satisfice quickly, or give up before finding necessary information. In RSI’s forthcoming guide for courts, we will provide more information about reading patterns and strategies common among people with low literacy and low digital literacy, with the aim of supporting courts in developing ODR material everyone can use.

Helping ODR Serve More Parties

The movement to use accessible court communications helps both parties and the courts. For example, civil courts throughout the country are investing in ODR programs. Many people can benefit from the convenience, lower cost and less intimidating process of resolving conflicts online. However, ODR participation rates remain very low, even when it is mandatory. If self-represented litigants understand the steps they need to take, more may take part in their case and do so in a more informed manner. This, in turn, will help courts by increasing participation in ODR.

The experience of the ODR programs we evaluated provides a valuable example of the need to develop effective court communications to help parties to more easily navigate court services and to improve participation in court programs. In a typical ODR process for debt and small claims cases, the court requires that defendants be given a notice of the ODR program. This notice provides written instructions to register on a third-party platform. Once they register, they can try to resolve their case with the plaintiff before their first hearing. The defendant will likely complete the entire process without interacting with court staff. Further, ODR is unfamiliar to most people, increasing the importance of courts’ efforts with written communications — and sometimes instructional videos — to provide parties adequate information to participate in their case.

Maximizing Court Programs’ Potential

If done well, court communications can narrow the information gap and maximize the potential of a variety of court ODR programs. Enhancing access to justice requires the development of effective and accessible court communications that people from all backgrounds and education levels can easily understand and use. RSI’s ODR Party Engagement (OPEN) Project is working to support courts in this effort by conducting focus groups in three areas of the US. Through these focus groups, we will hear from a diverse group of people who have a low income and low education, to learn what works for court notices, guides, websites and instructional videos pertaining to ODR. We will use our data, along with previous research, to develop a broadly applicable guide for civil courts to use when developing communications that reduce barriers to participation in ODR for people with low literacy.

This project is generously supported by the American Arbitration Association-International Centre for Dispute Resolution Foundation.

Tags: court programs, online dispute resolution (ODR)
Posted in Accessibility, Online Dispute Resolution, Program Design, Research, RSI News | No Comments »

Courts Can Take Steps to Design Text-Based ODR Programs that Better Serve Parties

Jennifer Shack, June 22nd, 2023

While conducting two of the first independent evaluations of text-based online dispute resolution (ODR) programs in U.S. state courts, Donna Shestowsky and I found those programs promoted access to justice in some ways, but inhibited it in others. To help other courts, we wrote an article about how they might reduce potential barriers when developing and implementing their text-based ODR programs. The following is a summary of our advice from the article, “Access to Justice: Lessons for Designing Text-based Court-Connected ODR Programs,” which was recently published in Dispute Resolution Magazine, a publication of the American Bar Association.

Court adoption of text-based ODR allows parties to communicate asynchronously, at their convenience, from anywhere. This suggests that ODR has the potential to increase access to justice, particularly for self-represented litigants,[i] and could lead to increased efficiency and reduced costs for parties and courts alike.[ii] Conversely, however, for parties who lack digital literacy or access to technology, mandated ODR could instead benefit already advantaged parties and leave others behind. Furthermore, in some instances, mandating ODR could reduce access to justice by overriding consent and party self-determination.[iii]

The Texas and Michigan Programs

The programs we evaluated differed in the issues involved and the platforms used. In Collin County, Texas, we assessed a debt and small claims pilot program in a busy Justice of the Peace Court (JP3-1) that used the Modria platform. In Ottawa County, Michigan, we examined a program for post-judgment family matters brought to the Friend of the Court (FOC), an agency under the aegis of the Chief Judge of the 20th Circuit Court. The FOC used the Matterhorn platform. Both programs, however, were intended to be mandatory once the program was referred. And both required that the parties register and communicate via text on the ODR platforms.

Litigant survey responses suggested that many parties were unaware of the ODR program or did not understand its main features. When asked what would make them more likely to use ODR for a similar case in the future, half said more information.

Although the programs we evaluated used different ODR platform vendors, the platforms worked similarly and had comparable limitations. The platforms provided a chat space and permitted third-party facilitation or mediation. Neither was available to those with significant visual impairments or limited English proficiency. Both allowed only one individual per side to participate. This limitation meant that in Texas if a party had a lawyer, the lawyer participated alone. In Michigan, only parties could participate, and those who had lawyers were not referred to ODR.

Possible Reasons for Not Using ODR

Although ODR was ostensibly mandatory in both programs, the majority of parties in each court did not use ODR. In Texas, both parties to a case used the platform in only 81 of 341 cases (24%) referred to ODR. In Michigan, ODR use was twice as high: For the 102 matters in which caseworkers determined ODR was appropriate, 48% used ODR. In 26 of the 53 matters in which the parties in the Michigan program opted not to use ODR, at least one party did not register on the platform.

Survey and interview data suggest a few reasons parties did not use ODR. In both programs, staff indicated they did not send parties who lacked digital literacy to ODR, and litigant survey responses suggested that many parties were unaware of the ODR program or did not understand its main features. In the Texas program, of those who did not use ODR, only one survey respondent (out of ten) indicated having received information about the program. When asked what would make them more likely to use ODR for a similar case in the future, half said more information.

In survey responses for the Michigan program, parties appeared to lack a basic understanding of how ODR worked. Half of the 50 parties surveyed near the start of their matter did not know ODR was offered free of charge.

According to Texas court staff, litigants received information about the ODR program via the notice the court sent to them (or their lawyers) about their court date, and through an email or text from the platform when the court uploaded their case to it — if the court had their email address or cellphone number. Both the notice and the email lacked information about how ODR worked. Similarly, the Michigan program’s automated email and text, platform, and FOC website missed opportunities to educate the parties.

Implications for Courts

Despite their accessibility issues, both the Texas and Michigan programs had similar access to justice benefits. Our evaluations suggest that for those parties who use ODR, the process is convenient. We found that 72% of ODR use in Texas and 52% in Michigan occurred outside of court and office hours, i.e., at times not available to them in traditional dispute resolution methods. However, in both programs, many parties simply did not register to use ODR. In addition, 50% of ODR users who responded to our survey noted that they liked that ODR was easy to use. These findings indicate that ODR can increase convenience.

Nonetheless, our finding that some parties lacked information or had nontrivial misconceptions about ODR also suggests parties did not always make informed decisions about whether to participate. To enhance access to justice and self-determination, courts should incorporate a communications plan. The plan should:

Specify how parties can learn about the program and detail what information court personnel should relay about ODR
Indicate what information about ODR to include on the court’s websites and the ODR platform to educate parties about how to use ODR and its potential risks and benefits
Outline outreach efforts to urge social services or other agencies to inform their clients about the ODR program

Additionally, courts should present information about ODR in a way that is comprehensible to individuals with low literacy. They should also explain the privacy and confidentiality implications of using ODR, especially regarding whether and how communications shared on the platform might be used in subsequent legal proceedings.

Further, ODR offerings should be accessible to all eligible parties. Courts should urge ODR providers to facilitate use by parties with visual impairments and limited English proficiency. Additionally, courts should direct parties who do not have reliable internet access to computers in the courthouse or other community locations — though as a result of limited business hours and privacy concerns, this solution is far from ideal.

Courts should also ensure that text-based platforms are user-friendly for smartphone users. In the Michigan program, 71% of participants exclusively used a smartphone for ODR. (We did not have information on the devices Texas ODR participants used.) Yet our findings indicate that text-based ODR may be difficult for smartphone users. Courts should urge ODR providers to include in-app voice control to facilitate ODR use on smartphones generally, a change that might be especially important for individuals with disabilities that restrict their ability to type. Parties should also be able to participate in ODR with their attorneys.

Finally, courts should explore ways to maximize access to their platforms for those who lack digital literacy. Usability testing, similar to that conducted for Utah’s ODR pilot program,⁸ can help identify challenges and potential solutions for given platforms. Courts might also consider providing parties with links to web-based resources or trainings that could increase their comfort with technology.

Given ODR’s current technological limitations and the percentage of the population that continues to lack reliable internet access or digital literacy, ODR is not a panacea for the continued access to justice problem in the U.S. Additionally, our evaluations suggest that parties have different preferences for how to resolve their disputes. To enhance access to justice, and to advance party self-determination, ODR might best serve parties as part of a constellation of alternative dispute resolution (ADR) options rather than being the only form of court-connected ADR.

[i] Amy J. Schmitz, Measuring “Access to Justice” in the Rush to Digitize, 88 Fordham L. Rev. 2381 (2020).

[ii] Amy J. Schmitz, Measuring “Access to Justice” in the Rush to Digitize, 88 Fordham L. Rev. 2381 (2020).

[iii] Amy J. Schmitz & Leah Wing, Beneficial and Ethical ODR for Family Issues, 59 Fam. Ct. Rev. 250 (2021).

Posted in ADR Resources, Family, Online Dispute Resolution, Program Design, Research | No Comments »

Ten Tips for Evaluating Your ODR Program

Jennifer Shack, May 18th, 2023

After Donna Shestowsky and I completed two of the nation’s first neutral evaluations of state court online dispute resolution (ODR) programs, we had some thoughts to share with courts about how they could best ensure the evaluations of their ODR programs were useful and of high quality, so we wrote an article, “Ten Tips for Getting the Most Out of an Evaluation of Your ODR Program,” which was recently published in Court Review, the journal of the American Judges Association. The following is a summary of what we wrote. For more complete guidance, see the complete article.

Tip 1: Negotiate data access when contracting with an ODR provider

RSI Director of Research Jennifer Shack, second from left, and University of California Davis Professor Donna Shestowsky, second from right, are the authors of “Ten Tips for Getting the Most Out of an Evaluation of Your ODR Program.” They joined Nick White, Research & Evaluation Director of the Maryland Judiciary’s Mediation and Conflict Resolution Office, and Dr. Deborah Goldfarb, Assistant Professor at Florida International University, on a panel at the American Bar Association Section of Dispute Resolution Spring Conference.

The best time to ensure you will have the data needed for a future evaluation, and to monitor program activity in general, is when you negotiate your contract with an ODR provider. As you screen providers, learn how each ensures data security and confidentiality. This is also the ideal time to negotiate data access for your evaluation. To keep your evaluation options open, you will want to secure terms that obligate the provider to share data not only with you, but with external evaluators you might hire later. And whether you hire a provider or develop an ODR platform in-house, be mindful of any data sharing or confidentiality rules or policies in your jurisdiction for the types of cases you plan to include in your evaluation.

Tip 2: Determine when to evaluate

Ideally, you would plan the evaluation of your ODR program as you design your program. But evaluation planning can happen at any time, as can the evaluation itself.

If you plan your evaluation before launching your program, you can increase the probability that the evaluation will accurately reflect your program’s use and effectiveness if you begin your evaluation after 1) the provider has addressed technology glitches that may emerge during early testing of your platform and 2) after you conduct outreach to ensure parties know about your program. If your program is already up and running, you should avoid scheduling an evaluation for a time frame when major changes are planned for your court or the ODR program itself. Significant changes while data are being collected may introduce noise into the data.

Tip 3: Find a neutral evaluator

Selecting a neutral evaluator is important for enhancing the quality, usefulness and credibility of your evaluation. In choosing your evaluator, consider whether they have experience evaluating court alternative dispute resolution (ADR) and/or ODR programs. If you hire someone who is not knowledgeable about ADR, be prepared to spend a lot of time explaining how ADR works, the theory behind it, and the specific issues involved.

Tip 4: Ensure that key personnel are involved in the evaluation planning process

Include court personnel who have knowledge that can assist with evaluation planning. These individuals are, at minimum, judges hearing the cases served by the ODR program and court staff who understand the processes involved and the underlying technology. They can help you determine the questions to answer, identify what data are needed, or work out how to access relevant existing data.

You will also want to decide who should serve as the point of contact for your evaluators. This person will answer the evaluator’s questions and help to obtain data. Staff members should be clear on their role in the evaluation.

Tip 5: Prepare to use data from a variety of sources

To best understand your ODR program, you should obtain information from multiple sources, such as your case management system, the ODR platform and participant surveys. To collect systematic feedback from parties (or other stakeholders, such as lawyers), your evaluator will need your help to facilitate distribution of surveys. They will work with you to determine which parties to survey and the best method for contacting the parties. They should also ask you to review the survey questions.

Tip 6: Expect to spend time with the evaluator

To conduct an effective evaluation, your evaluator will need to thoroughly understand your ODR program, how it fits with your overall process for handling cases, and how the platform interfaces with your case management system. Your evaluator will want to spend time with you to discuss your program processes and get answers to any questions throughout the evaluation process.

Tip 7: Facilitate the participation of court personnel and other program partners in the evaluation

Give court personnel and other program partners (e.g., mediators) who were not a part of the evaluation planning process a heads up about the evaluation and ask for their cooperation. Introduce your evaluator to relevant personnel and partners. These efforts should pave the way for your evaluator to reach out to them to get their perspectives on the ODR program and its impact on their work. When asking court personnel and program partners for their cooperation, reassure them that the evaluation’s objective is to improve the program, not to find fault with it or with them.

Tip 8: Help your evaluator to pilot test their survey materials

Before the evaluation period, your evaluator should obtain feedback on their surveys from individuals similar to those who will be surveyed for the evaluation—typically, parties to similar cases. Your evaluator will need your help to gain access to those individuals.

In addition, every court has unique terminology that should be reflected in how questions are worded. Your evaluator should work with your staff to ensure that the correct terminology is used so that it is more likely to be understood by those who will be asked to complete the survey.

Tip 9: Be flexible about the length of time set aside for data collection

The time allocated for data collection needs to be long enough to get the data necessary for analysis, but measured enough to provide timely results. Your evaluator will work with you during the evaluation planning phase to determine the time frame. But be prepared to be flexible. Your evaluator may recommend extending the data collection period if the level of program use and/or survey participation is lower than expected and they need more time to collect data to deliver a useful evaluation.

Tip 10: Survey those who use ODR as well as those who do not

Our evaluations have shown that the motto “If you build it, they will come” does not always apply to ODR. Surveying eligible parties who did not use ODR could help you identify issues that might be driving lower-than-expected usage. Surveys can point to marketing or party education problems, or in the case of voluntary ODR programs, uncover program attributes that parties find unattractive. You can ask parties whether they knew about your program, how and when they learned about it, and whether they knew they were eligible.

In the end, when a court invests resources to establish an ODR program, a major goal is to have it be used. It is imperative to commit resources to effectively market the program, which should include efforts to educate parties and ensure they know they are eligible or required to use it.

Conclusion

Courts that have their ODR programs objectively evaluated should be applauded for their efforts. Evaluations can facilitate program design that is data-driven and evidence-based, rather than guided by anecdotes or hunches. This grounding in data is especially important when making decisions geared toward satisfying the interests of litigants, since understanding their unique perspectives requires collecting data directly from them. Ideally, ODR evaluations will be conducted by neutral third parties who have no stake in the results and meet high research standards. Neutral evaluations are uniquely situated to offer an outside perspective on what works well about a program and to suggest how it might be improved. In addition, constituents, including lawyers, are more likely to accept the findings of a neutral, outside evaluation that concludes that a program delivers beneficial outcomes.

Posted in Evaluation, Online Dispute Resolution, Program Design, Program Evaluation, Program Management, Research | 1 Comment »