Resources / Study / Innovation for Court ADR

Just Court ADR

The blog of Resolution Systems Institute

Archive for the ‘Program Evaluation’ Category

Characteristics of Quality Court ADR Programs

Susan M. Yates, September 18th, 2019

What characteristics do you think are shared by quality court ADR programs? I took a swing at a list here. What would you add? Change?

1. Goals

There must be a shared understanding of the goals of the ADR program so that there can also be a shared understanding about whether it is succeeding.

  1. The court must be clear about its goals for the ADR program
  2. Those goals must be shared with stakeholders

 

2. Principles

The ADR program must operate on a common set of foundational principles, including ethics as appropriate for the ADR process being provided.

  1. Confidentiality (in mediation)
  2. Fairness of process
  3. Fairness of outcomes
  4. Procedural justice
  5. Accessibility
  6. Neutral quality
  7. Self-determination (in mediation)
  8. Timeliness

 

3. Accessibility

Everyone – including litigants and lawyers – must be able to readily access the ADR program.

  1. Parties who are unable to pay for ADR are afforded the opportunity to use ADR
  2. Parties who are representing themselves are able to participate fully in ADR
  3. Parties with disabilities are able to participate fully in ADR

 

4. Process Quality

Notwithstanding the importance of other characteristics, the true quality of a court ADR program boils down to what happens during each ADR session.

  1. Whichever ADR process is being used, it adheres to the foundational principles of that process
  2. Participants have an experience of procedural justice when engaging in the ADR process:
    1. They feel they had a voice in the process (e.g., had a chance to talk, felt they were heard)
    2. They feel they were respected in the process
    3. They feel the process was fair to them

 

5. Program Support

The ADR program will only succeed if it receives steady support from a number of sources.

1. Stakeholders understand and support the ADR program

  • Judges
  • Litigants
  • Court staff
  • Neutrals

2. The program has sufficient, stable financial support

 

6. Neutrals

Neutrals are the face of the program to litigants and lawyers, so they must provide quality services.

  1. Neutrals share a common understanding of the service they are to provide
  2. Neutrals provide services in the manner expected by the program and its stakeholders
  3. Neutrals operate in an ethical manner
  4. Neutrals have the necessary skills and knowledge
  5. Neutrals are selected for the roster fairly
  6. Neutrals are appointed to cases in a fair manner
  7. Neutrals receive appropriate initial and ongoing training
  8. Neutrals provide a sufficiently uniform version of the ADR process
  9. Parties have a way to lodge complaints about neutrals
  10. Neutrals are treated fairly when a complaint against them is made
  11. Neutrals are compensated fairly (which does not preclude volunteer mediators)
  12. Neutrals are assessed fairly
    1. Participant surveys
    2. Peer review

 

7. Lawyers

Lawyers must support, or at the very least accept, the ADR program.

  1. Lawyers are often repeat users of the ADR program, therefore their support is especially important
  2. This applies equally to those who typically represent one side or the other, e.g., landlords’ lawyers and tenants’ lawyers.
  3. Lawyers should:
    1. Find the program useful
    2. Be educated about the program
    3. Fulfill their responsibilities in the program

 

8. Program Safety

ADR processes must be safe, both literally and figuratively.

  1. Participants, neutrals and staff are all safe when participating in ADR and otherwise interacting with the ADR program
  2. Participants are screened prior to ADR, when appropriate, to identify intimate partner violence and other potential barriers to participation

 

9. Data Collection and Dissemination

Reliable data must be collected regularly and shared appropriately, or it won’t be available when it is time to sustain or improve the ADR program.

  1. Data is collected regularly and includes both quantitative and qualitative information (e.g., statistics and success stories)
  2. Changes in the program are tracked and acted on, e.g., changes in:
    1. The number of cases being sent to ADR
    2. The number of agreements being reached
    3. The number of mediators signing up to mediate
    4. The number of parties showing up for ADR sessions
  3. Data is turned into reports that can be readily digested
    1. Easy to read
    2. No use of insider language or acronyms
    3. Fitting use of charts and other visuals
  4. Reports are adapted to their particular audience and disseminated appropriately, typically as follows:
    1. Judges and court administrators with direct responsibility get the most detailed reports
    2. Those higher up in the court system get summaries with explanations
    3. Funders’ reports depend on what the funder requires. They may also get some anecdotes about how parties are experiencing the program.
  5. Reports include recommendations for action when appropriate
  6. The program is evaluated near the end of the first year, after a substantial period of time, or when there has been a major change in the program or the context in which the program operates

 

10. Program Promotion

Reminding stakeholders of the value of the ADR program will help maintain its support.

  1. Brochures are made available in courtrooms and other locations as needed
    1. Brochures are only produced if they will serve a particular purpose, such as giving parties the number they need to call to schedule a mediation
    2. If brochures are targeted to self-represented litigants, they must be in easy-to-understand language
  2. News about the program is publicized via:
    1. Local radio, TV, newspaper, news websites
    2. Court newsletter
    3. Bar association newsletter
  3. Program is featured on appropriate websites
    1. Information about the ADR program can be readily found on the court’s site
    2. If the program maintains its own site, instead of appearing as part of the court’s site, it must be easy to find and navigate and kept up to date
  4. As members of stakeholder groups change, the new individuals are familiarized with the program
    1. Judges are educated on how it works, ethical limitations, what to expect
    2. Lawyers, e.g., child protection attorneys, assigned to courtrooms are trained in how to use the program
    3. Staff at funders are educated about the program’s benefits and history
    4. Court staff are educated about the program’s benefits and how it fits into court processes

 

11. Program Administration

Someone wakes up every morning with the feeling that it is their job to do everything on this list to ensure the quality and continuity of the ADR program.

  1. Individuals with power over the program, e.g., funding or case referrals, are kept informed about the program
  2. The ADR program operates in accordance with applicable laws, court procedures and rules
  3. The program functions efficiently and effectively, e.g.:
    1. Cases are referred to ADR in a consistent manner
    2. Cases are scheduled promptly
    3. Reports are provided to court on time
    4. The benefits provided by the ADR program are reasonable in relationship to the costs of the program
    5. Changes in the program are tracked and potential responses suggested when needed

 

Participants Are Highly Satisfied with Nevada’s Child Protection Mediation Program

Jennifer Shack, July 1st, 2019

Last month, I talked about a new evaluation of child protection mediation in Michigan. I’m following this up with a 2017 evaluation of child protection mediation in Nevada. Both evaluations were of several programs taking place throughout the respective states, but their focuses are quite different. Where the Michigan study primarily examined time to permanency, the Nevada study focused much more on participant experience in the mediation and process issues.

The Nevada study, “Process Evaluation of Nevada’s Statewide Dependency Mediation Program,” by Shamini Ganasarajah, et al, of the National Council of Juvenile and Family Court Judges, found high levels of satisfaction with mediation and agreement, as well as a possible impact on whether scheduled hearings after mediation were cancelled. The study also found that there was no difference in satisfaction rate based on the stage at which mediation occurred, but that satisfaction was higher when mediation resulted in agreement as compared to when it did not.

The study looked at mediation in seven counties. In these counties, mediation can be used at any point of the case. However, most cases used the mediation program at the termination of parental rights (TPR) stage, which is at the end of the case. (This finding regarding the timing of mediation is skewed somewhat by one county using mediation almost exclusively at the TPR stage.) Time in mediation averaged two hours.

Those who participated in mediation were asked to complete post-mediation surveys. For the purposes of this study, these people were divided into program participants (these are natural parents and foster parents) and system stakeholders (the attorneys and case workers involved in the case). During the study period (July 2016 through April 2017), participants completed 113 post-mediation surveys and stakeholders completed 267. In their responses, 84% of the participants and 98% of the stakeholders expressed satisfaction with the mediation program. Their satisfaction was statistically related to whether they reached agreement in the mediation.

The participants (family members) were highly positive about all aspects of the mediation. All of them thought the process was fair. Almost all said they were able to voice their opinions, were treated with respect and were able to be a part of finding answers to the problems discussed. Almost 90% said the others really listened to them. For all practical purposes, these responses did not vary based on whether they were foster parents, natural mothers or natural fathers.

The evaluators analyzed whether there was a relationship among the participants’ responses. One that stood out was that when participants believed others in the mediation had “really listened” to what they had to say, they were more likely to express satisfaction with the mediation regardless of whether an agreement was reached.

The stakeholders (attorneys and caseworkers) were also highly positive about the mediation, with all or almost all believing the process was fair, that they had an opportunity to express their opinion, were treated with respect, were listened to and were able to be a part of finding answers to the problems discussed.

Interestingly, both participants and stakeholders were most likely to mention communication as what was most helpful about the mediation. Both groups were also most likely to say that parties being unable or unwilling to compromise was the reason no agreement was reached.

Also interesting was that the mediators reported agreements in 84% of cases, while the stakeholders reported that agreement resulted from only 71% of their mediations. There is no explanation as to why. The study also found that hearings were cancelled after 51% of the mediations were held. The evaluators recommended further examination of the relationship between mediation and vacated hearings.

Other recommendations included expanding the use of mediation to all stages of the case, as most mediations occurred at the TPR stage; assessing implementation of domestic violence screening protocols; and enhancing mediator training to include additional strategies for effectively listening to participants and stakeholders and making them feel heard.

Saving Homes, Building Understanding

Eric Slepak-Cherney, November 29th, 2018

Resolution Systems Institute is proud to share its latest publication, Saving Homes, Building Understanding: An Evaluation of the Eight Foreclosure Mediation Programs Funded by the Illinois Attorney General. This new evaluation looks at four-plus years of data across eight different programs to provide a comprehensive analysis of foreclosure mediation in Illinois, and to highlight how differences in program models impacted outcomes. (more…)

Looking at Process and Outcome to Improve an Effective Program

jenshack, August 28th, 2018
I recently had the great pleasure of evaluating Washington D.C.’s Child Protection Mediation Program. The court was interested in getting a better understanding of the issues involved in the process, such as whether the timing of mediation should be changed and whether the process was working for the participants. Therefore, the evaluation was a comprehensive examination of the outcomes and impact of mediation, as well as the program process from referral to completion of mediation.

To conduct the evaluation, I interviewed parents and judges, observed mediations and court hearings, developed post-mediation surveys and analyzed court files and program data. RSI Executive Director Susan Yates and I also conducted focus groups with groups of professionals who participate in mediation (guardians ad litem, parent’s attorneys, Assistant Attorneys General and social workers) as well as with program mediators. The results of the evaluation, Improving an Effective Program: A Comprehensive Evaluation of the Superior Court of the District of Columbia Child Protection Mediation Program, pointed to the program effectively achieving its goals, but needing to address some issues surrounding the program process.

Mediation in this program is mandatory and is supposed to occur within 30 days of the initial hearing. The goals for mediation, as expressed by the attorneys, social workers and judges, are to:

  • Make progress on the legal issues in the case and reach agreement on the stipulation (an agreement as to the facts of the case that takes the place of trial)
  • Help parents to understand their situation and their responsibilities going forward
  • Increase professionals’ understanding of the case, the parents and the family’s situation
  • Enhance communication among the professionals
  • Provide parents with an opportunity to talk about their concerns and be heard
The court and professionals agree that mediation at this early stage of the case brings all those involved in the case together at an opportune time to discuss the issues involved and progress that should be made at a time when they normally would not be focused on the case. This allows them to exchange more information, update services and visitation, and ensure that professionals are being held accountable for their tasks. Additionally, mediation is seen as a unique forum for parents to be important in the process and to speak about their concerns and be heard. When mediation is conducted early in the case, the exchange of information and parents’ input help professionals make better decisions. Mediators assist in this process by facilitating the discussion, clarifying and summarizing the points being made, reframing position statements in a more positive way, and, at times, supporting the professionals’ goals for the parents.

The results show that the program is generally achieving its goals. Of significance, parents who mediate are twice as likely to stipulate before trial as those who don’t mediate. Further, it is likely that they are more compliant with services, although limitations to the data make it impossible to state this with certainty. Limitations to the data also made it difficult to draw conclusions about mediation’s effect on time to permanency. The evidence, however, points to mediation not having an effect on the time it takes for a child to have a permanent home.

Other results indicate that the percentage of mediations ending with a signed stipulation declined from 2013-2014 to 2017, from 44% to 25%. This was most likely due to a policy change at Child and Family Services Agency in 2015, which judges and professionals said led to only the most challenging cases being brought to court. Despite this, approximately half of the mediations in 2017 ended with some progress on the stipulation.

Importantly, both parents and professionals are gaining understanding through mediation. Almost all parents who completed surveys after mediation said they better understood the points of view of the others at the mediation, as well as what they had to do next. The vast majority of professional participants who completed surveys believed that they gained understanding of others’ points of view and the parents’ situations. In their survey responses, almost three-quarters of parents said they trusted that those involved in their case wanted to do what was best for their children. Parents who were interviewed shed light on the effect of mediation on their level of trust in the professionals. Half of the parents interviewed trusted the professionals before they participated in mediation. Mediation for them was an opportunity to see once more that they could trust them. Of the other half, three entered mediation lacking trust in at least one of the professionals and nothing in mediation led them to change their minds; while for a two, mediation did change their minds from seeing the professionals as being against them to learning they could be trusted.

More than three quarters of the parents were satisfied with the mediation and 83% believed it was helpful to them. Both parents and professionals believed they had an opportunity to talk about what was most important to them and that they were understood. Most parents believed the mediator and, more importantly, the professionals, treated them fairly and with respect. All professionals believed that the mediator treated them fairly and with respect.

Despite the success the program has in achieving its goals, the program process can be improved. One such opportunity lies in the timing of mediation. There was general agreement among professionals that mediation shouldn’t take place within 10 or 15 days of the initial hearing because enough time needs to elapse in order to make the most effective use of mediation. Mediation too soon means that not enough time has elapsed for new information to be available or for parents and professionals to have started taking the steps required of them in the initial case plan. Without these, discussion in mediation is less productive. On the other hand, there was little interest in extending the deadline much beyond 30 days because some of the benefits of mediation are lost if it takes place too late. Generally, almost all professionals thought mediation around 30 days was an ideal time. Despite this, about 1/3 of mediations were scheduled either within 15 days or after the 30-day deadline. It was recommended that the court increase the deadline to 40 days so that more mediations could be held after 15 days had elapsed, but still not too late to keep the case moving forward.

The biggest complaint among the professionals in the focus groups was that mediations didn’t start on time. A review of the data found that most of the time, the delay is due to either a professional or parent arriving late. The mediation was often further delayed because a parent’s attorney needed to speak with his or her client. The recommendation to address this was to require that parents and their attorneys arrive 30 minutes before the scheduled time for mediation.

After speaking with the professionals and mediators in focus groups, it became clear that everyone could benefit from opportunities to learn from each other as well as others, and that many wanted this to happen. The focus groups became an opportunity for the participants to find out how their peers were approaching issues in mediation, and to find out what was possible. The focus group participants talked about wanting to gain more information or to explain to others what their own role is. Mediators mentioned areas of uncertainty for them. Professionals discussed areas in which they felt mediators could improve. All of this points to a need for an ongoing education program for both professionals and mediators, which was recommended.

The mediation program was first put in place in 1998 as a pilot. It has evolved over time, but hadn’t been comprehensively evaluated in more than a decade, and those evaluations were outcome-oriented, meaning that the process hadn’t been examined in a methodical way. This evaluation demonstrated that looking at both outcomes and process were essential to assessing the program and determining what could make it better.

For a quick take on the evaluation, see the Executive Summary.

For a full discussion and all statistics, see the Full Evaluation.

Evaluation of ADR in Michigan

Jennifer Shack, July 2nd, 2018

In 2011, an evaluation of Michigan’s court-connected case evaluation and mediation programs found that both case evaluation and mediation increased the probability of settlement, but that case evaluation significantly increased time to disposition. A newly published follow-up study, The Use of Case Evaluation and Mediation to Resolve Civil Cases in Michigan Circuit Courts: Follow-up Study Final Report (Courtland Consulting, May 2018), came to the same conclusion.

Case evaluation, in which a panel of expert neutrals makes a recommendation as to what the case should settle for, is mandatory for tort and medical malpractice cases. Mediation in most jurisdictions is voluntary, and can be used in conjunction with case evaluation. The study looked at a random sample of 358 cases (221 torts cases, 137 other civil cases) from three jurisdictions to determine what ADR process was used, the means by which the cases were resolved, and the time to disposition for each case.

The study found that for tort cases, there was no statistically significant difference in the form of disposition among the different options: no ADR, case evaluation only, mediation only, or both case evaluation and mediation, with a range of 71% (no ADR) to 92% (mediation) ending in a settlement or consent judgment. For other civil cases, both case evaluation and mediation (and both together) had higher rates of settlement than those cases that did not use ADR (47% for no ADR, 79% for case evaluation and 80% for mediation). The difference appears to be in the higher rate of dismissal/default judgment for cases in which no ADR process was used (49% v 21% for case evaluation and 13% for mediation). For both torts cases and other civil cases, time to disposition was considerably longer when case evaluation was used than when either mediation or no ADR was used.

When compared to mediation, case evaluation started later in the case and averaged longer to disposition from the point at which the ADR process ended. The delay could be attributed to case evaluation being rescheduled more often, although it wasn’t clear whether it was rescheduled without having been held or whether more than one session was needed.

While the findings regarding case resolution and time to disposition were similar to the 2011 findings, the lawyers and judges who responded to a survey about their perspectives on case evaluation indicated they were less satisfied with this process. Judges in particular were less confident in the effectiveness of case evaluation, with the percentage of judges who believed it was effective dropping from 69% to 53%. Attorneys had a much smaller dip, from 49% to 43%. Similar drops were seen in the percentages who would use case evaluation if it wasn’t mandatory. The percentage of judges who said they would use it dropped significantly, from 83% to 66%, while the already small percentage of attorneys who would in 2011 (36%) dropped to 29%.  The attorneys’ opinion of case evaluation was reflected in their comments about the panels. They complained that the panels lacked experience, were unprepared, were biased and did not address the merits of the case.

On the other hand, the judges’ and attorneys’ already high opinion of mediation remained steady. In 2011, 89% of the judges said mediation was an effective way to resolve disputes, compared to 93% in 2018. Attorneys were also much more likely to say mediation was effective than to say that case evaluation was, with 77% and 78% saying so in 2011 and 2018, respectively. While they had a high opinion of mediation, only 53% of attorneys said the mediators were highly skilled.

The comparison results were limited by a couple of factors. The cases that did not go through ADR processes were not similar to those that did. They were commercial cases, which are less complex, involve lower value claims, and require less discovery than other civil cases. Further, mediation was voluntary in most cases. This means the sample of mediation could be skewed by self-selection, in that the parties who decide to mediate could have been more motivated to settle and/or to settle early.