by:
John Voelker
Assistant to the Chief Justice
Wisconsin Supreme Court
and
Herbert M. Kritzer
Professor of Political Science and Law
University of Wisconsin-Madison
|
SJI |
This technical assistance project was developed under a grant from the State Justice Institute. The points of view expressed are those of the authors and do not necessarily represent the official position or policies of the State Justice Institute |
In July 1992, the Wisconsin court system published a statewide strategic plan. Unfortunately, strategic plans have a tendency to be placed "on the shelf" unless an organization makes the transition from strategic planning to strategic management. In Wisconsin, the continuing operation of the Planning and Policy Advisory Committee (PPAC) provides a vehicle for on-going strategic management.
One element that is lacking in the current strategic management structure is the on-going consideration of court system users' opinions. The objective of this project was to focus on gathering consumer information that is valuable to the courts in regards to the administration of the court system. This project was not focused on determining the perception of the users for public relation purposes but rather to gain information on issues that the court system can address.
The survey instruments in this project were intended to provide input on the need for systematic changes, ideas for court improvement, identification of court process weaknesses, and baseline information on consumer perceptions of court performance. This baseline information will be used by PPAC as a performance measure as consumer research is institutionalized into the strategic management structure. This information will supplement input received from justice system partners that PPAC uses to review, update, and prioritize issues to be addressed by the court system in Wisconsin.
There are a variety of possible approaches to collecting information on court users' perceptions of the courts with which they have recently had contact. This project adopted two primary methodologies: a mail survey of persons identified in electronic court records as case participants and an exit survey of persons at the courthouse. These two primary surveys of court users were supplemented with data drawn from a state-wide public opinion survey (which allowed a comparison of court users to the general population on several key questions).
The first step in selecting samples for the surveys was to select a subset of Wisconsin's 72 counties. With one major exception, the project design included a subset of counties that broadly reflects the state as a whole; the major exception was Milwaukee County which was omitted from the sample because the information system in Milwaukee County could not produce the required sample of cases. To capture the variation among Wisconsin counties, the counties were stratified by population and counties were selected within each strata. The specific counties chosen were based on geographic distribution, number of judges, representation of administrative districts, and the level of interest at the local jurisdiction. The result of the selection process was a sample drawn from the following counties:
Dane
Jackson
Green Lake
Chippewa
Waupaca
Kenosha
Sawyer
Eau Claire
Vilas
Manitowoc
From each of these counties, the Circuit Court Automation Program (CCAP) selected a specified number of respondents involved in family, civil, traffic, and small claims cases. Criminal cases were excluded due to the potential difficulty of identifying which institution a defendant may currently be assigned to reside. The overall sample consisted of 1,597 cases (552 family, 445 civil, 300 traffic, 300 small claims).
The survey was executed by the University of Wisconsin Survey Center (UWSC), which is a professionally-managed survey facility that has done a number of surveys related to the law and courts. UWSC was responsible for mailing questionnaires, tracking responses, and entering data. Each respondent was assigned a number; this number was used to track the response rate and to provide information on the county of origin and case type. The UWSC mailed the questionnaire which was accompanied by a cover letter signed by J. Denis Moran, the Director of State Courts; sent a post card reminder 14 days after the initial mailing; and, distributed a second questionnaire and cover letter to those who had not responded 14 days after the post card reminder.
A total of 522 usable questionnaires were returned; another 225 questionnaires were returned with a notation that the respondent had never actually been to court, 13 respondents communicated their refusal to participate, 10 communicated that they felt unable to respond meaningfully, and 13 questionnaires were returned indicating that the respondent was deceased, under 18 years of age, or non-English speaking. The balance did not respond at all (615), had bad addresses (197), or were out of the country. The response rate, excluding bad addresses, deceased, under age, and not having been to court, from the base was 44.4%. This is a reasonable, albeit not spectacular, result for a mail survey. One problem with the final sample was that it included very few members of minority groups; this was in significant part due to the absence of Milwaukee County from our sample, but even taking that into account, we believe that minorities are underrepresented in the sample. As a result, an analysis of the differences in responses based on race and/or ethnicity was not conducted.
The goal of the exit survey was to include all of the counties involved in the mail survey. We were only able to include seven of the counties (Dane, Jackson, Sawyer, Chippewa, Eau Claire, Kenosha, and Vilas); due to the timing of the survey and personnel changes in the local jurisdiction. The distribution of the surveys was the responsibility of local court personnel. These personnel were instructed to randomly distribute the surveys to individuals exiting a courtroom or court commissioner hearing room. Accompanying each questionnaire was a business-reply envelope. Respondents were asked to take a few minutes to complete the questionnaire and return it in the envelope provided.(1) A total of 1800 questionnaires were provided to the cooperating officials in the counties; the number of questionnaires distributed is unknown. However, a total of 235 usable questionnaires were received. The pool of respondents included the following composition:
61 defendants
48 plaintiffs
7 witnesses
20 jurors
11 law enforcement officers
56 lawyers
3 observers
8 "others"
2 unknown
Copies of the mail and exit survey instruments are attached as Appendix I. The mail survey was developed first. The development process involved reviewing goals of the American Bar Association's "Just Solutions" initiative and reviewing court user survey instruments from other jurisdictions. Based on this information, a survey instrument was drafted and reviewed by Professors Herbert Kritzer and Marc Galanter, court staff, PPAC members, judges, and staff from the survey contractor. Through a series of iterations, a final set of 35 evaluative items were identified tapping the following general dimensions:
Case ProcessingIn developing these items, potential response set problems were addressed by including a mixture of items worded positively and negatively toward the courts and the respondent's experience in court. One primary objective in the survey development was to design an instrument that would fit on a single ledger-sized sheet which could be folded into a four page document. In addition to the evaluative items, several demographic questions (age, gender, education, race/ethnicity, and income), were included as well as space for comments.
Safety and Convenience
Court Decisions
Public Perception of Proceedings
The mail survey instrument was the starting point for the exit survey instrument. A subset of questions from the mail survey was selected. The objective was to fit the questionnaire onto a single 8½ x 11 sheet (two-sided). The instrument required two additional questions to identify the type of case the respondent was involved in and the way in which they were involved. This allowed for an instrument with 15 evaluative items. These 15 items were identified from the most informative questions from the mail questionnaire. Unfortunately, some items were omitted that could have been beneficial in the final analysis.
An important but difficult issue that arose during the development of the survey instrument included the degree to which responses reflected experiences as court users versus general views of the court. Three general items were included on the mail questionnaire (only the first of these was included on the exit survey):
In order to assess the degree to which experience in court serves to shape these issues, statewide survey information was needed, preferably information on the county in which each respondent lives (to allow us to be sure that any differences between the general population's responses and the court users' responses). The vehicle used for this data collection was an omnibus survey called Wisconsin Opinions, which is carried out by another professional survey facility at the University of Wisconsin, the Wisconsin Survey Research Laboratory (WSRL). WSRL conducts Wisconsin Opinions on a quarterly basis, using a sample size of 400, and sells time on that survey to interested parties. Enough time for a total of six questions was purchased. In addition to the three questions above, we included the following three items:
The best way to understand the survey results is to examine the figures that graphically display those results.
Figure 1 shows the general pattern for all evaluative items on the mail questionnaire. The figures are prepared so that the left side of each figure represents responses that are positive toward the courts (shown in yellow and green in the color charts) and the right side represents negative responses (shown in dark and light blue). While the responses are shown as strongly agree, agree, etc., coding of the negatively worded responses has been reversed for consistent presentation.
The items have been grouped by the general type of issue. The letters in front of each item referred to the item number on the survey instrument (see Appendix). Also, the items shown in all capital letters involved general evaluations of courts and court processes rather than asking the respondent to evaluate the specific court experience.
Overall, the most striking aspect of the patterns shown in Figure 1 is the relative absence of negative evaluations. With some specific exceptions, the positive evaluations greatly overwhelm the negative evaluations. Only two items show a relatively strong negative, "gg. Cost too much" and "ii. Know how to complain"; for both of these, a majority of responses were negative. In fact, for only 9 of the 35 items were as many as 25% of the responses in the negative direction. In contrast, there were a majority of positive responses for 25 of the 35 items. (For eight items, there was neither a positive or negative majority).
For most of the items, a clear minority of respondents reported negative evaluations. It is encouraging, from the court administration perspective, to receive as few negative evaluations as possible. What is difficult to answer is, what is practical to achieve in terms of the satisfaction among court users? To use the old cliché, is the glass half full or half empty?
Several items are available for comparison to examine what might correlate with the court users' evaluation of their experiences. Two variables stood out as being related, the type of case (small claims, civil, family, and traffic) and gender. This review is discussed and explored in this section.
TYPE OF CASE
Many of the items showed statistically significant variation dependent upon the type of case that was sampled. Figures 2, 3, and 4 show most of these items (Figure 4 also shows a couple of items that were not statistically significant even though one might have expected significant variations). Unlike Figure 1, these figures are not reverse coded. Even though the statistical tests mean that differences shown cannot be explained by sampling, the differences do not tend to be large. The general pattern of positive rather than negative evaluations is evident in these figures, as it was in Figure 1. There is one interesting pattern worth noting and that is the traffic cases were more likely to produce either more negative responses or fewer positive responses.
Figure 5 shows one other set of items that demonstrated significant variations by case type; the items that asked respondents whether they felt there had been any differential treatment attributable to demographic factors (race, age, income, gender, or nationality). All of these did show significant differences by case type, but as with the other variables, the differences did not tend to be particularly large. The issue of perceived bias is discussed below.
GENDER
Perhaps the most striking pattern we observed in our data, besides the overall positive tone of the responses, is a relationship between gender and response. Women tend to have more positive evaluations than do men. For most items this difference is not large, and for many it is not statistically significant. It is the consistency of the pattern that is striking. Figure 6 summarizes the pattern found by showing the percentage responses for each item (item numbers shown under each bar) by gender. Men are shown in blue (dark) and women in pink (light). For only one item ("i. I felt safe in the courthouse") are men more likely to have a positive response than are women. While that pattern is extremely consistent, and a number of the differences are statistically significant, for none of the items do men and women have sharply contrasting evaluations. Nonetheless, the consistency of the pattern is striking.
For obvious reasons, whether or not there were any strong patterns of perceptions of bias based on demographic factors was of particular interest. As noted previously, very few minorities in the sample limits our ability to assess certain sources of perceived bias. Figure 7 is an enlarged version of part of Figure 1; it shows the response patterns for the five items which measured the respondents of the perception of whether their treatment in court was influenced by their race, gender, nationality, age, or income. As the figure shows, just under 20% of the respondents perceived some effect of income, 12-13% perceived some influence of gender, and just under 10% perceived some age affect (race and nationality produced percentages under 5%).
The next question is whether these perceptions correlate at all with the relevant demographic variables. Figure 8 shows patterns for gender, age, and income. Race was not examined because of the extremely small number of nonwhites in the sample. There was no relationship between gender and the perception of gender influence. However, there were relationships between income and perception of income influence, and age and perception of age influence. The stronger relationship is with age. Respondents in the youngest age category are more likely to see an influence of age than are older respondents. Actually, the pattern is strongest if we viewed in terms of disagreeing that age was an influence: about a third of the youngest set of respondents disagreed, compared to about two-thirds of the other respondents. With regard to income, there is something akin to a monotonic pattern showing more or an inclination to perceive income as an influence as income declines, with the lowest income group standing out as sharply more likely to see income as an influence.
General Patterns
Figure 9 shows the exit survey results in comparison to the corresponding items from the mail survey and from the Wisconsin Opinions survey.
There is a clear, and very striking pattern of difference between the mail and exit survey responses: the respondents to the exit survey are consistently more positive about the courts than are the respondents to the mail survey; in fact, there is not a single item for which the mail sample was more positive toward the court than was the exit sample. Some of the more striking examples of differences include:
One possible explanation for the apparent differences between the two surveys is that there were different types of respondents in the respective samples. Indeed, there are different types of respondents: the mail survey consists solely of plaintiffs and defendants. While the exit survey includes witnesses, jurors, law enforcement officers, lawyers, and other regulars around the courthouse (probation officers, social workers, etc.). Perhaps the difference reflect that the courthouse regulars are more positive than those who come in for a particular case? This was tested by comparing the means of the "regulars" (lawyers, law enforcement officers, and others) and the "transients" (plaintiffs, defendants, witnesses, and jurors). For most of the items, the means for the two groups were very close; for only two items was the difference in the two means large enough to conclude that it was unlikely to occur solely by chance. However, for one of these items, the regulars were less positive than were the regulars (the item that reads, "I felt safe in the courthouse"). Thus only for one item ("If I had a complaint about something that happened in court, I would not have any idea what I would do about it") were the regulars significantly more positive than were the transients.(2)
It is worth noting also that intuition suggests that those who are most likely to return a "customer satisfaction" questionnaire are persons who have something to complain about.(3) Thus, if one were to predict that it would be the mail survey.
How then do we explain the consistent difference between the mail and the exit surveys? This discussion is deferred until we consider the population survey because that helps in producing a possible explanation.
As with the mail survey, we examined the relationship between the evaluative items and role, case, and demographic factors captured on the survey instrument. While there were a number of statistically significant results, we identified only one pattern of difference that merits discussion: the relationship between perceived treatment and age. Younger respondents were less likely to see the court personnel as friendly, available to answer questions, or that the judge explained the process. This general pattern is consistent with some of the patterns in the mail survey. There were a couple of other specific differences that are worth noting:
There are only three items in common between the mail survey and the Wisconsin Opinions population survey, and only one item in common across the three surveys. The one item available across all three surveys is "Court decisions take too long." Interestingly, over 80% of the Wisconsin Opinions respondents agreed or strongly agreed with this statement, compared to about 26% of the mail survey respondents, and only 13% of the exit survey respondents. On the item, "I think it costs too much to go to court," over 80% of the Wisconsin Opinions respondents agreed or strongly agreed compared to about 65% of the survey respondents. Only on the item, "The use of formal court proceedings is the best way to resolve disputes," could one argue that the Wisconsin Opinions respondents more positive than the mail survey respondents, and here the gap was relatively small (just under 50% of the Wisconsin Opinion respondents agreed or strongly agreed compared to just over 40% of the mail survey respondents). However, the Wisconsin Opinion respondents were also more likely to disagree or strongly disagree with this statement (38%) than were the mail survey respondents (31%).
One possible explanation for the differences between the Wisconsin Opinions results and the two court user surveys is that Wisconsin Opinions covers the entire state (including Milwaukee), and views of the courts are more negative in the counties not in the court users surveys. This explanation was tested by splitting the respondents in the Wisconsin Opinions survey into two groups, those in one of the ten counties in the court users surveys and those in one of the other 62 counties. For one of the items ("I think it costs too much to go to court") there was a statistically significant difference in the responses of these two groups. However, those in the counties included in the court users surveys are more likely to agree or strongly agree with this statement (92%) than are those in the other counties (79%), which is opposite what one would find if differences between the counties accounted for the differences between the Wisconsin Opinions survey and the court users surveys.
Adding the Wisconsin Opinions results to that for the mail versus exit surveys suggests the following hypothesis: the general image of the courts is inconsistent with the experiences of those who go to court. Going to court actually tends to result in a short-term improvement in people's views of the courts. However, the "good feelings" produced by actual court contact fade relatively rapidly. If one gets responses immediately after the court experience, they are most likely to be positive. If one waits 2 to 6 months, the decay in good feelings is apparent. Interestingly, we included a question in the Wisconsin Opinions survey which asked if the respondent had been to court in the last 12 months; comparing those respondents who responded yes to this to those who responded no, there are no significant differences.(4)
One other possible explanation for the difference between the Wisconsin Opinions responses and the users surveys is that by asking about specific aspects of a particular court experience, the respondents frame their response about courts more generally in the context of that set of responses. The result would be that positive specific responses would increase the positive general responses in the court user survey: the absence of any specific questions in the Wisconsin Opinions survey would then result in less positive responses. However, it is doubtful that this effect, if it does exist, accounts for all of the differences between the Wisconsin Opinions patterns and the court users patterns.
The Wisconsin court system's strategic plan entitled Framework for Action includes a discussion concerning on-going strategic planning and management. This section states that:
"Successful implementation of this plan requires courts to move from strategic planning to strategic management. In particular, the Wisconsin court system needs to institutionalize an approach to management which emphasizes continuously evaluating court system performance in light of longer-term goals and objectives."The report continues by noting that PPAC should "examine information about court users' perceptions of aspects of court performance and determine how to reflect court user views in the goals, objectives, and tasks included in the Framework for Action.
The data collected during this court user opinion project provides PPAC with the information necessary to review perceptions of the court system. Prior to this study, this type of information was not available. The next step is for PPAC to determine how to best reflect these user views in the development of goals, objectives, and system priorities.
An initial approach to this review is to compare the elements of the court's vision statement to the results of related questions included in the survey instruments. This analysis provides an overall perspective on how well the court system is meeting the ideal characteristics identified in the vision statement. The Wisconsin court system's vision statement emphasizes the need to be fair, understandable, consistent, flexible, compassionate, affordable, user-friendly, truth seeking, accessible, efficient, and final. Specifically:
In the future the Wisconsin court system will be understandable, readily accessible, and flexible. It will provide prompt, user-supportive, and appropriate forums for the types of disputes that need to be resolved. It will operate efficiently and effectively. The core values of integrity, compassion, fairness, and consistency will be reflected throughout the Wisconsin court system.
Table 1 provides a snapshot of how court users perceive the court system's performance in the core characteristics identified in the vision statement. The table shows the percentage of respondents in the mail survey that provided positive responses to questions related to the vision characteristics.
Table 1
Court Performance on Vision Characteristics
| Characteristic | Question Reference | Positive Response Percentage |
| Fair | e. Race Influenced k. Not Fair q. Gender Influenced v. Age Influenced z. Nationality Influenced aa. Fairness ee. Income Influenced |
81.1 52.4 64.9 65.6 78.0 61.1 62.4 |
| Understandable | j. Decision, No Sense y. Court Didn't Care bb. Didn't Understand dd. Explaining ii. How to Complain |
53.1 59.6 68.3 48.6 30.9 |
| Consistent | h. Followed Law | 68.1 |
| Compassionate | d. Not Interested t. Respect w. Address Form x. Treatment | 49.6 65.1 52.4 58.1 |
| Affordable | gg. Costs too Much | 11.8 |
| User-friendly | f. Friendly/Courteous n. Too Many Forms o. Forms Understandable r. Answer Questions s. Explained Things cc. Assembly Line |
67.2 54.2 43.0 50.0 47.7 50.9 |
| Truth seeking | g. Had Information | 58.0 |
| Accessible | c. Trouble Finding i. Felt Safe l. Hardship to Attend m. Opportunity to Explain |
66.6 73.0 47.6 58.0 |
| Efficient | a. Wait too Long b. Reasonable Time hh. Take too Long |
63.8 68.7 28.3 |
| Effective | p. Effectiveness u. Brought Out Issues ff. Best Way to Resolve |
55.1 41.2 40.7 |
As this table shows, the Wisconsin court system is moving in the right direction. This is evident by the fact that only 8 of 35 questions received less than 50 percent favorable responses. Specifically, court user opinions indicate that specific strengths exist in the areas of fairness, accessibility, and consistency. As PPAC reviews this information closer, strategies should be developed that build upon these strengths.
The opinions of court users also indicate areas of potential weaknesses. Specifically, only 11.8 percent of respondents thought it did not cost too much to go to court. This combined with the fact that 28.3 percent of the respondents thought it did not take to long for court decisions raises issues that should be reviewed by PPAC.
1. We considered the options of trying to structure a situation where the respondents would complete the questionnaire prior to leaving the courthouse (e.g., setting up a room with cookies and beverages where respondents could relax while completing the questionnaire), but this idea was abandoned both for logistical reasons and because we believed that many, if not most, potential respondents would be anxious to leave the courthouse and get on with their routine business.
2. We should note that for the other 13 items, the statistical test (which was a two-sample t-test) did not even approach statistical significance.
3. We sought to determine if there was any published research on who returns customer satisfaction questionnaires. We unsuccessfully searched indexes of business and social science research articles. We talked to several specialists in marketing research who said that they shared out intuition, but could not identify any specific research. One person we spoke to pointed us to research showing (1) those with extreme reactions one way or the other are more likely to respond, and (2) this is more so for those with negative reactions: see John J. Skowronski and Donal E. Carlson, "Negativity and Extremity Biases in Impression Formation: A Review of Explanations", " Psychological Bulletin" 105 (1989): 131-142.
4. It is unlikely that some of those who responded yes to the question about going to court actually had that experience more than 12 months ago; there is a well known phenomenon of respondents underestimating how long ago an event took place.