1. INTRODUCTION

An integral part of the Law Reform Commission’s research and review of legislation is consultation with interest groups and the community at large. This is often what sets the Commission apart from other agencies that undertake reviews of specific laws. Consultation strategies are developed according to the type of project involved, the timeframe for the project and the stakeholders with whom the Commission needs to consult. As the strength and validity of the Commission’s final recommendations depend on thorough research and consultation, it is appropriate that the Commission evaluates its consultation techniques on a regular basis.

Because of the nature of the project, the diverse interest groups which were consulted and the comparatively short timeframe for completion, the Commission felt it was appropriate to undertake an evaluation of its consultation strategy in the review of the Disability Services Act 1993 (NSW) (“DSA”) and the Community Services (Complaints, Appeals and Monitoring) Act 1993 (NSW) (“CAMA”).

2. METHODOLOGY

Design and development of the questionnaire

The main aim of the project is to assess the effectiveness of the research and consultation methods used by the Commission in the review of the DSA and CAMA. The Commission sought the views of those who participated in the consultation process by means of a written questionnaire.

In developing and designing the questionnaire, the Commission was mindful of the need to ensure that the survey was short and simple. It needed to be easy to read and understand and not take too long to complete. To assist in the design of the questionnaire, the Commission consulted with people with expertise in statistical design and people with experience and knowledge of how best to consult with people with disabilities. The Commission obtained valuable input from: Joanne Baker, a statistician with the Bureau of Crime Statistics and Research; Juliet Haraksin, the Disability Co-ordinator of the Attorney General’s Department; and Juliet London and John Moxon, both of whom are consultants who were engaged by the Commission to conduct a series of focus groups. The Commission staff also contributed useful feedback. A copy of the survey can be found at Appendix A.

Developing a questionnaire in an appropriate format achieved the desired outcome, namely, to ensure the Commission also reached consumers of disability services. In the end, people with disabilities constituted almost 20% of the total pool of respondents. Although one respondent indicated that they found the survey difficult to complete because of their disability, others said that the opportunity to complete the survey over the telephone was very useful.

Subject areas

The subject areas that the survey inquired about were:

• how the project was publicised (questions 1 and 2);
• the ease with which people could obtain information from the Commission (questions 3 and 4);
• who received a copy of the Issues Papers, in what format they received them in, and whether or not people were satisfied with them (questions 5 to 9);
• how people participated in the research and consultation process (question 10);
• the effectiveness of the seminars, focus groups, and the submission process (questions 10 to 15);
• views on the research and consultation process as a whole (questions 17 and 18); and
• demographic details of respondents, including whether or not they had access to email.

The survey questionnaire was mailed out on 14 October 1999 to 190 persons with a covering letter explaining the purpose of the survey and a reply paid envelope. Free copies of the Commission’s reports on its review of the DSA and CAMA were also sent. Respondents were instructed to return the questionnaire before 8 November 1999, which gave them what was considered to be a sufficient amount of time to complete the survey and return it to the Commission. The same survey was distributed to everyone. However, for identification purposes they were colour coded. The colours and categories are outlined below:

Table 1: Distribution of survey questionnaires

There was some overlap between those who sent submissions and those who attended one of the seminars. Where this occurred, white copies of the survey questionnaires were distributed. Duplication was avoided wherever possible. However, in the case of some larger organisations (some with regional offices), the Commission sent multiple questionnaires directed to different persons.

In relation to those persons who participated in focus groups, the Commission arranged to send questionnaires to them via the consultants (or agencies used by them) who were engaged to conduct the focus groups. This was done for privacy reasons. The response rate, not unexpectedly, was poor (at 9%) compared to the other categories to whom the questionnaire was sent. The low response rate can be explained by two factors. First, as the Commission did not directly mail out the questionnaires to focus group participants, no follow-up measures were taken with this group. But more significantly, persons who attended the focus groups tended to be persons who were unlikely to make written responses. All had one or more disabilities. The majority had an intellectual disability, and some were Aboriginal or Torres Strait Islander people or people from non-English speaking background. Indeed, it was precisely because they were unlikely to make a written submission to the Commission that consultants were engaged to conduct focus groups with them.

Follow-up measures

Table 2 highlights the value of using follow-up measures to ensure a satisfactory response rate. Close to half of all the responses would probably not have been received or recorded had follow-up measures not been taken. The measures used included sending a follow-up letter to all respondents (except those in focus groups) with a new copy of the survey and a reply paid envelope. In addition, several respondents were telephoned and given the opportunity to complete the survey over the telephone. This was very useful but time-consuming.

Table 2: Follow up measures

3. RESPONSES

Overall response rate

The overall response rate (excluding focus group participants) stands at 69%. If the focus groups participants are included, the overall response rate is 55%. In the light of this response rate, which was much better than initially forecast, the Commission is able to analyse and interpret the results with reasonable confidence.

Response rate by category

104 questionnaires were returned by 10 January 2000 from a mail out of 190.

Of these:

• 56 were returned from those who made submissions (white);
• 11 were returned from those who attended the seminar for advocates and carers (yellow);
• 14 from those who attended the seminar for service providers (blue);
• 5 from those who attended the seminar for consumers of disability services (pale green);
• 14 from those who attended regional seminars;
• only 4 of those who attended focus groups.

4. DEMOGRAPHICAL ANALYSIS

Question 19 of the survey asked respondents to provide some demographic details about themselves including their gender, age, where they lived and whether they were Aboriginal or Torres Strait Islander or from a non-English speaking background. The results appear below.

Gender

Of the total number of respondents, 69% said they were female, 26% said they were male and 5% did not say. The gender bias is due largely to the proportion of women in the community services sector, and not to the particulars of the survey itself. Only 23% of the total pool of people who participated in the review, and who were thus sent surveys, were men.

Table 4: Gender of respondents

Age

A large majority of the respondents said that they were over 35 years of age.

Only 14% said that they were aged between 18 and 35 and none of the respondents were aged less than 18 years. This result is not surprising and in fact is the reason that the Commission engaged consultants to conduct focus groups specifically with children and young people.¹ The low response rate from focus group participants also contributes to the minority of people in the 18 to 35 age group.

Table 5: Age of respondents

Where respondents live

Although the largest number of respondents lived in Sydney, almost one-third of respondents came from areas outside of Sydney. This is a comparatively good result. The Commission was very conscious of avoiding a city-centric review and specifically arranged to conduct a series of seminars in major regional centres. These seminars were advertised in local papers and were well-attended. The short timeframe for the review precluded the Commission from visiting more regional centres.

Focus groups were also conducted in regional areas and in non-metropolitan Sydney. Of the 32 respondents who said that they lived outside of Sydney, 14 said they had attended a regional seminar, two attended the seminar for advocates and carers in Sydney, one said that they had attended a focus group and 14 made submissions to the Commission. Most of the people who said that they did not live in Sydney were aged over 35. Only three said they were aged between 18 and 35.

Table 6: Where respondents live

Non-English speaking backgrounds

Only 5 respondents (or 5%) indicated that they came from a non-English-speaking-background. All were women who said that they lived in Sydney. Two were aged between 18 and 35 and the other three said they were over 35 years of age.

Three of the five said that they had a disability (two had a sensory disability and one had an intellectual disability). However, only one respondent said she was a consumer of disability services. Two identified themselves as being both advocates for non-English speaking people with disabilities and service providers for adults and young persons with disabilities. One respondent identified herself as solely an advocate while another said she was solely a service provider.

In recognition of the enhanced difficulty for people with disabilities from a non-English speaking background to access the usual consultation processes, the Commission engaged consultants to conduct small focus groups specifically with people with a disability from a non-English speaking background. Tha aim of the focus groups was to ensure they had an opportunity to make comments on the legislation. Two of the five respondents who returned the survey were in fact, participants in focus groups.

Aboriginal and Torres Strait Islanders

None of the respondents identified themselves as being Aboriginal or Torres Strait Islander. Again, the Commission took proactive measures, hiring a consultant to conduct focus groups in both city and rural areas, specifically with Aboriginal and Torres Strait Islander people with disabilities, to ensure they participated in the review.

Disability

Seventeen of the respondents identified themselves as being persons with disabilities or as consumers of disability services. Of these: one said that they had an intellectual disability; 7 said that they had a physical disability; 8 said that they had a sensory disability while 6 respondents said that they had another type of disability. Of these six respondents: one said that they suffered from Multiple Sclerosis; 4 said that they had an acquired brain injury; and one said that they suffered from a chronic degenerative condition. The numbers add up to more than the total number of 17 respondents owing to the fact that several people said that they had multiple disabilities.

Table 7: Types of disabilities

5. SUMMARY OF RESULTS

How the LRC publicises its projects

Question 1 asked how people found out that the LRC was conducting a review of disability services legislation. As Table 8 below illustrates, by far the largest number of respondents indicated that they had found out about the review either through a newsletter from a government department or other agency (41%). This was true for all the categories.

A number of respondents said they found out about the review by seeing copies of the Commission’s Issues Papers on the DSA and CAMA (26%) or through direct contact with the Commission (22%). While this was the case for those who made submissions, and those who attended regional seminars or the seminar for service providers, very few advocates and consumers said they found out about the review in this way. Most advocates and parents/carers found out through a colleague or friend (18% overrall found out about the review from a colleague or friend) or through a newsletter, as did consumers.

A total of 15% across the board said that they read about the review through a newspaper. Others heard about the review through a community visitor (7%) or through a conference they had attended (3%) or through the radio (1%).

22 respondents said that they heard about the review through other means. Of these, a large number (12) said that they heard about the review from a community disability group such as the Western Sydney Intellectual Disability Support Group, The Spastic Centre, the Illawarra Disability Trust and the Brain Injury Association. Others found out through peak bodies such as ACROD, NCOSS or from a government agency including ADD and the Community Services Commission or through the Disability Council. One said that they were directly affected by the review. Another person said that they found out through another parent and some said that they heard about the review by virtue of the fact that they were part of the advocacy sector.

Table 8: How respondents heard about the review (total and by category)

How should the Commission publicise its projects in the future?

Question 2 allowed people an opportunity to suggest any other ways the Law Reform Commission could have publicised a similar project. While many of the suggestions are valuable, most of them are specific to this reference. With some modification, however, they may also be applied to other projects at the Commission.

The majority of people (53) did not have any further suggestions and 13 respondents did not reply to this question. Some suggestions had actually been carried out by the Commission, for example, contacting key bodies and non-government organisations directly or via their newsletters, publicising the reviews on ethnic radio stations and on the Commission’s website. The fact that these suggestions were made indicates that such techniques should be continued and their scope increased. It was also suggested that the Commission publicise similar projects in disability specific magazines such as “Quad Wrangle”, on ethnic television stations and on notice boards in human services departments, neighbourhood centres and libraries.

The Commission found it quite effective to obtain mailing lists from various sources including the Ageing and Disability Department, the Disability Council, the Disability Safeguards Coalition and ACROD.

A further suggestion was that the Commission publicise similar projects on other websites. The Commission has a link to current references on its website and would look favourably on requests by organisations to have a link to the Commission’s website on their own site. Publication in major newspapers such as the Sydney Morning Herald and in local or regional papers outside of Sydney was also suggested. The Commission did place advertisements for regional seminars in the local papers and also advertised the review in a small number of ethnic newspapers.

How easy was it to obtain information from the Commission?

The majority (55%) believed that it was easy to obtain information from the Commission, compared to 8% who responded in the negative to this question. A large number of people did not require any additional information (33%) and 4% did not respond to this question. A remarkable 88% of respondents reported being satisfied with the information they received about the project and how they could participate. It is evident that the retrieval of information from the Commission itself is perceived to be easy and that the nature of the information received is such that it was not necessary to request any further information.

Issues Papers

A large majority (100 out of the total of 104 respondents or 96% of the total number of respondents) indicated that they had received a copy of the Issues Papers. Of these 100, the majority (92%) received the Issues Papers in the standard format together with a four page summary.

The Issues Papers were available in other formats such as large-print and spiral bound, diskette, on the Commission’s website. A large-print summary of both and a summary on audio tape were also available. Despite this, some respondents indicated that it would also have been helpful if the Issues Papers were written in plain English. Others suggested that the Issues Papers be translated into community languages and be passed onto people from non-English speaking backgrounds. It was also suggested that the Issues Papers could be presented in a visual format by pictures, video and disc. In order to cater for blind people, it was specifically suggested that the Issues Papers be printed in braille on request. The Commission notes that the summary of the Issues Papers was available on audio and several of them were distributed. A diskette version was also available for people who had access to talking computers.

As Table 9 below shows, most people were satisfied with the quality and accessibility of the Issues Papers. A very large majority of those who received the Issues Papers reported that they were easy to read (88 respondents) and informative (91 respondents). Very few people disagreed with these propositions. However, one-quarter of respondents indicated that the Issues Papers were too long.

Table 9: Quality and accessibility of Issues Papers

Also, in response to question 9 of the survey, 78% of respondents reported that the Issues Papers covered the main issues of concern to them. A number of other respondents, however, said that some other issues should also have been addressed or explored in greater detail. These issues included the realities of funding, community housing issues, administrative matters (which were not covered in the legislation itself), community attitudes to people with disabilities, other options for redress when breaches of the Act occur, how to resolve issues between family members and service providers and more discussion on carers and children in permanent care.

How people participated in the consultation process

Question 10 asked people in what way they participated in the consultation process: 56 people or 54% said that they had attended a seminar in either Ballina, Wagga Wagga, Maitland or Sydney; 2 people participated in a small focus group or in a one-on-one interview; 17 people met with or telephoned an LRC representative while

60 people made a submission to the LRC. These results are illustrated in Table 10.

People participated in other ways as well. For example, one person wrote a letter to the Commission, another person was part of the reference group set up by the Commission and yet another said she was on a working group organised by the Disability Council of NSW. Others said that they had attended a consultation forum organised by an agency other than the Commission, including the Disability Safeguards Coalition, the Disability Council of NSW and the Intellectual Disability Rights Service, at which the Executive Director of the Commission, Mr Peter Hennessy, was also present. Two persons said that they participated by reading the Issues Papers.

Table 10: How respondents participated in the consultation process

Seminars

Question 11 asked those respondents who attended a seminar arranged by the Commission a series of questions on how those seminars were conducted. As Table 11 below shows, the majority of respondents who answered the questions expressed general satisfaction with the way the seminars were conducted and the discussion promoted.

A large majority of those respondents who attended a seminar said that the venue was easy to get to (93%) and that the purpose of the seminar and the issues were explained clearly (93%). However, almost half of these respondents agreed that there had not been enough time for discussion. Further, while 45 respondents (or 80%) disagreed with the proposition that they felt too intimidated to speak up (7 of these strongly disagreeing), 8 (or 14%) agreed with the statement and 3 did not say. Similarly, 80% of respondents felt that their views were listened to but almost 15% felt otherwise. There was also some dissatisfaction with the issues covered at the seminars with one-fifth of respondents claiming that not all the major issues of concern to them were dealt with.

The Commission was interested to find out whether participants felt more inclined to make a submission after they attended a seminar. 20 respondents (or 36%) said the seminar did prompt them to make a submission. However, 28 respondents disagreed with the proposition in question 11(h), 3 people said they intended to make a submission anyway and 5 respondents gave no response.

Table 11: Evaluation of seminar by participants

How could the seminars have been run better?

Of the 56 people who said that they had attended a seminar, the majority (33 or 59%) indicated that they did not have any suggestions as to how the seminars could be organised or run better. A small number made the following suggestions. Some suggested that the seminars be organised in a greater number of outer regions of Sydney, especially the Illawarra area. It was also suggested that the venue of Parliament House was too formal with its furnishings and hushed atmosphere and that a more informal environment would be appropriate. Another venue used by the Commission was considered to be too small for wheel chairs to fit in.

It was further suggested that there should be separate seminars for people with disabilities and their carers and for service providers. It was felt that mixed audiences could hinder the focus on individual issues of concern. Separate seminars would also ensure that seminars could be smaller in size, as was also suggested. The other problem with large groups was that speakers may not be so easily heard, necessitating the use of microphones. The Commission notes that separate seminars for people with disabilities, advocates and service providers were held in Sydney to promote frank and open debate.

Another suggestion made was that facilitators should ensure that excessively vocal people did not dominate the discussion. Some people said that the seminars needed to start on time and have a longer duration.

None of the respondents reported that the overall structure was heavily flawed. Indeed, most were actually glad to have had the opportunity of contributing in the seminar.

Focus Groups

The very small response from focus group participants while not unanticipated, means that the Commission cannot confidently assess the effectiveness of the strategy to conduct focus groups nor the performance of the facilitators engaged to conduct them.

Of the four respondents, two revealed that they were unable to suggest any alternative ways in which the focus groups could have been run better. The remaining two made the following suggestions. First, that the consultant needs to follow the group’s agenda rather than her own. Second, in light of the fact that people from non-English speaking backgrounds constitute 20% of people with disabilities, there needs to be more focus groups with people from non-English-speaking-backgrounds.

As stated previously, the Commission is unable to assess the effectiveness of the focus groups given the very poor response rate.

Submissions

Almost two-thirds of the total number of respondents said that they had prepared a submission. Of the 64 people or 62% who did prepare a submission, most did so in written form. Five respondents said they made a submission in electronic format and 8 respondents said that as well as making a written submission they made an oral submission.

In question 16 of the survey, the Commission posed a series of propositions relating to the submissions process to which respondents could agree or disagree. The results of the responses are illustrated in the graph below (see Table 12). 92% of respondents who had made submissions said that they had found it easy to contact the Commission, with only 3% disagreeing. 43 respondents (or 67%) agreed that they had had enough time to prepare their submission but one-third disagreed. The majority of respondents (98%) did not have any concerns about the confidentiality of their submission. While almost half of respondents considered it useful to be able to make a submission by email, 11 disagreed and 20 respondents did not answer, presumably also because making a submission electronically was not available to them.

Table 12: Evaluation of submission process

Access to email

The Commission was interested in determining whether email access was being more widely used as it is moving towards receiving submissions in electronic format and developing an email list of subscribers. The Commission also currently places all of its publications on its website.

While 62 respondents (or 60%) said that they did have access to email, one-third of respondents did not. Despite the large number of respondents that did have access to email, only 8 respondents said that they had submitted their submission electronically. These statistics show that while increasing numbers of people have access to email, there are still significant numbers of people who do not. As a result, while the receipt and distribution of information electronically is a useful supplement to other traditional forms of dissemination, it should not be used exclusively.

Table 13: Access to email

Views on the LRC’s consultation and research methods

Question 17

Question 17 asked respondents “[i]n what ways could the consultation and research process have been improved?”. 48 people responded, making the following suggestions.

Although the Commission produced the Issues Papers in multiple formats including large-print and spiral bound, diskette and on the Commission’s website, one person suggested that the Issues Papers should have been in a format to encourage young people to participate. A few people suggested that better advertisement of the seminars was required. Some felt that there needed to be more information and plain English advice on the submission process. Others suggested that there needed to be more time for preparing submissions. Another suggestion was that where an electronic submission was made, there should have been an acknowledgment of whether or not the submission was received.

It was also suggested that people with developmental disabilities and disabled children should be included in the consultation process. A further suggestion was that carers should be allowed to attend seminars on the DSA and CAMA rather than be discouraged from attending them. Another suggestion was that consumers of services should be given a copy of questions and an agenda before attending the focus groups. It was also suggested that there should be more information on submission preparation at seminars. When seminars are conducted, the transport of disabled people needs to be considered. For example, one person commented that taxis cannot be re-booked because seminars are running overtime. One person also remarked that small service providers were overlooked in favour of larger service providers.

Question 18

60 respondents gave an answer to question 18, “[w]hat features of the consultation and research process would you like to see used again?”. There were numerous features that respondents believed should be used again. Many respondents commented that the Issues Papers, seminars and focus groups were well organised. Some commented that the meetings were well-conducted, promoted open discussion, were held in good venues and that participants were treated respectfully. Another person added that it was good that consumer consultations were kept separate from service providers. It was also said that the reference group which was set up to advise the Commission and working closely with the Disability Council was a good feature as it allowed input into the Report.

Many remarked that the focus groups were a good feature. Others noted that the wide range of opportunities for consultation and various formats of papers available were very good. The availability of telephone contact was identified as an excellent feature. The attendance of Commission staff at other (non-Commission organised) seminars was also said to be beneficial. One person commented that the use of a person with disability as a facilitator was very impressive. It was also said that personal consultations were an excellent feature since it is difficult for carers to get to meetings, focus groups etc. The meetings with community visitors were also a popular feature.

Another person commented that this follow-up survey itself was a feature they would like to see used again. However, it was suggested that such a survey be conducted closer to the date that the consultation took place. One respondent said that a similar review, using the same strategies, in 3-5 years time would be good.

Conclusion

The response rate was excellent overall and across all groups, bar the focus group. The Commission was pleased to have received feedback from respondents in regional areas and was especially pleased to find that almost 20% of respondents were people with disabilities themselves.

The survey results show general satisfaction with the consultation and research strategies used by the Commission in the review of the DSA and CAMA. The use of a Reference Group was considered a highlight of the Commission’s consultation strategies, as were the use of seminars and focus groups. The attendance of staff at seminars organised by other persons and groups was also considered to be very useful. Many respondents said that the availability of the papers in multiple formats was appropriate, and were also supportive of this follow-up survey.

A majority of respondents were pleased with the methods used to publicise the project. Advertising and spreading the news about the review through relevant government newsletters and the newsletters of relevant industry and community groups was particularly effective. One good suggestion to help raise awareness of the Commission’s review is to encourage relevant groups to instal a link on their website to the Commission’s site.

A large majority of persons were satisfied with the information they received from the Commission about the project and how people could participate. Most were pleased with the quality and accessibility of the issues papers although some felt they were too long. The submission process was also considered satisfactory bar some criticism that there had not been enough time to prepare a submission.

Of those who attended the seminars, most agreed that it was a worthwhile exercise and promoted community debate. The criticisms of the seminars centred mostly on the fact that there was not enough time for discussion and hence some of the issues of concern to participants were not dealt with. Unfortunately, conclusions cannot be drawn about the focus groups that were held due to the low response rate. Perhaps what this low response rate does highlight, however, is the need for any evaluation to be conducted at the time of the consultation. Distributing evaluation surveys like this one at the time of the seminars and focus groups will ensure that the responses are accurate. It will also mean that assistance can be given to those who have difficulty reading forms and giving written answers.

Another interesting result was the high proportion of people who indicated that they have access to email. This suggests that more and more people may make their submissions electronically. The Commission needs to look at its procedures for receiving submissions and make any necessary alterations to take this into account.

FOOTNOTE

1. See NSW Law Reform Commission, Review of the Disability Services Act 1993 (NSW) and the Community Services (Complaints, Appeals and Monitoring) Act 1993 (NSW): Consultations (Research Report 9, 1999) Ch 2 and 3.