Appendix A: Methodology

Online survey

To meet the survey's objectives and funding available, using the Internet to survey an online audience was selected.

Questionnaire

The questionnaire was prepared in-house by the State Services Commission, drawing upon both internal subject-matter expertise and international examples. Demographic questions were based on Statistics New Zealand's Census questions.

Testing

After internal review and fine-tuning, the questionnaire was tested in two stages.

In the first stage an independent expert conducted two rounds of cognitive tests with 12 internal and external volunteers. The main focus was to confirm that individual questions were appropriate, intelligible, precise, unbiased, and were interpreted in the intended manner. Timing tests were also conducted to ensure that it would take no longer than 10 minutes on average to complete the questionnaire online.

The second stage of testing involved reviewing the first 20 live responses to confirm that respondents were selecting a variety of responses (within the bounds of expected behaviour), response options were not being left blank, and that there were no negative comments about the survey and/or questions in the open comments fields.

Administering the questionnaire

The questionnaire was administered by Buzz Channel Limited  from 25 November to 2 December 2008.

Their brief was to obtain about 1,000 responses of which about 100 were from people who had not used government's online services in the past 12 months. Respondents were to be notified that the questions were being asked on behalf of a government agency. Questions were in English only.

Buzz Channel set up and hosted the questionnaire. They emailed invitations to participate in the online survey to randomly selected members of their panel. Towards the end of the survey period, invitation emails were targeted based on demographics so that the overall panel sample broadly aligned with the New Zealand population demographics.

Only people who were on Buzz Channel's panel  were invited to respond. Buzz Channel recruits their panel both online and offline and currently has about 28,000 people on the panel. It is reported that about half the people choose to join the panel to have their say on things that matter to them while the other half join for the rewards (for themselves or as donations). The maximum any single respondent earns is $1 of reward for a single survey of up to 10 minutes.

According to Buzz Channel, the response rate was about 18%.

Cleansing the raw data

Cleansing of the raw data and analysis has been done in-house by the State Services Commission.

Of the total 1,320 responses, 307 responses were not usable and deleted. Most of these deleted responses had no data in them, which was due to filtering during data collection (respondents who did not fit the demographic profile, especially toward the end of the survey, were declined).

This survey is therefore based on a total valid sample of 1,013 responses.

Demographic analysis

Respondents were asked seven categories of questions related to demographics:

• Age group
• Gender
• Ethnic group(s)
• City
• Region
• Total yearly income
• Internet connection

Weighting

The raw data set has not been weighted.  Each individual respondent has been treated as equally weighted and assumed to come from a simple random sample.

Quoted comments

Spelling mistakes have been corrected.

[ Next ]