Surveys in general1
Surveys are a good way to get a little bit of information about a lot of different topics. They’re also good if you think you already know some of the answers. But they are especially good if you need to collect information in a way that can be translated into numbers. 2 This means that they are a very useful way of gathering information for outcomes reporting.
To create a survey you need people to want to, and be able to, complete your survey for you. You also need them to be able to give you thoughtful answers to your questions. You have two choices about what kind of survey you will use to attract these people – online surveys and paper-based surveys.
This How To covers how to design an online survey and how to get the most responses possible. Some of it overlaps with How to create a paper-based survey.
There is no doubting the efficiency and usefulness of online survey software. It not only helps you construct your survey but will also collate the responses and help you analyse the results.
There are two ways you can ask people to respond to an online survey. One is to embed it in your website, having it available as a workable survey, not as a word document or pdf that has to be downloaded, and then promote the fact that it is available. The other is to send an invitation by email that includes a link to the survey.
Purpose of survey
Write down the reason you want to do a survey, that is, what it is you want to know (eg, ‘is our service meeting people’s needs?’). Now write down what you need to find out in order to answer that first, most important question. These topics should break the main question down into parts like this:
What we want to know: is our service meeting people’s needs?
Breaking it down:
- Are clients satisfied with the way we conduct assessments?
- Does the way we do case coordination, support and referral work for them?
- Do they think we are helping them achieve the outcomes they want?
- Are there services clients want that we aren’t providing?
Make sure the subsequent decisions about who you survey and how your survey is structured relate closely to these basic ideas.
Choosing who to survey
Working out who to survey for reporting purposes might be as easy as deciding you want to survey everyone who is a member of your organisation, or everyone who participated in one of your programs.
One of the biggest mistakes you can make with your survey is to exclude people from having the chance to fill it in. This could be because you distributed your survey in such a way that only a few people could complete it, or you wrote it in such a way that not many people could understand it. 3 Basically you need to give everyone who falls into the category of people that you want to survey an equal chance to fill it in. This means you have to try to predict all the things that might stop people from answering your survey and try to find ways around these obstacles.
Writing the questions
For more information on working out what questions you want to ask, how to write them and how to sort them into order, see How to write questions for a survey.
Optimise your survey for the devices it is going to be viewed on
Surveys that look good on computers can cause all kinds of problems on mobile devices. Even people with desk top computers might open your email on their phone and look at your survey there − and possibly delete it if it isn’t easy to read.
Here are a few things you can do to make sure your survey works on a mobile device:
- Don’t ask many open-ended questions. Normally a mix of closed and open-ended questions is recommended in a survey, but not if the survey is being viewed on a mobile device; it’s too hard to type the answer, plus, it’s likely a person using a mobile device is on the move somewhere.
- Don’t put links to images in your survey. It slows the download.
- Pretest your survey on a mobile device. There are lots of design differences, like different light settings on screens that make colours look different. You might need to modify your design so that it can be seen well on a mobile device.
Some online survey providers (eg SurveyMonkey) automatically adjust the format of the survey for the size of the screen of whatever device they are viewed on but it’s still a good idea to test them and see.
Keep the subject line in your email short! You want to inform readers and attract participation. iPhones cut off subject lines after 35 characters in portrait view and 80 in landscape view. Androids cut them off after 33 characters in portrait view and 72 in landscape view.
You might decide that you don’t want to alter your survey in these ways too much. For example, you might think that it is very important to include the open-ended questions. The trade-off of that decision is that you might get fewer responses from people who use mobile devices heavily. That decision might be fine if it is staff in other community organisations that you want to survey. But if, for example, you want to hear from young people it’s worth adjusting your survey for mobiles.
Check and double check
Get a variety of people to test your survey to make sure it makes sense and that it works. Online surveys have features which help people skip from question to question, missing the questions that aren’t relevant to them. These are called piping logic or skip logic. Make sure you double check that any commands like these that you have placed in your survey are working.
When you place the link to the online survey in your email invitation check and double check that it works.
Get feedback on every aspect of your survey, from the message in the email subject line to the messages on the opening and closing pages.
Get as many responses as possible to your survey
In the social sciences getting 50% of your surveys filled in and returned is the general goal; more than that and you are a survey maestro. However, you will be very lucky to get that many surveys back. Online surveys have lower response rates than paper surveys, and telephone surveys have the lowest response rates of all.
This means you have to do your best to get the most responses that you can. Here are some ways to do that.
Send an email to the people you want to survey telling them that the survey is coming. This is an invitation to them to participate. Make your email invitation personal – use people’s names. There is software that helps you to personalise correspondence (eg MailChimp).
Make the most of your email subject line. Research has found that the following words increase the chance that people will open your email: words that imply time sensitivity (urgent, breaking, important, alert), words that sound intriguing (announcement, invitation) and words that imply gratitude (thank you).
Reminder! doesn’t work at all apparently. Oh, But First Letter Capitalisation Can Help Slightly. 4 Whatever you write, keep it short. (‘Breaking news! Community Sector Gets Thanked!’ might work well.)
If your subject line has intrigued people enough to open the email, you should have a well-written invitation in the email for them. Again, keep it short but make sure it has this information in it:
- Who you are and the purpose for the survey
- The survey’s benefit to the person you are asking to complete it
- The length of the survey
- What confidentiality you are ensuring
If the survey is short emphasise that but be honest about how much time it will take.
Then, send reminder emails at intervals. Don’t send more than two. If you have collected contact details filter out people who have completed the survey. Change the subject line and if you can, change the sender (get a colleague to send it for you).
Give people enough time. Don’t rush people, and be flexible with your time frames. It’s recommended that you allow at least two weeks for people to fill in a survey.
You can consider offering incentives – gifts or prizes. This needs to be thought through. See Information on Prizes, Reimbursements and Icentives. Incentives don’t have to be large – you can even make them so that people choose whether they want them or not and they will still increase response rates.
And finally, don’t put people off! Poor design can make people unwilling to respond to your survey; this is called non-response bias. This could happen if you ask about information that is very sensitive, or because your email link didn’t work. If your survey doesn’t work well on smartphones you might not get much of a response from young people who you might need to hear from.
Make sure the people who respond actually finish your survey
The percentage of survey starters who make it to the finish is called the survey completion rate. Getting people to complete the survey is very important because some of your most important questions will be towards the end. But it’s quite hard to achieve; a lot of people start surveys and give up. Here are some tips to help your completion rate be as high as possible:
- Keep your survey as short as possible
- Keep the purpose of the survey in your mind while you are writing your questions
- Make your questions flow logically so people are clear about what you’re asking them to do and don’t get frustrated.
- Show a ‘progress bar’. This is a feature on automated online surveys which visualises how much of the survey a person has completed, and how far they have to go.
How many people should we send the survey to?
Obviously the more people you can survey the more accurate your information will be. If you think that the people you are surveying are going to have very different views then you’ll need to survey more to capture that variety.
If you are doing a survey of a broader group of the community and you want to be able to work out whether what you were concluding is true of all Tasmanians you have to consider your ‘sample size’ but unless your organisation has a funded research team it is unlikely you will do research on this scale. Just try to give everyone who falls into the category of people that you want to survey an equal chance to complete the survey and be clear about the size of the group you are talking about when you report the results.
Be accountable to the people who participated in your survey
Let the people who participated in your survey know its results. You can publish them online, let people know through your newsletter or through your email distribution list.
- 1. The research into how to do surveys is racing to keep up with the emerging technologies. This How To therefore draws on the lessons emerging from the world of commercial marketing, where online surveys are used regularly for customer feedback.
- 2. If you want to gather new information, a wide variety of views or to learn about something deep or complex you would be better off using qualitative research methods, like a focus group or an interview.
- 3. This is called non-response bias.
- 4. This article on choosing subject line words may help.