Design better surveys - improve survey responses

Planning a survey

Quantitative survey design hints and tips from the Snap Surveys seminar series.

Before you start creating your questionnaire

Take time to ask questions, think, plan your survey.

Ask yourself

• What are the key objectives for the survey?
• Is the information you want to gather going to be used to inform critical decision making for your business or organisation?
• Does the information already exist? Someone in your organisation may already have done a similar survey. It’s worth asking around before you start. Other departments may also have an interest in your project.
• Who is the target audience for your survey?
• Is this survey likely to run again or be required for benchmarking subsequent performance?

Also think about

Consultancy - depending on the level of experience that you have in conducting research projects, it may be worth investing in working with a research agency like Snap Surveys who can offer advice on framing questions to avoid bias, sample sizes, and give help in determining what the data is actually telling you once you have your results.

Incentives - are you offering incentives for people to complete your survey? If it is a prize draw, a number of smaller prizes may work better than one large prize. In the UK, MRS rules state that research agencies are not allowed to offer the products of the client sponsoring the survey as prizes. Check prize draw and other rules that apply to your country.

Invitations - how will you invite people to take part in your survey? Have you factored-in the cost of postal invitations and reminders if required? If it's a pop-up survey on your website how many people will see it if they have pop-up blockers enabled?

Analysis - what kind of reporting will you require? Thinking about this at the beginning may help to focus your questions.

Sample size - when determining sample size is there likely to be sub-group analysis you would like to carry out afterwards? Ensure you have large enough sample sizes (ideally 100+ per sub-group).

Write a short brief - outline your objectives, audience, timelines and budget

Planning the survey format

Multi-mode

Online surveys are quick and cost effective to implement, however if you need a representative sample you may need to consider using other methods (modes) to capture information. Consider multi-mode surveys if you need to reach different economic or age groups for example. Snap's survey software makes it easy to develop questionnaires for different modes, routing and other rules are automatically applied specifically to the relevant mode.

Paper and online survey combination

As you can see from the Greenlaw & Brown-Welty paper extract below » A Comparison of Web-Based and Paper-Based Survey Methods, for some surveys mixed mode may be more effective even though it may cost more to implement. Ask yourself how important this data is to your organisation or project.

And don't forget, with paper questionnaires you need to plan and cost how you are going to distribute the questionnaires, and manage reminders. If you have a large volume of paper surveys it may be worth considering scanning-in responses which speeds up data capture, and can be more cost effective than using in-house resource for direct data entry » We offer a scanning module with Snap.

Kiosk, iPad and PDA

Capturing data in the field used to be done with paper questionnaires, now the interviewer may well be using a PDA iPad or other mobile device. One of the key benefits is that results can be sent back immediately via the internet, or downloaded to a USB stick.

In certain areas kiosks also come into their own, and can provide an engaging way to gather information from respondents in waiting rooms etc.

Questionnaires for each mode can be set up easily in Snap including in multi-language formats. And of course there are other methods such as phone and SMS to consider.

^ Back to top

Creating a questionnaire

Value your respondents

You are asking respondents for their time to fill in your survey. In many instances there is no reason why they should take part in your survey. They are doing you a favour.

A well thought out survey tells respondents that you value what they have to say.

It may seem obvious, but when putting your survey together, look at it from a respondent’s perspective. For example, jargon used within an organisation can creep into a questionnaire which has no meaning at all to your respondents.

Keep it short

It’s a temptation to keep on adding questions because it would be nice to know, or you think it might be interesting. When colleagues hear that you are doing a survey they may also suggest adding questions too. Before the questionnaire balloons in size, think, do we really need to ask this or is it simply nice to know?

If questionnaires begin to seem irrelevant or over long to respondents, you may find your completion rates decrease. Snap captures partial responses which help to identify respondent drop-out points in your questionnaire.

An online questionnaire should generally take no longer than 15 minutes to complete. There will be exceptions….

Do tell people up-front approximately how long the survey will take to complete, they should be given the choice as to whether they want to invest the time to complete your survey or not. And it’s generally good practice to include a progress bar in an online survey.

Questions, questions

It may seem obvious, but do avoid ambiguous, leading and contentious questions. It's sometimes harder than you think.

Open ended questions

“People completing questionnaires can feel cheated if they don’t have the possibility of adding a comment. It’s particularly useful to include a free text box when people have ticked a ‘disagree strongly’ box, so they can elaborate on their answer."

Some research experts suggest that open ended or literal questions should be kept to a minimum. In online questionnaires in particular, it is very tempting to keep adding open ended questions. Piloting a survey can help to reduce the requirement for open ended questions in the main survey. Snap can collate frequent responses which can then be added to the questionnaire.

‘Free comments’ from open-ended questions often relate to action that needs to take place. This is particularly likely to be the case if the survey is about customer service or customer satisfaction. You can set up email triggers in Snap surveys to send an email to the appropriate department for actioning.

“It’s great having statistical data, but verbatim answers help me to drill down to key issues and expose things that I need to deal with quickly.”

Presenting literal responses in reports can be tricky for some. A recent post on the Snap forum asked whether spelling errors should be corrected for the results presentation. The consensus was “no”, although one 'poster' did correct literals as stakeholders otherwise spent time debating the finer points of grammar rather than focussing on the results!

Grid questions

Grid questions are widely used in online surveys. The real example above uses horizontal bars which guide the eye and make it more attractive and respondent friendly. We suggest however that the choice of a 10 point scale is perhaps 'overkill', and without explicit labelling would appear quite daunting to a respondent.

This is another real example of a survey we have come across - it may have been better in this instance to have used a grid.

Branding your survey

Snap's in-house designer has put together some general design hints and tips.

» Click to view design tips powerpoint

Involving and engaging

Where online surveys can really score over other types of survey is the ability to engage respondents. But as one research commentator put it, ‘Web surveys still tend to be the electronic equivalent of their paper counterparts’.

Sliders can be used in online questionnaires instead of grid questions. Image maps can also engage (or re-engage) a respondent. You might be interested in this short paper » Watch What I Do! Using graphical input controls in Web Surveys (PDF) on the use of sliders and image maps, written by Dr Nicola Stanley, Silver Dialogue, and Dr Steve Jenkins, Snap Surveys.

There is a range of options you can use to make online surveys a more enjoyable or rewarding experience – such as text substitution and masking. Using routing effectively is also a good way to gather the information you want. Don't forget: remove question numbers so respondents don't get confused when questions are skipped by routing.

» This example survey demonstrates text substitution, routing, image maps and scoring

Demographics

"Should demographic questions be included at the beginning or at the end of a questionnaire?" is often asked at our seminars.

Some will answer, “Definitely at the beginning”, others will say, “No, at the end is the best place”.

The short answer is - there is no right answer!

Positioning may depend on how sensitive the questions are, the level / guarantees of anonymity and so on. If questions appear too intrusive or seemingly irrelevant to the general overall topic of the questionnaire, you risk alienating your respondents. Online surveys might be better than interviewer led surveys for certain types of question.

You could use pre-seeding from a database to put in basic information (care needs to be taken when choosing the information to present). The respondent can check postcode and other information, which may make it easier for them to complete your questionnaire.

Stick by the general rule - if you don’t really need to know – Don’t ask.

Respondent confidentiality - the most important thing is to be completely open and honest about who will have access to the data and whether individuals will be identifiable or not.

^ Back to top

Publishing your survey

Pilot your questionnaire

Piloting your questionnaire is good practice, especially if the information you want is critical to your organisation. In the real world there isn’t always the time to pilot a survey, but it’s something to consider. Piloting can show you where there are potential problems.

• Questions being weak, difficult, or even impossible to answer
• The point/s at which respondents drop out, (Snap has a facility to capture partial responses which will identify these areas quickly)
• Piloting enables you to give accurate guidance on how long the survey should take to complete
• Highlight data capture problems

Data from a pilot gives you the opportunity to ensure that the questions you’re asking can be analysed and will provide the answers to whatever you’re investigating.

Inviting a response

At a recent Snap seminar, everyone agreed that a personalised email invitation was much more likely to elicit a response than a generic ‘Dear Sir / Madam’ email. Taking that idea a step further, people were often encouraged to reply if they felt personally involved in the subject of a survey. Asked exactly what might prompt them to participate, delegates gave the following reasons:

• “If I feel that my view counts.”
• “If the topic is interesting and I want to comment.”
• “If the survey is relevant to my job or interests.”
• “If the results of the survey might improve a product or service I use, or conditions at my workplace.”
• “If I will receive a copy of the report and can learn something from it.”

General rule: Keep invitations simple and to the point whether on paper or by email. » Setting up emails in Snap

Invitation reminder frequency

One of our long-time Snap customers, John Lemon, Student Liaison Officer and Survey Coordinator at the University of Aberdeen in Scotland experimented with the effect of reminder intervals on response rates.

The university needed to increase the number of staff and student surveys being run through the year, but the existing survey timetable couldn't accommodate more surveys on the same timelines. John tested the effect of shortening the length of time surveys were open and the reminder cycles, to see if there was any negative effect on response rates.

The overall conclusion was that there was no negative effect on survey responses, and they actually showed a slight improvement in some surveys. The shorter cycle has now been adopted by the university » The Effect of Reminder Intervals on the Response Rate of Web Surveys

As a general rule: the Snap Research Services team host an online survey for 2 weeks, send a reminder after 1 week, and if required, a second reminder a few days later. But you need to keep an eye on response rates and if they’re tailing off earlier, or later, change timings accordingly.

Response rates

At every seminar we are asked what the average response rates for surveys should be - it depends on the survey is the answer!

Dr Nicola Stanley one of our speakers suggested a 3 - 7% response for cold b2b emails. Staff surveys should have a much greater response rate, at Snap we have found response rates of between 80-90%.

^ Back to top

Evaluate and share results

You have successfully run your survey – how can you make your data work harder for you?

In Snap, results from multi-mode surveys can be easily combined to give overall results. These can be reported as overall responses - or broken down to show differences between response types.

Make use of derived variables

• To group responses
• Exclude don’t know codes
• Convert quantity responses into codes - in age groups for example.
• Combine responses to more than 1 question.
  

In Snap you can collect paradata for online surveys which you can use to show when respondents replied; how long they spent doing the survey; record ID’s that enable you to link in other data through database links.

Literals can be filtered by satisfied/dissatisfied etc. Build coding frames and turn them into coded questions which can be charted.

Take the boredom out of reporting with batches

In an ongoing survey the same set of results may be required on a regular basis. For example, to produce a set of the same tables and charts on a monthly basis. Rather than specify the same tables and charts individually each month, in Snap a batch with a filter can be set up. You can take this a step further with continuous reporting » Read how Hugh Inwood of The Research Box produces reports for his client, BH&HPA

Sharing the results

As we said earlier, you should value your respondents, they have given their time to answer your questionnaire. Where appropriate tell them about the results, but in a way that has relevance to them. Not everyone can read cross tabs or pages and pages of charts!

The same applies to stakeholders within your organisation, think about how to present the data for your target audience. Summarise the findings, if people want to dig into the results, give them the means to do so.

And another benefit of sharing the information is ...

"The combined knowledge of colleagues and stakeholders will, if managed effectively, always reveal better insight than one lone individual leafing through a research report." Mark Thurstain-Goodwin, Geofutures

^ Back to top

---

We hope we have covered the basics of setting up a good survey! If you would like a summary of this presentation
» Click here for our poster booklet