ASC Conference 2011: the importance of multi-mode surveys, respondent respect and more

Tamara Gooderham, Research Services Manager at Snap Surveys reviews the first day of the ASC 6th International Conference held on 22 & 23 September 2011: “Shifting the Boundaries of Research”.

Keynote speakers on the first day included: Tom Ewing, Social Media Knowledge Leader at Kantar (now at BrainJuicer), and Jeffrey Hennning renowned market research blogger and Chief Marketing Officer at Affinova.

Tom Ewing in the opening presentation “Twilight of the respondents” entertained us with three respondent stories.

  • “Ghost Town” – respondents had been so badly treated they have disappeared and we should aim for increased efficiency in panels and improved engagement through gamification.
  • “Science Fiction” – respondents are likened to aliens and we should analyse existing data out there already rather than asking more questions.
  • “Spy story” – we are paranoid about what respondents tell us and we can’t trust respondents to self-report.

We were presented with a quadrant with low to high mediation on one axis and low to high trust on the other. Social Media Listening, Traditional Survey Research, Passive Data Collection and “We” Research (buzz phrase for 360 degree/ or peer-ratings) appeared in their respective corners.

In the future Tom thought Data Brokerage would be a potential as well as building Data Ecosystems. He also called for us to find a niche within “Big Data” and reminded us that questions are indeed the fuel of the internet and the engine of social media.

During the Q&A session Tom admitted that “respondents” should be called “participants” in line with one of the key themes that emerged from the conference – the need to respect our respondents or indeed participants more.

Alex Johnson, Director of the Innovation Group at Kantar Operations  discussed multi-mode surveys in his presentation “Engagement, Consistency, Reach – why the Technology Landscape Precludes All Three”.

He said that sticking to one survey mode reduced the representativeness of the sample, but each mode called for a different approach.

For example

  • PC-users want greater interaction
  • Mobile users simplicity
  • Tablets surveys need to have touchscreen functionality.

If the survey is very short and simple no-one really cares too much if it isn’t interactive. If you opt for a low common denominator no-one is happy. The alternative of setting up different versions of a questionnaire for each mode can be really expensive and consistency is lost.

(Editor. Not necessarily, Snap Survey Software enables customers to set up and manage multi-mode surveys efficiently and cost effectively although consideration needs to be made for different modes, for example smartphones).

The client needs to be made aware of the complex trade-offs between engagement, representativeness and consistency. Perhaps there is no solution to genuine consistency or maybe we just can’t have the control over the consistency we had before as the number of ways in which questionnaires can be delivered increases. 

We should monitor what devices are out there, the demographics of people using these devices and what people are willing to use their devices for. Maybe we just need to have simpler shorter questionnaires and break longer surveys up and join the data together later?

This thought-provoking paper complemented a joint paper presented later in the day by Snap’s Dr Steve Jenkins and Dr Nicola Stanley MD at Silver Dialogue, on the preferred survey modes for visitor surveys: Optimus Modus: Comparing interviewing modes for visitor surveys.

Jon Puleston of GMI gave a presentation on “Online research – Game on!” He had won the prize for best methodological paper earlier in the week at ESOMAR congress. I’m now following Jon’s blog – and am pleased to read that he’s a fellow fan of sliders in questionnaires!

The panel presentation comprised Tom Ewing, Alex Johnson and Jon Puleston. Honest, insightful feedback and suggestions abounded from speakers, panelists and audience during this session. Discussions included the much supported suggestion that the Market Research industry should set up a regulator which rates surveys on how engaging they are. Hear! Hear!

Jeffrey Henning, Chief Marketing Officer at Affinova gave a fascinating keynote speech at the end of the day in “Crowd Shaped Surveys”. He was very excited and optimistic about the future of research and picked up on Tom Ewing’s earlier presentation saying we’d been telling ourselves respondent ghost stories since the mid 80s when he first joined the industry.

He pointed out that our industry is very well aware of the problems we face with methodology and suggested seven ways that we could modify questionnaires based on the results so far.

The one that appealed to me in particular was “Question Buckets” where respondents were presented with only one of three possible open-ended responses. Once 100 responses had been collected these questions were dropped, reducing the burden of lengthy questionnaires and the cost of coding masses of open-ended responses which would most probably not change much beyond the first 100 responses collected.

At the end of his presentation Jeffrey asked us to stay sitting if we had been involved in survey computing two years, five years, 10 years ago and so on.  Those who were still seated after 15 years were applauded as true pioneers of our industry including our MD Dr Steve Jenkins, and Snap Chairman and ASC Chairman Pete Wills!

A thoroughly stimulating and enjoyable day.



Comments are closed.