Surveys are a very useful tool for understanding your stakeholders. Amongst other things they can tell you what influences purchasing decisions, what customers are willing to pay more for, the levels of maturity within a particular area. It can uncover pain points, it can drive thought leadership and help uncover what you need to emphasize as part of a particular marketing drive.
Over the past 20 years or so I have been involved in fielding hundreds of surveys, mostly examining the sentiment of decision makers using or planning to use IT and business services – from outsourcing IT processes, implementation services, consulting, and business process outsourcing. These are a few pointers on getting the most from survey work. If you only take one thing from this, read the book on statistics I recommend below. This advice is aimed primarily at surveys being used for thought leadership purposes.
Ask the right questions
OK, this is probably the most important point. You’ve gone to the expense of paying for a survey or gotten people to setup interviews, or emailed clients asking for feedback, you really ought to spend time on getting the questions right. The best way to know what questions to ask is to do a few qualitative interviews. This way you know what the main issues are, particularly if you want to populate multiple choice questions with the most likely categories. It helps you to understand what’s likely to be important and get the categories and options right. Then it helps to test the survey to elicit feedback from some likely respondents, particularly if you are unsure about the complexity of the questionnaire or it contains any technical details. This way you can simplify anything that needs it and expand or even reduce the number of categories for some questions where it is necessary or make the questions easier to answer. However, it’s important to make this decision early on. It is a key part of the process and it can be time consuming and result in changes which inevitably lead to delay. Therefore, allowing ample time for this phase, before your survey goes live, will save a lot of stress.
When writing the questions themselves it’s important to keep it simple and make the questions understandable. This is not an opportunity to demonstrate the depth of your vocabulary, particularly if the survey is going to be translated. This is about making sure the language is simple and the meaning as unambiguous as possible. Also try to be polite and bear in mind that people are giving you their valuable time.
It may seem like an obvious point, but for most surveys the questions must be capable of being answered off the top of their head by most respondents from the group you are asking. So don’t expect people to have to refer to anything else or provide complicated metrics they’d have to look up.
Some questionnaire design debates you can ignore, particularly for thought leadership surveys:
- Middle / neutral responses for scale (Likert) questions. Often there is a debate about whether a scale should include a neutral or middle ground and you should force the respondent to make a choice. This is largely nonsense – particularly if the question is well worded and having a neutral position is a valid choice.
- Left-side bias. There is a school of thought that says you ought to start with negative options first to avoid positive response bias in scale questions. Left-side bias does exist – it’s around 30% of the standard deviation. Although you could argue that presenting the negative options first gives a negative bias, given that it is the left position that causes the bias.
- Rather obvious perhaps, but make sure it is really clear which way round the scale goes and stick to it consistently (yes, people have genuinely been tripped up by this!)
Think about the data you need
Before you even start writing the questions, it’s vital to list the main objectives for the survey in the first place and make sure that you have questions that align to these goals. It’s a simple point but very often it is missed. Also, it’s important to think about the likely headlines (the overarching themes, if you like) you want to extract from the survey and work backwards. It’s important not to let the story you want to tell blind you to the message your respondents are trying to give you. This exercise is far more about thinking in terms of meaningful analysis than pre-empting or even biasing the outcome because you are looking to be able to tell a particular story.
This is a good opportunity to have in mind how the charts are likely to turn out and think whether they will help tell the compelling story that you need. This means making sure that the type of question is capable of delivering data that can be visualised in a simple and straightforward manner. This is particularly important for the main headlines you want, so that the question is unequivocal and is saying what you want to say. Getting good quality data support for the key messages is more important than asking a huge quantity of questions, where the quality of the interview becomes diluted.
Don’t ignore the statistics, but don’t be a slave to them
OK you can’t really talk about surveys without someone talking about statistical relevance and how this applies to surveys, particularly in driving the initial sample size. Of course, this is very important particularly if your survey is driving an important marketing push and you want to present the data to the press. You, and the audience, want to know that the information produced from the survey is accurate and relevant and can be applied generally with customers i.e. it is a representative sample.
I don’t want to get into too technical a conversation about statistics but if you want an engaging way to understand this important topic I thoroughly recommend this book: Naked Statistics by Charles Wheelan. It is a really useful and amusing guide to stats, particularly if you want to understand the basics but don’t want to get stuck too much in the weeds.
On the flip side, it’s easy to get bogged down in trying to be rigorous where it is simply not possible in the same way as it is with quantitative data sets.
My biggest piece of advice is not to get too hung up on it, but here are a few important things to consider:
- If you want your survey to be relevant for a big group you need a sample of around 380. This gives you an error of plus or minus 5 percent with a confidence level of 95 – this means that for one in twenty questions the error would be larger than the plus/minus 5%.
- If you want to look at subgroups – so industry or geography for example – plan to have at least 25-30 responses in each group. Although the margin of error will increase for the smaller sample, you can gain insights and it makes the whole survey more representative.
- Ultimately with surveys of this nature the biggest errors come from a lack of understanding of the question, the questions being ambiguous and the respondents not being engaged – so you need to get this right because a sample of 1000 won’t be valid if the questions aren’t understood or the options are incomplete or wrong.
There is always help available!
At Market Prescience, we can work with you to build concise questionnaires that probe respondents in an engaging and understandable way. We excel at making sure that the survey elicits the right information and can be analysed to extract insight and deliver the value you need. Check out our analysis services here.
