How to Measure Member Value: Four Disciplines that Separate Good Research from a Software Export
Most associations run member research internally at some point. We understand why. On the surface, surveys can appear straightforward. The tools are accessible, response collection is automated, and it can feel faster and more cost-effective to manage internally.
The challenge is that good research is not defined by how many responses were collected. It is defined by whether the evidence is robust enough to support confident decision-making.
Poorly structured questions, biased wording, unrepresentative samples, low engagement from key member segments, or misinterpretation of results can all create a false sense of certainty. The findings may appear credible, but subtle flaws in the methodology can quietly distort priorities, hide emerging risks, or lead organisations toward decisions based on incomplete or misleading evidence.
And that matters because associations increasingly rely on member research to guide strategy, shape advocacy, prioritise investment, strengthen member value propositions and engage boards and government with confidence. When the evidence base is weak, the consequences are rarely immediately obvious, but they can significantly influence long-term direction and outcomes.
With that in mind, it is important to understand the difference between a survey that collects feedback and a piece of research that can genuinely inform strategy.
Strong research relies on a series of disciplines that are easy to underestimate when projects are being managed internally or under time pressure. Questionnaire design, sampling, respondent engagement, question order, interpretation of results and statistical validity all shape the quality of the evidence produced. When these elements are not carefully managed, surveys can still generate charts, percentages and commentary that look convincing, while failing to provide the depth, accuracy or rigour needed for meaningful decision-making.
The good news is that there are some clear principles that consistently separate robust member research from surveys that simply generate data.
Good research is a discipline.
Four parts of that discipline are what separate meaningful member insight from a simple survey export.
Question design.
The wording of a question changes the answer. Valid research tests for leading language, double-barrelled items, and the difference between what members value and how well they think you are delivering it, so the two can be compared cleanly. Asking ‘how satisfied are you with our professional development’ tells you next to nothing about the value your training courses provide. Asking ‘how important is access to professional development to you’ alongside *how well does the association’s professional development meet your needs’ lets you see exactly where any value gap sits.
Sample integrity.
A response rate is not a sample. While admittedly it can sometimes be difficult to control who responds or get balanced representation across all member cohorts, effective research thinks about who responded, who did not, and what that biases, and uses weighting and segmentation so the voice of the loud minority is not mistaken for the view of the whole membership. If two hundred members respond to a survey and most of them are the same active members who turn up to everything, the data is not telling you about your membership. It is telling you about your most engaged group. Effective weighting can fix this.
Benchmarking discipline.
A number on its own is interesting. The same number, compared against past research, against other segments of the membership, or against comparable associations over multiple years, is real evidence. However, that comparison only works if the methodology has been kept consistent and the benchmark is genuinely comparable. A detailed one-off survey is often less useful than a slightly shorter one that can be measured against three years of comparable data.
Interpretation.
Two associations can run the same survey and reach different conclusions because of how the data is read. A satisfaction figure of 65% looks low if you have not seen the benchmark but looks much better if you know the sector average sits at 60%. Or that last year your result was 50%. Interpretation is the part most easily underestimated and the most expensive to get wrong.
Four questions worth asking before you sign the research brief
So, before signing the brief on the next round of member research, four questions are worth putting to whoever is delivering it.
- How are the questions structured so that you have clarity around what the results mean, and so what members value can be compared with how well they think you are delivering it?
- How will the sample be drawn so it reflects the whole membership and not just the most engaged segment of it?
- What will the results be compared against, and is that comparison genuinely useful?
- And who is interpreting the numbers, and with what experience to draw on?
If the answers to those four do not give you confidence, the research is more likely to confuse the next board paper than inform it.
At Survey Matters, we approach research differently. Our role is not simply to run surveys. It is to ensure associations have evidence they can trust. That means rigorous methodology, thoughtful questionnaire design, meaningful interpretation of findings, and a clear focus on producing insights that stand up to scrutiny and lead to better decisions. We have been designing and benchmarking member research for Australian associations for fifteen years. If you would like a second opinion on a research brief before it goes ahead, contact us via info@surveymatters.com.au
---