What's A "Good" Survey Response Rate?

Recently, a client I’m building a data dashboard for asked me: “Our survey response rate was 30 percent. Isn’t that really good?”

I could see where she was coming from. Given how hard it is to get people to respond to surveys these days, 30 percent may feel strong—and it may be higher than what your organization typically sees.

But imagine she asked this instead: “I just bought a used car for $15,000. Isn’t that really good?” You’d probably say, “Well, it depends on the quality of the car.” If it’s a junker that can’t make it to the end of the block, it’s not a good deal.

Survey data is no different. A “good” response rate depends on the quality of the data. Rather than asking “Is my response rate good?” a more useful question is: “How representative is my survey data?” Your survey results might reflect what your clients, volunteers, or participants think and experience overall. On the other hand, they might mainly reflect the views of a small group of highly engaged, eager survey-takers.

To assess representativeness, compare your survey respondents to the larger group they’re meant to represent. Ask questions like:

  • Do respondents reflect the broader group’s demographics?

  • Are different programs, locations, or levels of participation represented?

  • Are newer participants responding at the same rate as long-time ones?

Even a high response rate can be misleading if certain subgroups are underrepresented.

If your respondents don’t look like the larger group, here’s what you can do:

1. Talk to staff and clients.

Gather insight on why certain groups may be less likely to respond. Barriers might include time constraints, language, survey length, digital access, or lack of trust in how data will be used.

2. Adjust how you collect data from underrepresented groups.

Based on what you learn, consider strategies to collect additional survey responses such as:

  • Offering the survey in multiple languages

  • Providing paper, text-based, or in-person survey options

  • Shortening the survey or breaking it into sections

  • Using trusted staff or community partners to encourage participation

  • Offering small incentives or emphasizing how feedback will be used

3. Be transparent about limitations.

Whenever you present survey results in data dashboards, presentations, your website, or social media, clarify who responded and who didn’t. For example:

“These results reflect primarily long-term program participants; newer clients were underrepresented.”

4. Apply lessons to the next survey.

Use what you learned to improve both response rates and data quality next time.

Let’s talk about YOUR data!

Got the feeling that you and your colleagues would use your data more effectively if you could see it better? Data Viz for Nonprofits (DVN) can help you get the ball rolling with an interactive data dashboard and beautiful charts, maps, and graphs for your next presentation, report, proposal, or webpage. Through a short-term consultation, we can help you to clarify the questions you want to answer and goals you want to track. DVN then visualizes your data to address those questions and track those goals.

Schedule A Free Consultation
 
 

A Better Alternative to Surveys?

When organizations want to understand the concerns, opinions, beliefs, or needs of their communities, clientele, or participants, they often turn to surveys. But I probably don’t have to tell you that surveys have many downsides. To name just a few:

  • The difficulty of asking the right questions in the right ways to really understand issues accurately and fully.

  • The difficulty of getting a decent response rate so that you can feel at least somewhat confident that responders’ viewpoints reflect those of the larger group.

  • The difficulty of extracting actionable knowledge from survey responses without a degree in data analysis.

I recently read about a tool that addresses some of the downsides of surveys. Polis is an open-source, real-time system for gathering, analyzing and understanding what large groups of people think in their own words.

Surveys present people with questions like this:

Source: https://www.examples.com/business/assessment/community-needs-assessment.html

By contrast, Polis allows participants to submit their own short comments on a topic specified by the “conversation” creator. Comments are then sent out semi-randomly to other participants to vote on by clicking agree, disagree or pass.

Most interesting to me are the visualizations that Polis generates in a report, which can be shared with all participants. The report includes, a viz like this:

Source: Polis

Each dot represents a comment or statement and is placed along a continuum to show the degree of agreement with the statement. In this conversation, you can see that there were many more consensus statements than divisive statements. And Polis says that’s usually the case. Polis can make consensus visible and thus may be a powerful tool when division so dominates our attention that we may be skeptical that any consensus among diverse groups exists.

When you scroll over a dot, the related statement appears below with stacked bar charts showing the amount of agreement (green), disagreement (red) and passes (gray) among participants overall and by opinion groups. An opinion group is made up of participants who tended to vote similarly on multiple statements and also have voted distinctly differently from other groups. Because the statement shown above (represented by the red dot in the chart) is toward the consensus end of the spectrum, the majority of participants in both opinion groups A and B agreed with the statement. That wasn’t the case for the statement shown below. Participants in opinion group A were much more likely to agree with this statement than those in opinion group B.

Source: Polis

The report also includes a summary of all of the consensus statements. (See example below.)

I can’t vouch for Polis — never used it myself — but I find its basic idea intriguing. If your organization is looking for a better way to understand a large group of people and is particularly interested in finding consensus hidden among all the noisy division, you may want to look into it.

Source: Polis

Let’s talk about YOUR data!

Got the feeling that you and your colleagues would use your data more effectively if you could see it better? Data Viz for Nonprofits (DVN) can help you get the ball rolling with an interactive data dashboard and beautiful charts, maps, and graphs for your next presentation, report, proposal, or webpage. Through a short-term consultation, we can help you to clarify the questions you want to answer and goals you want to track. DVN then visualizes your data to address those questions and track those goals.

Schedule A Free Consultation