There's a Woody Allen joke in Annie Hall where someone complains about a restaurant: "The food here is terrible." "Yes," the other agrees, "and such small portions."
Nonprofit survey data can work the same way. Many nonprofits seem determined to satisfy the second person in that exchange. They may not be able to guarantee the data is good, but they will make sure the portions are large. The bigger the sample size, the better — regardless of what's actually in it.
It shows up like this: "We surveyed 847 community members and found that 73% felt more connected to resources after participating in our program." Impressive. But dig a little deeper. How many of those 847 actually completed the survey? How many answered only the first few questions and dropped off? How many responses came in suspiciously fast, suggesting bots or click-throughs? How many were from people who had nothing to do with your program?
A survey with 847 responses and rampant data quality problems is not better than a survey with 200 careful, complete, thoughtful ones. It's worse, because it creates an illusion of rigor while quietly undermining it. And it's that illusion — a big N, a confident percentage — that tends to end up in funder reports and board presentations without anyone asking hard questions about what's underneath.
Before you report your survey findings, ask a few basic questions. What was your completion rate? Did you screen out incomplete responses? Did anything in the data suggest respondents weren't engaging meaningfully?
More data isn't better data. It's just more.
Let’s talk about YOUR data!
Got the feeling that you and your colleagues would use your data more effectively if you could see it better? Data Viz for Nonprofits (DVN) can help you get the ball rolling with an interactive data dashboard and beautiful charts, maps, and graphs for your next presentation, report, proposal, or webpage. Through a short-term consultation, we can help you to clarify the questions you want to answer and goals you want to track. DVN then visualizes your data to address those questions and track those goals.
