When four of the world's leading data visualization experts used AI to remake the same chart, they were thrilled with the results. Aesthetically gorgeous. Impressive. Done.
Then someone checked the numbers.
Some of the key performance indicators were wrong. And here's the thing: these are not beginners. These are people who have spent careers thinking critically about data. Yet a polished-looking output threw them off. As one of them put it, "if something looks pretty, we love it, and it weakens our defenses against verifying the numbers."
That's the core risk of AI-assisted data work. It's not that AI always gets things wrong. It's that when it gets things wrong, it does so confidently, and the output often looks too good to question. Imagine asking AI to summarize your program's outcome data, and it reports that 68% of participants improved. Even though that number may sound plausible, it could be calculated on the wrong group. If you didn't pause to ask where that number came from, you might repeat it to a funder.
So how do you protect yourself? Ask AI one question at a time, and check each answer against what you already know before moving on. And always ask: does this result match what I'd expect based on my experience with my community or programs? If something looks off, trust that instinct. It's a signal to slow down, not speed up.
The good news: you don't need to be a data expert to catch AI's mistakes. You need to know your data and apply the same critical eye you'd bring to any summary someone handed you. The tools have changed. That habit of mind hasn't.
Drawn from Chart Chat 71: Supercharge v. Sabotage, with Jeffrey Shaffer, Steve Wexler, Amanda Makulec, and Andy Cotgreave. Watch the full episode here.
Let’s talk about YOUR data!
Got the feeling that you and your colleagues would use your data more effectively if you could see it better? Data Viz for Nonprofits (DVN) can help you get the ball rolling with an interactive data dashboard and beautiful charts, maps, and graphs for your next presentation, report, proposal, or webpage. Through a short-term consultation, we can help you to clarify the questions you want to answer and goals you want to track. DVN then visualizes your data to address those questions and track those goals.
