When Visualizing Data, You Have to Fail to Succeed 05-01

When Visualizing Data, You Have to Fail to Succeed

by Jeff Bladt and Bob Filbin  | 

At 4:26 a.m. one morning last December, an email from our COO arrived in our inboxes. "Literally can't sleep — thinking about [your data presentation]..." As a data scientist, if your data causes co-workers to lose sleep, you've done your job, perhaps too well. The rest of the email outlined new strategies for member acquisition and engagement that will save our organization over $200,000. The inspiration? One map. It was a simple depiction of the U.S., which showed member engagement by city, but also hinted at the potential of segmenting our users in order to give them the right content at the right time.

Our data hasn't always prompted action. There have been plenty of times when it has hit the cutting room floor, even if the insights were technically valuable (they would save us money or acquire members). But data is only valuable if people are willing to act on it. Which means, as a data analyst, you've got to sell it.

Data visualizations are not art, they're advertisements.

Data presentations often feel like window-shopping — the data looks pretty, but co-workers aren't buying. This is especially true when people feel like they're at an art gallery looking at beautiful pictures rather than at a meeting discussing business problems. How do you make your data visually appealing, but also compelling enough that your colleagues are willing to spend their resources to act on it?

The world of product design points to two different approaches. The first is the "predictive" black-box approach: Build a big, fancy visualization in private, release it to the world in your presentation, and assume it will succeed. This rarely works. Data scientists are not often trained artists or designers who know exactly what their audience wants. The second, more successful tactic is an "agile" approach, similar to how restaurants use test kitchens: Create variations on a theme and see what sticks. This means crafting and testing ten different scallop dishes and recognizing that at best, one will make it to the menu.

Going with your first best idea is a high risk, low probability recipe for success. The sauce could be wrong, the portion size off, or the coloration may be unappetizing. Diners might love scallops, just not what you've created.

At most companies, failing is risky, because failure can be read as weakness. But here's a way to minimize risk: Make it part of your culture. We do that three ways at DoSomething.org: 

1. We hold an annual event called "Fail Fest," during which staff members speak about what they learned from a failure while wearing a pink boa.


2. We A/B test new product features.


3. We test our content, including data visualizations. To create the one map that impressed our COO, we failed at least 10 times.

Ten failures for one success. Quantity over quality is a common strategy in the natural world. When a fern unfurls in spring, it releases hundreds of thousands of spores, the vast majority of which never take root — some are caught by the wind, others stray onto tree branches. Of those that reach soil and germinate, many are trampled, eaten, or starved. Only a few reach adulthood and start the process again.

The lesson from nature is simple: The more we try, the more we succeed, even if the quality of each subsequent attempt does not improve.

It's foolhardy to believe that your first best-effort attempt is going to be the right one. But producing many quality versions is resource intensive. So you've got to move from many possible versions to the right one in an efficient way.

Sequence development to save time.

We do this by sequencing development phases: First, we make prototypes. Second, we test them. Third, we go to production. Many of our prototypes are nothing more than sketches on a whiteboard. Often we test five versions of an idea on co-workers, see what resonates, and then create five robust versions of the best one. We minimize the risks inherent in making just one visualization, while effectively allocating time away from ultimately doomed iterations.

For our engagement visual, we knew our goal was to convey that user segmentation is valuable. But which variable should we show: age, gender, mobile carrier, city, or first name? To answer this, we created simple chart visualizations of all five like this one:

Instead of perfecting the form, we showed these basic mock-ups to co-workers. Their responses were clear: the city data resonated most. So the next step was to create five more visualizations with that data, fleshing out the form.

Here we struggled with questions like: Should we show data in a table, a bar chart, or on a map? Should engagement be represented by a two-color scale, or one? So again we created five rough versions, collected informal feedback from co-workers, and iterated quickly. In the end, we created 10 visualizations, but covered a multitude of possibilities.

Below is the rough visual that resonated most with colleagues — they responded to seeing our membership engagement by city in a real-world context, but had trouble seeing patterns in the data.

In our final visualization, we moved to a two-color scale to highlight relative differences in engagement between cities. We also added lists of the five most and least-engaged cities. The lists alone provide the insight; the map adds depth and credibility.

There are lots of ways to sequence development. But the end principle is simple: create more data visualizations than you need to show, because your first idea is unlikely to be your best. Even for this article, we pitched five headlines and synopses to HBR editors. We then wrote five versions of their favorite. So, this article is one of the best of 25 permutations.

But is it good enough to change your behavior?

View at the original source