Collaborative vs. Solo Forums for Market Research

CEO Luke Hohmann on why it’s more effective to use the Collaboration Framework Buy a Feature collaboratively than with a single customer.
Recently, I was asked why it’s more effective to use our Decision Engine platform (a.k.a. Buy a Feature/ Buy a Project) collaboratively, rather than using Buy a Feature with a single customer and asking him to allocate his sole budget to features he chooses.

It’s not an uncommon question, and it frames Decision Engine and Buy a Feature as an alternative to other choice or preference modeling techniques, notably “Forced Rank.” Forced Rank is a good comparison, because it helps highlights the differences between our collaborative prioritization engine and more traditional market research techniques.

Let’s start by comparing the results generated by the both Buy a Feature and Forced Rank. Both techniques uncover the priorities of the participants. However, using Forced Rank or Buy a Feature with a single person, will only give you insight into that person’s priorities.

Buy a Feature, when used collaboratively, gives you insights into two additional and critical pieces of information: The reasons behind the choices and the conditions of acceptance.

For example, if you scan the chat logs for a Buy a Feature forum, you’ll find that the player chats can be characterized as follows:

[unordered_list style=’circle’ animate=’no’]

  • Social: “Hello”, “How are you?” ….
  • Help / Usage: “How do I make a bid?”
  • Negotiations: “Sally, You should buy the Flibble because it will help us solve [problem] by [reason].
  • Conditions of Acceptance: “OK, I’ll help you buy Flibble, but only if it ….”

[/unordered_list]As you can guess, the payload of the negotiations and the conditions of acceptance chats produce far superior results for analysis. And skilled facilitators can encourage people to state their reasoning by asking them to elaborate while participating in the forum. For example, you may ask a player, “I see you’ve made a partial bid on the Flibble. What can you say to the other participants to motivate them to join you in purchasing the Flibble?”

Because Buy a Feature allows for solo purchasing, facilitators are also trained to prompt participants who’v made solo purchases for insights: “Ming, I see you’ve used a big portion of your money to purchase the Gleeble. Why is this so important to you? What impact will purchasing this item have in your work?” (Note, that we instruct our facilitators to typically “whisper” these prompts, so that the chats are not visible to other participants, affecting their choices.) We’ve found that these prompts generate highly actionable results. And if the facilitator prompts are timed immediately after the purchase has been made, we’ve found that people are far more likely to respond – especially because whispers are discreet.

Self-Reported Behavior vs. Actual Behavior
A common objection to forum results is that the data is based on the self-reporting of behavior and respondents often lie. Of course, they don’t mean to lie, but they do. After facilitating thousands of online forums, we’ve found that the online component tends to inhibit (inadvertent) lying, because it’s easier to call each other’s bluff in a calm and rational way when you’re not face-to-face.

Higher Quality Negotiation
We have chat log data that suggests for people who are used to negotiating complex priorities face-to-face, the forums produce more “equal” results, because they eliminate body language and other forms of negative coercive behavior. There is no leaning-in, raising voices or banging tables.

This is probably less important when leveraging the forums for market research, but I recall one online session from last January’s San Jose Budget games (in which we used the platform to engage citizens in making budget priorities). One participant was clearly getting frustrated as evidenced by his behavior in the forum. The facilitator skillfully encouraged this person to practice his arguments via whispers, thereby defusing a situation in which the facilitator might have had to kick the player out of the forum. (There are special powers that facilitators have in our system: “Whisper” “help” to the “System” in an online forum, and you’ll see them. We’re going to make these more prominent in an upcoming release.)

Data Quality
Let’s explore the quality of the data generated in these forums. I’m going to focus on the techniques that are most comparable: solo Buy a Feature (because simply giving people money is a solo version of the game) vs. collaborative Buy a Feature.

One dimension of data quality concerns the strength of the expressed preferences. When participants are acting alone, they hedge their bets… For example, an individual may think,“OK, I’ve got $100 to allocate. What do I want? A is really important – so I’ll bid $35. And B is also pretty important, so I’ll give it $30. That leaves me with $35. I think C and E are good, too, so I’ll give them each $15. And K also sounds nice, but I only have $5. Oh well, let me put $5 on K.”

The lack of conviction for C, E and K create less actionable results. Let’s contrast this with what happens when participants are collaborating to purchase an item. Let’s say that in this forum, Sally initially allocates her money to the items as stated above. But, the other players disagree with her priorities. They work to convince Sally to reallocate her money. And the final results are usually quite important.

Here are two real-world examples. First, in a series of forums we produced for VeriSign on how to improve tech support, less experienced engineers consistently supported a project that would introduce some self-service capabilities into the platform (like Cisco). More experienced engineers talked them out of this during the forums by pointing out inherent security flaws. The side benefit, of course, was critical education for the less experienced employees.

Second, in a series of forums that we produced for VersionOne, many of VersionOne’s customers initially purchased Item A, but were later convinced by other customers to instead purchase Item B. The reason? While both items were clearly important, A had a workaround. B didn’t. And it was through the conversations of the customers – the wisdom of the tribe – that the workaround was shared, allowing a better set of priorities to be generated.

You’ve probably noticed that when you’ve used Buy a Feature, there have been a few moments of laughter and associated feelings of “enjoyment” (or even joy) when an item was purchased. That emotion is related to a small amount of dopamine being released in your brain, precisely because you were able to overcome a shared challenge by working together as a team.

We’ve recently started discussions with Steve Martin, the noted social psychologist, to better understand the importance of these phenomena. Steve suggests that these feelings induced through play are especially important in project and portfolio management, because the positive emotions experienced in the forum contribute to teamwork and commitment to implement the project. Note that we’re not entirely sure how this impacts forum participants when they are outside customers engaged in market research, as opposed to internal team members.

Effective / Impactful Research vs. Statistical Significance
If you absolutely require statistical significance in your research, I would recommend using Conjoint or similar market research technique. The reasons are very deep, but the summary is that statistical sampling theory is based on non-collaborative behavior. As we have not yet worked out the mathematical foundations for statistical significance in collaborative forums, I can’t recommend our techniques when statistical significance is required.

The closest research that we’ve found in this domain is from Abbie Griffin, who suggests that you can get something like 75% of your core priorities correct by talking with as few as 31 customers. Our experience is that we start to see very actionable patterns around 5 to 8 forums. That’s more people than Abbie suggests is required, but I think it is related to the collaboration in the forums.

I continue to believe that our techniques generate high impact, more actionable results, far faster, and for significantly lower costs, than conjoint analysis. And unlike a survey, participants find the process extremely enjoyable.

San Jose Residents Play 4th Annual Budget Games

In January 2014, San José, CA residents had a chance to influence their city’s forthcoming budget by playing games. Representatives from San José Neighborhood Associations and the Neighborhoods Commission and Youth Commission joined 100s of their fellow residents  to share their perspectives and opinions regarding budgeting priorities for the City of San José. This year’s event was the fourth annual Budget Games, produced by Conteneo for the City of San José, and included, for the first time, online games to expand game play  to 100s more San José residents. (And was supported by Microsoft through  it’s donation of 100+ Surface Tablets for the event.)


Putting the Fun in Participatory Budgeting
The goal for the Budget Games is the same as other participatory budgeting initiatives — provide citizens with the ability to participate in their government’s budgeting process and provide city officials with actionable insight as they make the difficult decisions about city resources.

Since 2011, Conteneo has produced a specialized version of its “Buy a Feature” game for the City of San José. These Budget Games have been used to engage neighborhood leaders regarding priorities for spending, tradeoffs, and budget cuts for the City’s annual $2.9 billion budget. During the second year of Budget Games, 80% of the recommendations generated by participants were adopted and integrated into the City’s budget.

“Resident input is critical to San José’s community-based budget process,” said Mayor Chuck Reed. “When we’ve had shortfalls, our priority setting session provided early input to help rank difficult choices.

“More recently, we’ve added police officers, restored funding for gang prevention and intervention, and opened shuttered library branches based on feedback from our community.”

2014 Results: Budget Matters
This year residents who played both the in-person and online games were given 24 hypothetical funding proposals and the budget expected from a proposed ¼ cent or ½ cent sales tax ($34M and $68M) and were asked to purchase the items that were most important to them. Subject Matter Experts (SMEs) representing Police, Fire, Parks, Recreation and Neighborhood Services, Libraries, Budget and other disciplines, answered questions related to proposals during the game, as requested by the players. The games were facilitated by Conteneo’s global team of Certified Collaboration Architects, who donated their time for the event.

Road repair, fire and police department staffing, funding of community centers and public library hours are among the topics that residents prioritized during the 13 in-person games and 21 online games. Analysis of game results revealed that residents of San Jose continue to be concerned about public safety, gang and crime prevention, and that game “purchases” correlate with the available budget. Residents who played the game with potential revenue from the larger tax increase purchased more expensive items, made more solo purchases and invested more in pavement maintenance and a contingency reserve.  See the final Budget Games Results here.

“Like many cities, San José continues to face difficult budget situations that require tough prioritizations with direct effects on San José residents,” said Luke Hohmann, CEO and founder of Conteneo. “In the past, these effects have included reduced community services, employee layoffs and pay cuts, and deferred maintenance to balance the budget.

“Along with the city, we strongly believe that residents should have a voice in this process since budget decisions can make a big difference in the quality of life for all members of the community.”

[box_holder background_color=”]

Check out Luke Hohmann’s presentation to the City of San Jose on the results here.

To bring Budget Games to your community, contact Conteneo.





Innovation Games as Story Listening

I recently completed an unusually fun project: Paul Mantey from NetApp invited me, and my colleague and Certified Collaboration Architect John Heintz from Gist Labs, to make a series of short, educational films for the NetApp sales team. John covered Agile and DevOps, Paul presented NetApp’s completely unique value proposition for Agile DevOps, and Paul and I discussed how NetApp’s Impact Discovery Workshops, which are powered by Innovation Games® collaboration frameworks, radically change the sales process. It was a lot of fun hanging out in the NetApp film studios–Green screens! Super cool video gear! “On Air” signs!

NetApp’s Cathie Staley moderated and helped produce our sessions. In one session, Cathie interviewed Paul and myself on the art of story telling in sales. Our focus was on helping strategic account managers use stories to connect NetApp value propositions and market differentiating features to customer needs. And I loved this session because it allowed Paul and myself to make a full-circle link between the storytelling that shares value propositions in a compelling way and the story listening that is the foundation of the Innovation Games® collaboration frameworks.[separator type=’transparent’ color=” thickness=’1′ up=” down=”]

Beware PowerPoint Paula and the What and Why? Guy

Two of my favorite negative salesperson stereotypes are PowerPoint Paula and the What and Why? Guy. PowerPoint Paula blows into your office, demands an overhead projector, and then proceeds to bore you to tears with her carefully rehearsed slide deck. Her carefully rehearsed stories (cue customer story 3 on slide 7) is what I call a “show up and throw up”. Paula shows up, throws up slides — and you simply want to vomit.

The What and Why? Guy is at the other end of the spectrum. He comes into the office with a notebook, a pen and a set of questions that always seem to end in Why: “What do you need? Why?” or “What are your strategic priorities? Why?” or “What can we improve? Why?” At best, the What and Why Guy is sincere (albeit creepily sincere). At worst, the What and Why Guy is merely interrogating you in an effort to close a deal.

In stark contrast to this are the approaches that Paul Mantey is pioneering at NetApp and Kevin Parker is taking at Serena.[separator type=’transparent’ color=” thickness=’1′ up=” down=”]

Changing Complex Sales Through Story Listening

A NetApp Impact Discovery Workshop is a structured workshop in which NetApp customers play tailored Innovation Games® collaboration frameworks to identify high impact business opportunities. In the process, the NetApp account team and NetApp partner sales and service teams gain a deep and thorough knowledge of customer needs.

The key is that these workshops are designed to allow customers to tell their stories. And when customers are telling stories, NetApp is learning what is really needed to serve them.

For example, in one workshop NetApp customers played Speed Boat to identify the anchors that would prevent them from rapidly deploying a new production system. By asking customers to draw their own boat, describe their destination, and then identify the anchors that might prevent them from moving quickly, NetApp was able to create an environment that allowed customers to tap into their true goals. By simply asking customers to share stories about their anchors, NetApp was able to identify a significant number of opportunities.[separator type=’transparent’ color=” thickness=’1′ up=” down=”]

Creating Alignment on Priorities Through Knowsy®

Every salesperson involved in a complex sale will tell you that to close a complex sale you must do at least two things: You must determine the priorities of each person, and you must create alignment on a shared set of priorities that will drive the sale. While most successful salespeople go about this process a bit more effectively than the “What and Why? Guy”, the reality is that determining decison-maker priorities in a complex sale is not all that much fun. Until now.

The Social IT Game is Serena’s game to identify IT buyer priorities in a complex sale. Powered by the our Alignment Engine (aka the Knowsy® platform), The Social IT Game turns the act of identifying and understanding the degree of alignment that exists within a team with a super fun game. And once a salesperson has a group of decision makers talking with each other about their shared priorities, they know that a deal is in the making. Check out Kevin Parker’s video explaining this game.[separator type=’transparent’ color=” thickness=’1′ up=” down=”]

Becoming a Better Story Listener

Everyone who has taken a Certified Collaboration Architect course featuring Innovation Games® from one of our qualified instructors  learns that one of the most important aspects of an Innovation Game® is the way that the collaboration framework induces the participants to tell stories while participating. More precisely, we strive to teach Facilitators how to induce stories during the forums, we discuss how Observers should be listening to stories, and we even lightly explore what kinds of stories each forum is likely to produce.

And while there are a lot of articles and books about becoming a better story teller, to build truly innovative products and services, you need to become a better story listener. Here are some suggestions on how to improve your story listening skills.
[unordered_list style=’circle’ animate=’no’]

  • Match the collaboration framework that you’re using to the stories you want to hear. If you want stories that explain relationships, consider Spider Web. If you want stories of an uninhibited future, or stories that capture the passions of your customers, consider Product Box. Stories of how adversity was overcome can be motivated by Remember the Future.
  • Listen for stereotypic story structures. Here are some common structures: I need (feature or capability) {so that, in order to, because} I want to accomplish (goal). My friends in the Agile community will recognize this as the User Story format, which is a great way to capture and communicate requirements. The key difference, however, is that in this post a Product Owner isn’t just sitting down and generating a lot of user stories. Instead, the user stories are generated directly by your customers through game play.
  • Let the rules of the framework you’re playing help you draw out stories from your players. Consider a common scenario: Branden, a Product Manager for a car company, is playing several online Buy a Feature collaboration frameworks with customers to help them prioritize their product backlog. During one forum, Branden notices that Susanne has made a significant bid on a new feature which allows the car to be configured so that it can automatically send signals to devices like garage doors to open them when the car is within a preconfigured distance of a specified location. This bid positions Brendan to learn more about the reasons this feature is so important and the conditions or requirements of acceptance by using the structure of the framework to get the stories that drive requirements.

[/unordered_list][separator type=’transparent’ color=” thickness=’1′ up=” down=”]
Branden: Susanne, you’ve made a substantial bid on the automatic arrival feature .What can you say to the other player’s to convince to join you?
Susanne: C’mon everyone — get the automatic arrival. It’s cool.
Ming: Susanne, don’t put your money there — buy the MPG monitor instead. We all need to save gas.
Satish: I agree — gas savings are really important.
(Brendan, whispering to Susanne): It looks like the other players are interested in saving gas. You’re going to have to work a bit harder to convince them.
Susanne: I agree that saving gas is important.
Susanne: But I live a kinda bad neighborhood so I installed an alarm system. Sometimes I forget to turn it off properly when I get home, so we get false alarms. If my car could somehow tell my home when I’ve arrived, I’d feel safer.

The important point is that it was the rules of the framework that motivated Susanne to tell a mini story on why a feature was important to her. This information can be used in a number of ways: determining the requirements of home alarm system interactions, improving marketing messages, developing more compelling personas, building patent fences around novel technologies, and so forth.[separator type=’transparent’ color=” thickness=’1′ up=” down=”]

Making Your Move

While the popular press is motivating you to tell better stories, we think you might find that listening creates even better results. What’s your take? Let us know at