The ROI of Innovation Games

We’re often asked about the “ROI” of Innovation Games®, as in “What kind of ROI should I expect from using collaboration frameworks?”At times the question seems absurd, for reasons that I’ll explain later. Most of time, though,we’re quite sympathetic to those asking the question. Most of us see too many new products fail, experience painfully ineffective meetings, watch our companies waste precious resources on projects that are just plain bad ideas, and use bloated products with unnecessary features. People who have had these experiences seek assurance that investing in using Innovation Games® collaboration frameworks to create better products, improve meetings, eliminate unnecessary projects and prioritize feature backlogs (and project portfolios) will produce an appropriate return.

In this post, I will start a conversation on how you can develop an ROI model suitable for your use of the games/frameworks. Over the next few months, we’ll supplement this post with additional tools that you can use to calculate the value of the games. Be forewarned, though, that I will challenging simplistic notions of ROI, and when and why you think you need to calculate it.

[separator type=’transparent’ color=” thickness=’1′ up=” down=”]

ROI Factors

On paper, ROI could not be simpler. To calculate it, you simply take the gain of an investment, subtract the cost of the investment, and divide the total by the cost of the investment. Mathematically, it is:

ROI = (Gain – Cost) / Cost

In practice, ROI is hard to calculate because it is based on what you consider the gains and costs used in the calculation. Let’s call these ROI Factors. Because ROI is often characterized as hard, or  or soft, the factors themselves must also be characterized as hard or soft. Examples of hard, or direct, ROI, include such things as:

[unordered_list style=’circle’ animate=’no’]

  • Meeting Time: The time saved by using an in-person or online game to reach a conclusion in a meeting faster than traditional brainstorming;
  • Travel Costs: Using online games to eliminating the costs of travel, including such things as airfare, hotels, meals, and so forth;
  • Time-to-Market: Using prioritization games such as Buy a Feature to eliminate unnecessary features, thereby enabling a product team to release products more quickly.

[/unordered_list]

Soft, or indirect benefits, are based on less easily quantified, or second-order effects. Examples of soft ROI include such things as:

[unordered_list style=’circle’ animate=’no’]

  • Engagement: Companies who have asked their employees to participate in portfolio prioritization games have reported significant improvements in their employees engagement in the resultant projects: instead of questioning the priorities of the company and delaying work, these employees feel eager to get to work because they were asked for their opinions;
  • Customer Smiles: “This was fun, thanks.” “That was a fun exercise.” “It was hard.  But lots of fun.” These are actual copies from the chat logs of customers playing our games. Playing games with customers brings a smile. Put another way, when was the last time your customers thanked you for asking them to take a survey?
  • Sharing Tacit Knowledge/Experience: Games provide a natural way for the players to transmit tacit knowledge. Within a company, this tacit knowledge can help employees learn from others why a certain proposal is not optimal. When customers are playing games, we see them routinely sharing best best practices (“You really don’t want to buy the flibblebopper feature, because if you configure your system this way you can work-around the problem. What you need is the geeble, because it solves a problem for which there is no work-around”).

[/unordered_list]

As you can guess, hard ROI is easier quantify and calculate than soft ROI.  And while ROI isn’t perfect – it doesn’t include time, and trying to determine if an ROI factor is hard or soft can leave a team exhausted. Still, ROI does provide a useful tool for considering the use of the games, because it leads us to first determine the ROI Factors that we’ll use in the analysis.[separator type=’transparent’ color=” thickness=’1′ up=” down=”]

Choosing the Right ROI Factors For Analysis

Because Innovation Games are used to solve specific problems, it is important to select the ROI Factors that are relevant to the games chosen and the surrounding context. And while there are many ways to organize the frameworks (a subject for both a future post that is also part of a pending new website design), let’s consider three types of games based on our Ideas to Action Process, which follows well-known innovation and meeting processes. As you can see in this table, there are several ROI factors that you can use to help you frame your use of the games.

Ideation Shaping Prioritizing
Games that Fit This Category:
  • Buy a Feature
  • 20/20 Vision
  • Prune the Product Tree
Hard ROI: Generic Factors, affecting games in all of these areas. Let’s start with Hard ROI factors that cover all three of these areas:

  • Meeting Time: The time saved by using an in-person or online game to reach a conclusion in a meeting faster than traditional brainstorming.
  • Travel Costs: Using online games to eliminating the costs of travel, including such things as airfare, hotels, meals, and so forth.
  • Total Cost-Per-Participant: What is the total cost per participant? Note that Innovation Games Online tends to be neutral when used for market research regarding recruitment and participant incentives. The really big advantage for Innovation Games Online is that our subscription-based model does not force you to pay per-participant research fees.
Hard ROI:
Unique or specific ROI factors for these areas.
  • Idea Quality: What is the quality of the ideas generated?
  • Idea Quantity: How many ideas are generated?
  • Development Costs:The Shaping process will typically motivate some initial prioritization. This can be captured as part of the cost savings of not doing implementing a feature or a project.
  • Time-to-Market: Using prioritization games such as Buy a Feature to eliminate unnecessary features, thereby enabling a product team to release products more quickly.
  • Development Costs: The cost saved by not implementing a feature or a project.
Soft ROI:
Generic Factors, affecting games in all of these areas.
Here are some Soft ROI factors that cover all three of these areas:

[unordered_list style=’circle’ animate=’no’]

  • Engagement: Companies who have asked their employees to participate in portfolio prioritization games have reported significant improvements in their employees engagement in the resultant projects: instead of questioning the priorities of the company and delaying work, these employees feel eager to get to work because they were asked for their opinions;
  • Customer Smiles: “This was fun, thanks.” “That was a fun exercise.” “It was hard but lots of fun.” These are actual copies from the chat logs of customers playing our games. Playing games with customers brings a smile. Put another way, when was the last time your customers thanked you for asking them to take a survey?
  • Sharing Tacit Knowledge/Experience: Games provide a natural way for the players to transmit tacit knowledge. Within a company, this tacit knowledge can help employees learn from others why a certain proposal is not optimal. When customers are playing games, we see them routinely sharing best best practices (“You really don’t want to buy the flibblebopper feature because if you configure your system this way you can work around the problem. What you need is the geeble because it solves a problem for which there is no work-around”).

[/unordered_list]

[separator type=’transparent’ color=” thickness=’1′ up=” down=”]

But Some of These Factors are Still Subjective!

I know. ROI Factors such as “Idea Quality” is a completely subjective concept. It depends on how you define quality. When you’re dealing with ROI factors, especially in areas like ideation, your best approach is to be completely candid with about what you and your company are willing to pay for an idea. This can be challenging, especially since no one can accurately predict apriori how many ideas will be generated in an ideation event or the quality of these generated ideas. But, it is helpful to try.

In one client project, we were working with a group of about 140 executives. The client felt that if they got 500 total and 50 “high quality” ideas from the 1/2 day session they would be happy. The definition of “high quality” was that these ideas would pass the first gate in their Stage-Gate New Product Development Process. This means that each executive would need to contribute 4-6 ideas and that each group of 8 executives would need produce between 3 and 4 high quality ideas. This seemed entirely reasonable to us, but… what would this cost? And would the costs be worth it?

To put numbers behind this, let’s say that the fully loaded costs of an executive are $200,000. Let’s further assume that executives work 50 weeks/year, or 250 days, or 2,000 hours, for an effective “executive rate” of $100/hr. The 1/2 meeting therefore cost them $100/hr/executive * 4 hr * 140 execs = $56,000. Let’s add in travel costs, meeting room costs, facilitator costs, and say that the 1/2 day meeting cost an even $100,000. That means that this team was valuing an “idea” at $200 and a “high quality idea” at $2,000. Based on on their New Product Development metrics, this client felt that these numbers would be very good. (And yes, the math here is simple, but trust me… all ROI is simple math. If your ROI ends up looking like a Collateralized Debt Obligation, chances are pretty good that you don’t have any ROI and you should avoid the investment).

When we finished the event, the client team claimed they had identified 560 ideas and 65 high quality ideas – about $1500/idea. They were very pleased with the “ROI” of the event. I’m not at liberty to disclose which of these ideas have materialized as actual products, but the result of this initial engagement is that this client continues to use Innovation Games® in these meetings.[separator type=’transparent’ color=” thickness=’1′ up=” down=”]

Sometimes Hard ROI Isn’t Hard To Calculate

Product Managers and Product Owners in Agile Software Development teams use our games to prioritize their features. In Agile terms, an “Epic” is a really big chunk of work — something that you’d probably include in a press release! Epics are typically decomposed into smaller, more digestible chunks of work called “stories”. A story has the quality that it can be completed within a specified time box referred to as a “Sprint”. To see why Agile Teams work so hard on prioritization, let’s look at the hard costs associated with delivering a new epic into the market. This is an interactive spreadsheet, and I encourage you to enter your own values. As you can see, the point is that sometimes it can be quite easy to calculate the ROI of playing our games.

[separator type=’transparent’ color=” thickness=’1′ up=” down=”]

When ROI is a Substitute For Fear

There are times when a potential customers’ insistence for “guaranteed ROI” borders on the absurd: It is actually quite rare that business can generate ROI from collaborative activities without some degree of work and/or changes to their existing process. And these kinds of ROI come with risk. Unfortunately, not all people want to acknowledge risk, especially when it comes to investments. To see what I mean, consider a supermarket that needs to choose between repairing a broken rooftop compressor or replacing it with a new one. The comparison can be done quite straightforwardly, and depending on the payback period of the new compressor the supermarket manager can generally make a clear choice. This clear choice is not so clear when it comes to the kind of ideation and collaboration infrastructure created by the games.

I suspect the real reason that most people attempt to demand “guaranteed ROI” is not because they need a precise and accurate ROI, or even a perfect guarantee. Instead, I think their question is motivated by fear, such as:

[unordered_list style=’circle’ animate=’no’]

  • Fear of the unknown: The concept of serious games to solve business problems is relatively new. Early adopters may not need economic justification, but the early majority does.
  • Fear of the word games: After all, aren’t games supposed to be fun? We’re a serious business. We do serious stuff. We don’t have time for games or fun.
  • Fear of delayed ROI: Suppose you acquire an annual enterprise license for Innovation Games® online. What if you don’t get the “ROI” in the first 6 months? Indeed, how will you know which game your company played produced the ROI, as any single game could provide the requisite ROI.
  • Fear of looking bad: What happens if we try a game and our customers or employees don’t like it? Will I look bad? Will my boss fire me? Better not try this until it is “guaranteed”.

[/unordered_list]

Note that I’m not backing away from the need of these people to calculate an estimated or potential ROI from the use of the games. Calculating a game-based ROI is definitely a way to reduce these fears. And these ROI calculations, when used as a precursor to playing the games, can help sharpen the focus on producing actionable results, which we consider a great use of ROI. However, ROI can’t overcome the fear of changing the way you work or the fear of the unknown.[separator type=’transparent’ color=” thickness=’1′ up=” down=”]

Using ROI to Compare Options

There are times when ROI is used to compare different ways of achieving the same goal. In these cases, your first step is to determine the full set of costs associated with the project before considering the potential gain. In many situations, our clients find that the costs associated with using the games are the same or lower than costs of other techniques. But watch out! You’ll find that these kinds of ROI analysis can’t really be done as easily as you’d like, because choosing a different way to achieve the goal actually changes the goal you’re trying to achieve.

For example, let’s suppose that you’re using the games to prioritize potential features for a new product and you’re comparing traditional surveys with Buy a Feature online. Some costs that are the same, regardless of the tool:

[unordered_list style=’circle’ animate=’no’]

  • Developing the participant screener.
  • Recruiting participants.
  • Paying a research incentive.

[/unordered_list]

Some costs that will be different:

Traditional Surveys Buy a Feature ROI Considerations
Preparing the questionnaire. Preparing the features for the game. Most of our customers report that preparing the features for our games improves communication between product management, product marketing, and product development because features are described in terms of benefits and projected costs. Note also that you can’t easily design a single questionnaire to get the same data that you get in a Buy a Feature game, because a Buy a Feature game produces three kinds of results: the priorities of the customers, the reasons behind these priorities, and critical requirements that shape features.
Administering the questionnaire. Playing the game. Surveys don’t include facilitators, so it is easy to assume that the costs of playing a game with a facilitator must be higher. However, make sure that you’re comparing the right numbers. A single game played with 7 people is equivalent to 7 completed questionnaires.
Analyzing results. Analyzing results. The costs associated with analyzing results varies with the design of the survey. For example, while surveys that don’t include any free-form text input are easier/cheaper to analyze, they lack the ability to generate deep insights into the motivations behind survey choices. Our games, on the other hand, include rich chat logs that provide deep insights.
People hate surveys. People love games. This is the kicker, isn’t it? How do you calculate the ROI of customers who were happy to give you their feedback vs. customers who reluctantly took a survey? Since it has inherent value, you can put a number against it and include it in your calculations. But keep in mind that “taking a survey” needs to factor into the “cost” factors of ROI (since people don’t like to do it) while “playing games” needs to factor into the “gain” of the method (since people like to do it).[separator type=’transparent’ color=” thickness=’1′ up=” down=”]We have had a mixed set of experiences with customers who calculate comparative ROIs. In some cases customers were expecting certain categories of costs to simply disappear (e.g., “What? You mean we still have to develop a screener? Uh, yeah, if you want to get the right people playing your games.”). When these costs didn’t disappear, they decided to go with their tried and true method. While this is disappointing, a far greater disappointment occurs when the ROI for the games is better than traditional approaches but the customer still decides to go with a traditional approach. This isn’t a rejection of the ROI, but a fear of the method (see previous section).

[separator type=’transparent’ color=” thickness=’1′ up=” down=”]Fortunately, most of the time a comparative ROI calculation opens up more meaningful conversations about the merits of games vs. traditional approaches, which inevitably leads to discussions of “Business Value” vs. ROI. There are a large number of actions that we take in business where the individual ROI of any one action is either unknown, impractical to calculate, or negative, and yet we continue to do them because the accumulation of these actions produces sustainable business value. For example, every month I wrote this blog and Tami Carter publishes our newsletter. These are the actions of a company who is committed to the long term success of its customers and has a similar commitment to success of a new industry. However, I’d never ask Tami to calculate the ROI of single newsletter. It just doesn’t make sense.

It is really quite exciting when customers change their conversation from ROI to business value, because this means that we’re getting even closer to determining how games can help them accomplish their goals, and, typically results in customers using the games.[separator type=’transparent’ color=” thickness=’1′ up=” down=”]

When ROI Is Absurd

Many years ago, I was hired by Qualcomm to help them create an ROI tool for their FleetAdvisor System. As I was developing this tool, one of the Qualcomm executives I interviewed said “I never believe the ROI shown in these ROI tools. They all demonstrate absurdly large ROI after the second year. Our customers aren’t stupid, and telling them to expect an unrealistic ROI from our systems insults them.” He was right, so we worked on several ways to present a realistic ROI.

This, then, is my own concern with calculating the ROI of Innovation Games. It can be absurd. We’ve seen teams playing Product Box generate the seed idea for a multi-million dollar revenue software product platform. What portion of the ROI should we claim when the idea is clearly shown to be an outcome from playing the game? While some companies might try claim the complete ROI, we can’t, because we know that good ideas are not enough. Smart Product Development, Effective Marketing, Efficient Sales… the list goes on. All contribute to the total ROI.

We’ve seen globally distributed teams playing Buy a Feature online eliminate more than $10M of unnecessary projects from the project portfolio. Amazing, stupendous, and yes, a bit absurd, ROI. But if that helps you make the case to your management team, go ahead and use it.

What’s the value of a new idea? What’s the value of avoiding unnecessary work? Tell us your numbers for these and we’ll show you how Innovation Games can exceed your ROI requirements. Without being absurd.


On Choosing a Game (a.k.a. Collaboration Framework): Goals, Verbs, Nouns and Context

Designing and producing effective Innovation Games® and other collaboration frameworks boils down to gaining an understanding of these four key elements: goals, verbs, nouns and context. This post explains how to use these four key elements to design and produce great in-person and online forums, drawing on some examples from our client successes over the years.[separator type=’transparent’ color=” thickness=’1′ up=” down=”]

Goals

Goals are pretty straightforward: They are what you want to accomplish, ideally framed as a higher level business outcome. For example, you might be working for a company who wants to increase the average price per sale of large deals. Alternatively, you might want to improve your New Product Development (NPD) process by getting better concepts from the “fuzzy front end”.

Framing your goals in terms of measurable outcomes (“increase ASP by 10%”, or “generate 100 ideas that pass the first gate in our gate review process”) helps ensure that you’re choosing the right game – or no game at all. To illustrate, let’s look at two closely related business goals, both related to portfolio management.

A few years ago, Sallee Peterson, Senior Vice President of Global Customer Care for VeriSign, asked us to help her team prioritize a list of 46 projects. The business goal was to leverage the “wisdom of the crowd”, her globally distributed team, in selecting the best projects. (The careful reader will note that this last phrase is part of the context; I’ll return to this in a bit). We have a number of prioritization games, so this initial goal was already a good fit for the games.

Another leader contacted us to see how we might be able to use the games/collaboration frameworks to communicate the selection of projects within his company’s portfolio as they progressed through its gate process. In this case, the business goal of communication is not well served by the games. We recommended a simpler approach: a Wiki with an RSS feed and small, personal video updates on project status prepared by the project leaders. And it is easier than you think, especially if you own a phone that does video or a small video device. [separator type=’transparent’ color=” thickness=’1′ up=” down=”]

Verbs

Verbs are the actions that we take to accomplish our goals. Indeed, I am starting to think that if a business goal cannot be directly related to a verb, then it isn’t a good fit for a collaboration framework.

Returning to Sallee’s portfolio selection problem, the verb that equated to the goal was prioritize. Which is an amazingly powerful and common verb in business. We prioritize sales deals. We prioritize product features. We prioritize market segments. We even prioritize personas into “primary” and “secondary” personas.

When teaching the games, I will often take a few minutes to list as many business verbs as possible in five minutes or less. One group at Cisco generated more than 30 verbs, including generate, create, define, elaborate, group, establish, plan and develop.

As you gain experience in using collaboration frameworks, you’ll also gain experience in how the tense informs the business goals. Most of the time the tense is in the future, and you have to determine the time scale to determine the time frame of action (tactical or near term; strategic or long term). Other aspects of how the verbs of your business goals are used are important, so pay attention to the verbs associated with the goals.[separator type=’transparent’ color=” thickness=’1′ up=” down=”]

Nouns

Verbs are useless without the nouns they operate on. We prioritize sales deals, product features, the location where we’re going to have our corporate off-site, which customers we’re going to invite to our strategic advisory board meeting and so forth. The nouns/objects of the games become the metaphors we use in the visual collaboration forums, the items in a forum using the Buy a Feature collaboration framework, the objects and relationships between them in a Spider Web collaboration framework and so forth.

Once you understand the goals, verbs and nouns of the project, you’ll be well on your way to selecting one or two collaboration frameworks. Sometimes you don’t even realize how quickly this happens, so it is good to take a step back to ensure that you’ve got the right goals, verbs and nouns.

To illustrate, a few years ago Aladdin Knowledge Systems (since acquired by SafeNet) hired us to design and help produce a two-day sales training course for its world-wide distributors for a major new product launch, HASP SRM. Acting as strategic account managers, these distributors help SafeNet’s customer design and implement comprehensive Digital Rights Management solutions.

A simple statement of the goal was to train the distributors. A better goal was to focus on the actual outcomes a well-trained distributor would produce: more sales. We get more sales when the distributors can sell (verb!) HASP (noun!). We therefore designed a forum based on Product Box,  in which the distributors created boxes that demonstrated their ability to sell in complex situations.

To see if this was the right design, we had to compare it against the last element: context.[separator type=’transparent’ color=” thickness=’1′ up=” down=”]

Context

Context refers to such factors as who will be participating in the forum, their physical location, how many players will be included, whether or not the game will be part of a larger event and so forth.

Returning to Sallee’s challenge, the context included a 230+ team of customer support employees distributed across four locations around the world. In this case, the only option was to use Decision Engine and the Buy a Feature collaboration framework and construct a “feature tournament”. The resulting design enabled approximately 60% of the global workforce to participate in the games.

Changing the context will change how you design and produce the forums. NetApp, Wyse, and Rally Software Development are clients who have used the collaboration frameworks within customer advisory board meetings. In all of these cases, the small in-person context motivated them to use in-person versions of the collaboration frameworks.

Note that in these examples the selection of participants was also quite straightforward: Sallee wanted to include her employees, while the other examples wanted to include key/large accounts. Sometimes, however, choosing participants isn’t so straightforward, and you have to develop careful screening criteria to ensure the players are producing results that enable you to realize your goals.[separator type=’transparent’ color=” thickness=’1′ up=” down=”]

Two Kinds of Verbs

It is important to realize that there are often two verbs associated with a well-designed and well-produced set of games. The first verb is the ultimate action that the client who commissioned the event wants to take. The second verb is the verb that is associated with the selection of the frameworks. These may or may not be the same verbs.

To illustrate, let’s one more time refer to Sallee and the VeriSign portfolio event. The ultimate business goal of the customer care projects was to improve the quality of customer care. To serve this goal Sallee needed to prioritize her projects. An online Buy a Feature tournament was the best way to accomplish this goal. And by including as many of her employees as she could in the prioritization process, Sallee also created “global buy-in” for the results of the forums.

Of course, Sallee and her team still needed to implement the selected projects. And while frameworks like Remember the Future and Product Box can help plan projects, and frameworks like Start Your Day and Spider Web can identify hidden requirements, and games like Speed Boat can identify potential risks, ultimately Sallee and her team needed to implement the selected projects.

While this post has been informed by years of using collaboration frameworks to solve business problems, special thanks goes to Jenna Cline from Cisco, as our conversation on the nouns and verbs of collaboration enabled me to think that much more clearly about this topic. Thanks also to Harbinder Kang, also from Cisco, who further challenged these ideas.