Case Study: How to Take the Pain out Portfolio Prioritization

Ed. Note: This is the story of how Decision Engine and the Collaboration Framework known as “Buy a Feature/Budget Games” transformed the fourth largest telecommunications company in the world. 

MasaMaeda
Masa K. Maeda, CEO of Valueinnova.

In June 2014, Masa K. Maeda, CEO of Valueinnova, Playcamp organizer and Conteneo Certified Collaboration Instructor, began work on an Agile Transformation project at the Ecuadorean office of the fourth largest telecommunication company in the world. As you’d expect, this company has a corporate presence in each country where it offers services and among all of its offices, the Ecuadorian headquarters was considered the most innovative by the senior leadership team.

“The agile transformation began with a very positive impact,” Masa relates, “spreading from 34 people in one department to more than 200 people in eight departments in only six weeks. This happened despite the fact that original contract was for the transformation of just one department.”

“The key to such an accelerated rate of adoption,” Masa continues, “was the ubiquitous introduction and widespread use of high collaboration frameworks (a.k.a. “serious games”) in the teams and at most levels of decision making.”

This initial success gave Valueinnova the opportunity to propose to the general manager that the company use Decision Engine and the collaboration framework “Buy a Feature/Budget Games” to prioritize the company’s 2015 project portfolio, and Valueinnova’s proposal was accepted.

The Backstory

The company’s typical project portfolio prioritization process would begin in October and be complete in December. Each of the company’s twelve departments first prioritized its own project portfolio, which was comprised of 10 to 15 project proposals. The set of twelve prioritized project lists were then handed to a board led by the general manager.

“The board would then go through the painstaking and time-consuming task of merging all those projects to generate one project portfolio of around 140 projects!” Masa relate. “They also preserved the order of projects from each department. No project proposed by any department was rejected, save rare exceptions.”

The issues with the original process were:
[unordered_list style=’circle’ animate=’no’]

  • Resource and time consumption: Many employees and decision makers were involved for too long—they had to give up a good portion of their daily activities during the three month period.
  • Quantity and quality repercussions: Because the company didn’t make hard prioritization choices, they ended up with too many projects, causing some projects to be delivered late due to insufficient resources and other projects to be delivered with poor quality due to cutting corners.
  • Local optimization: Since each department did its own project filtering, the board rarely rejected any projects, resulting in green-lit projects that had little relevance to the company’s bottom line. This localized optimization problem meant that some departments which should have been given more resources to grow faster were starved of their potential.
  • Silo mentality: Each department focused on its own projects without knowledge or interest in the projects from other departments. This is also why the board only merged the departments’ portfolios and did no filtering.
  • Failing economy: All the issues above ultimately had a negative impact on the overall profitability and economic viability of the company.

[/unordered_list]

“By using the collaboration framework “Buy a Feature/Budget Games” and the online prioritization platform Decision Engine, we sought to minimize—and possibly eliminate—those issues,” reported Masa.

In the Beginning

The first step was to ask all 12 departments to create a Business Model Canvas (BMC) for each project that was to become part of its proposed portfolio.the-business-model-canvas-shadow-hero

“There was some hesitation,” Masa said, “because the teams were afraid that this would increase the time needed to create each project portfolio.”

However, creating the Business Model Canvases ended up saving time overall; the act of creating the BMCs collaboratively meant that the teams actually better understood each project and were able to eliminate irrelevant projects early on. The total number of projects in the portfolio of each department was reduced by 30% to 45%, Masa reports, so the total number of projects to be sent to the board was considerably smaller than in past years.

To make sure the person was focused on the most important needs of the business, each project was classified as either strategic or progressive during the second and third week of November. The progressive projects remained under the decision-making control of the departments while the strategic projects were elevated to be used in Decision Engine under the belief that collaborative prioritization among the department heads would produce the best overall choices for the company.

Preparation

In the second week of December, each department generated a spreadsheet that included each project name, a one-paragraph description, and one paragraph indicating its benefits and compromises.

What Happened When: The overall project timeline and deliverables.
What Happened When: The overall project timeline and deliverables.

“I used these spreadsheets to prepare the three-round Decision Engine tournament,” said Masa. “I gave a copy of the list to all the managers and the board who were to participate, three days prior to the tournament for them to read and start getting acquainted with all the projects. In hindsight, I should have given them more time, but the schedule didn’t allow it.”

The day prior to the tournament, Masa organized two activities. First, the department managers and the board gathered together for a set of presentations by each department on its proposed projects. Each project was allotted 5-minutes (3 minutes for presentation and 2 minutes for Q&A). Second, everyone participated in a practice session using the online platform, Decision Engine, using dummy data to ensure everyone was comfortable with the platform and the game mechanics. “I wanted them to be able to focus entirely the prioritization activity,” reports Masa.

More Data

Masa also added two new elements to the process to gather even more data. The data analysis done after a Decision Engine forum typically compares the exhaustive data gathered by the online system (chats, bids, purchases etc.). Sometimes, the producers will also assign observers to work with the facilitator to record notes on participant behavior, which is very valuable information that influences the study for better results. In this case, Masa decided to add:

[ordered_list]

  1. Video and audio recordings of the sessions, and
  2. Heuristics based on fundamentals of Bayesian Statistics, to weigh variables taken from game observation such that applying the corresponding algorithm together with the game results would provide a better prioritization.

[/ordered_list]

Show Time

Room preparation began one hour in advance. In addition to Masa, the facilitation team included two volunteers with experience in high collaboration dynamics. One volunteer handled logistics, the other audio/video recording.

PrunetheTree
The team also used the collaboration framework “Prune the Product Tree” to collaboratively prioritize projects in the overall portfolio.

“We placed three session tables so that I could monitor all of them at the same time from a central table where I had 3 computers set up to facilitate the sessions,” detailed Masa. The team also had high-quality video and audio equipment set up to record each table. And they posted Prune the Product Tree posters on one wall, with large sticky notes printed with all the projects titles.

Everyone was on time and when the tournament began, all tables prioritized the first 50% of the projects in around 50 minutes.

“This first session started a bit slow,” Masa relates, “mostly due to discussion about the projects, and fortunately not due to the platform or game mechanics, demonstrating the benefit of the practice session done the previous day.”

The participants had a 15-minute coffee break at the end of the first session, so that Masa could set up the second forum. The participants then prioritized the remaining 50% of the projects in only 40 minutes.

“At that point I had to take the results of both games from all three tables and extract the top 10 projects to run the third session,” Masa reports. “We didn’t waste the time, however. While I set up the next set of forums, the two volunteers facilitated a Prune the Product Tree forum with all the participants to prune the entire project portfolio. I was ready to run the final prioritization session by the time they were done with the trees.”

The last Decision Engine forum took less than 30 minutes to complete, and all participants were able to leave earlier than scheduled. According to Masa, their familiarity with the projects was a huge contributor to more effective and proactive discussions. The discussions were also shorter because they focused on the value of the projects, rather than on understanding them.

The day-of agenda for the in-person prioritization.
The day-of agenda for the in-person prioritization.

Analysis

Masa collected the video and audio recordings, and the Prune the Product Trees (thus pruned!) and returned to his hotel room to begin analysis.

“This was a very involved activity,” said Masa. “I had to listed to every recording very carefully and map the information onto relevant variables to apply my algorithm. This was rather dynamic, since the variables emerged from the observation itself rather than being pre-determined, but this made it more effective.”

“I also added the results of Prune the Product Tree as a variable. Criteria included aspects such as the order in which projects were being discussed and purchased, the level of interest, amount of participation and other for a total of 15 variables.”

Masa reports that the analysis consumed the better part of two days. Once the data mapping was done, he ran the algorithm over the data. “I was very pleased with the results, because with the exception of one project, all were in agreement with what I had learned and observed during the past weeks. There was no bias since I didn’t participate on the games, and the data feeds were based on the observation captured by the cameras, microphones and the Prune the Product Trees.”

Masa used the one project that was in a higher priority than expected as a point of verification by reviewing all the data related to it. He found that the data effectively gave the project higher ranking. He then proceeded then to write the full report.

The results

Masa met with the team who helped him organize the Decision Engine tournament first, and they were amazed by the results and pleased with his explanation. Masa reports, “They were also surprised by the same project that I had surprised me. But they too agreed based on the data that its higher priority was correct.”

“The low esteem, so to speak, towards that project was because it wasn’t a sexy project. So while most people didn’t care for it, it absolutely needed to be done because it had to do with external governance.”

The next step was to present the results to the board. They were very impressed by the quality of the results, the process itself, the fact that the entire process took less than three weeks, the reduced number of projects and the already obvious economic benefit that was taking place.

The department heads and those who participated in the prioritization were also very pleased. The teams that generated the business model canvases and their department’s portfolio, also related to Masa that the experience was fun and helped them truly understand the projects.

“The decision makers said that it was the first time in the history of the company that they truly understood all the projects, and truly collaborated,” said Masa. They even gave higher priority to projects that weren’t their own; whereas in previous years, it was a battle to defend their own projects.

Moral of the story? Using Decision Engine and collaborative prioritization to prioritize their annual project portfolio brought the best out in all of them.


Are Participatory Budgeting Games the Perfect Game?

We’ve completed our fourth Sprint for the San José Budget Programs, adding resident recruitment, improving copy, and providing more information on the overall process to our d3decides.com web site. The City has now started to work on the actual content for the sessions, which has motivated me to revisit the core design of both Buy a Feature and the Budget Game, my extension of Buy a Feature created in 2011 for Participatory Budgeting with limited resources.

A Theory of Fun
Buy this book – you’ll love it!

I’m looking at my own work with fresh eyes, as I’ve recently read A Theory of Fun, Ralph Koster’s marvelous book on gaming and gaming design. Ralph laid out a set of qualities and attributes that a “perfect game” would possess and it has me wondering if Buy a Feature and The Budget Game are the world’s perfect game.

To explore this, let’s start with a review of Buy a Feature and its progeny, the Budget Game, and then compare these to the attributes of a perfect game outlined by Ralph.

The Development of the Budget Game

We started producing Participatory Budgeting events for San José in 2011 with our first Budget Game. The detailed design of this first event can be found in my original post here, but for this post I’ll just summarize the rules of Buy a Feature and the Budget Game.

There is a list of 12-20 items for sale. These could be features for a dishwasher or government services, like keeping a library open. There is a set of scarce economic resources that individual players in the game control to buy what they want. Five to 8 players collaboratively purchase the items they think are most important. Once an item is purchased, it is purchased for the group. We explore the results to learn what was purchased (the priorities!) and why these were important (the negotiations among players).

 
What makes Buy a Feature especially “fun” is that most items require collaborative purchasing — that is, if the item costs $120, and each player has $50, then at least three players must contribute funds to purchasing the item.

What makes Buy a Feature serious is that you can’t have everything you want, so you have to choose, and choose carefully, because your choices will impact the city budget. What makes the game scalable is that you are working in a large number of small groups and we can scale these groups to the size of the physical space, or, using our online platform, to an unbounded number of groups.

What makes the results actionable are that people are purchasing “whole and complete” items. Specifically, if an item is not purchased, then it just isn’t as important as items that are purchased.

The “fun” aspect that I’ve come to appreciate better from reading Ralph’s book is that this negotiation is an intense form of learning – and this learning is “fun” – adult fun.

Buy a Feature works perfectly for collaborative Budget Allocation or Budget Investment activities. For example, in 2014 San José residents used Buy a Feature to determine how they might allocate a ¼ or ½ sales tax project to respectively raise $34M or $68M (official results from 2014 available here).

Buy a Feature does not work as well when organization is facing a significant deficit and needs to make cuts in a budget. This was the situation in San José in 2011, 2012 and 2013. We needed a different game – the Budget Game.

The Design Of the Budget Game

The Budget Game builds on the core mechanics of Buy a Feature in two ways. First, it starts with a list of potential items to purchase but gives the players a very limited, and typically zero, budget (we refer to these as the “green sheet”). Second, it provides a means for the players to acquire items by giving them a list of items they can CUT from the budget or a list of taxes they can raise (the “red sheet”). The trick is that players must unanimously agree to a red sheet item before they are given money. If agreement is reached, the funds associated with that item are distributed equally among the players.

I’d like to stress the importance of unanimously agreement. This is a very powerful, hard to achieve requirement. It requires subtle negotiation, listening, understanding and accepting the impact of a given set of choices. I invite you to look at the results of the Budget Games from 2013—you’ll see that unanimous agreement was NOT achieved on every item. Some groups of residents decided to raise taxes; others didn’t. Some groups of residents decided to cut services; others didn’t. The reasons were varied and compelling.

So the design of the Budget Game is:
[ordered_list]

  1. We have a list of budget items that community leaders can fund (the “green sheet”).
  2. Community leaders do not have enough money to purchase these items.
  3. We have a second list of budget cuts or tax increases that community leaders can select to get more money (the “red sheet”).
  4. The pricing and structure of items on either list are set by the City and cannot be adjusted.
  5. Community leaders are placed into groups of 5 to 8 people. Two Conteneo Certified Collaboration Architects manage the process, one as the Facilitator and one as the Obsever.
  6. There is no requirement that any items are purchased or cuts. The Community leaders are in complete control of their virtual money.

[/ordered_list]

Is Buy a Feature and the Budget Game the Ideal Game?

Let’s review Koster’s attributes for an ideal game and see how Buy a Feature and the Budget Game stack up.

Koster Ideal Game AttributeBuy a Feature / The Budget Game (BAF/BG)
It would be thought-provoking. BAF/BG are so much more than "thought-provoking": You're not only deal with the financial challenges of balancing a budget, your choices matter! If you agree to cut the police helicopter program (as San José residents did in 2011), you're agreeing to a significant change in operations. Understanding how this will affect both you and your fellow residents is deeply thought-provoking and the most essential attribute of a serious game: the results of the game materially affect the players, and in this case, the entire City (which of course explains why we're working so hard to scale this event to 50,000 individuals in Feb 2016).
It would be revelatory.BAF/BG reveal a number of hidden assumptions along with providing insight into how the City actually works. We've found that once residents understand the actual cost of specific programs they are better equipped to make choices on funding.
It might contribute to the betterment of society. The research from the United Nations is extremely clear: Participatory Budgeting substantially improves societies. It provides a direct voice from citizens to their elected officials. Over time, it creates positive system dynamics, as citizens see that their choices in the games are directly impacting their lives. In other words - the "game" becomes serious.
It would force us to reexamine assumptions.One of our favorite stories from the BAF/BG sessions is how residents of San José tear down assumptions of their fellow residents from other parts of the cities through these games. They walk in thinking they're unique in their values, hopes and dreams, only to walk out with an understanding that they share similar values, hopes and dreams with their fellow residents. They create strong relationships that form the foundation for action.
It would give us different experiences each time we tried it.We've been playing BAF/BG for years with residents of San José, and each year the experience varies based on the content within the game, the specific players in the game, and the individuals who are playing. The content changes are based on the emerging needs of the City. The players are randomly slotted from different parts of the City. And the participants are changing, individually growing and learning during and between each game.
It would allow each of us to approach it in our own ways.BAF/BG provide participants with a wide variety of negotiation patterns. We've seen "Collaborationists", people who promote a number of items early and strive to build consensus. We've seen "Kingpins", people who discuss and strategize and then use their money to make purchases and sway decisions at the end of the game. And while rare, we've also seen people who choose not to spend ANY virtual money as a signal to the City that they value fiscal restraint more than the funding of any specific services.
It would forgive misinterpretation-in fact, it might even encourage it.While BAF/BG are quite forgiving of misinterpretation, we certainly don't encourage it. Indeed, one of the key jobs of the facilitators are to identify potential misinterpretations and invite subject matter experts from police, fire, libraries, parks and other departments to answer questions from citizens.
It would not dictate.This is a subtle attribute, because all games "dictate" in a number of ways - the rules of the game, the resources available, the content (story) that drives the game, and so forth. I believe, though, that Koster was referring to the role of the game in "dictating" the actions (moves) of the players and the outcome of the game. In this regard BAF/BG score a perfect "10" - once you have money in the game, you can choose to spend it as you wish, including not spending it at all.
It would immerse, and change a worldview.You can't "change a worldview" without "reexamining assumptions", so I'm not entirely sure why Koster separated these concepts. However, he separated them, so let's explore how BAF/BG changes worldviews through two stories. In one session, a woman from an affluent neighborhood was advocating for increased library hours. She changed her mind, and everyone at the table needed tissues, after another woman explained why she needed support for the anti-graffiti project to reduce gang violence in her neighborhood. In other session, a staunch republican who vowed he never would support raising taxes joined his fellow citizens in supporting a $100M bond to repair city streets.

Is Buy a Feature and the Budget Game the Best Game Ever?

If you’re reading this far into the post I hope you realize that we’re not so full of ourselves that we truly believe that Buy a Feature and the Budget Game are the best game ever. Games are considered “ideal” or great in context and, as many game industry experts have pointed out for years, we play games until they become boring. Indeed, I think it is safe to say that there is only one (infinite) game that is the best game ever, and it isn’t Buy a Feature and the Budget Game.

That said (or read, if you’re picky), it is undeniable that Buy a Feature and the Budget Game are terrific serious games (ahem, collaboration frameworks) that are optimal for increasing civic engagement and creating actionable feedback for government officials.

But don’t take my word for it! I hope that you’ll read this post with enough skepticism that you decide to join us in San José on Feb 20th, 2016, as a Facilitator or Observer (register here). Create your own experience with these games, track me down, and let me know how you’d put on your game designer hat and mod these games to make them even better.

And in my next post, I’ll elaborate on my Agile 2015 keynote on why surveys SUCK and why games are better.