Case Study: How to Take the Pain out Portfolio Prioritization

Ed. Note: This is the story of how Decision Engine and the Collaboration Framework known as “Buy a Feature/Budget Games” transformed the fourth largest telecommunications company in the world. 

MasaMaeda
Masa K. Maeda, CEO of Valueinnova.

In June 2014, Masa K. Maeda, CEO of Valueinnova, Playcamp organizer and Conteneo Certified Collaboration Instructor, began work on an Agile Transformation project at the Ecuadorean office of the fourth largest telecommunication company in the world. As you’d expect, this company has a corporate presence in each country where it offers services and among all of its offices, the Ecuadorian headquarters was considered the most innovative by the senior leadership team.

“The agile transformation began with a very positive impact,” Masa relates, “spreading from 34 people in one department to more than 200 people in eight departments in only six weeks. This happened despite the fact that original contract was for the transformation of just one department.”

“The key to such an accelerated rate of adoption,” Masa continues, “was the ubiquitous introduction and widespread use of high collaboration frameworks (a.k.a. “serious games”) in the teams and at most levels of decision making.”

This initial success gave Valueinnova the opportunity to propose to the general manager that the company use Decision Engine and the collaboration framework “Buy a Feature/Budget Games” to prioritize the company’s 2015 project portfolio, and Valueinnova’s proposal was accepted.

The Backstory

The company’s typical project portfolio prioritization process would begin in October and be complete in December. Each of the company’s twelve departments first prioritized its own project portfolio, which was comprised of 10 to 15 project proposals. The set of twelve prioritized project lists were then handed to a board led by the general manager.

“The board would then go through the painstaking and time-consuming task of merging all those projects to generate one project portfolio of around 140 projects!” Masa relate. “They also preserved the order of projects from each department. No project proposed by any department was rejected, save rare exceptions.”

The issues with the original process were:
[unordered_list style=’circle’ animate=’no’]

  • Resource and time consumption: Many employees and decision makers were involved for too long—they had to give up a good portion of their daily activities during the three month period.
  • Quantity and quality repercussions: Because the company didn’t make hard prioritization choices, they ended up with too many projects, causing some projects to be delivered late due to insufficient resources and other projects to be delivered with poor quality due to cutting corners.
  • Local optimization: Since each department did its own project filtering, the board rarely rejected any projects, resulting in green-lit projects that had little relevance to the company’s bottom line. This localized optimization problem meant that some departments which should have been given more resources to grow faster were starved of their potential.
  • Silo mentality: Each department focused on its own projects without knowledge or interest in the projects from other departments. This is also why the board only merged the departments’ portfolios and did no filtering.
  • Failing economy: All the issues above ultimately had a negative impact on the overall profitability and economic viability of the company.

[/unordered_list]

“By using the collaboration framework “Buy a Feature/Budget Games” and the online prioritization platform Decision Engine, we sought to minimize—and possibly eliminate—those issues,” reported Masa.

In the Beginning

The first step was to ask all 12 departments to create a Business Model Canvas (BMC) for each project that was to become part of its proposed portfolio.the-business-model-canvas-shadow-hero

“There was some hesitation,” Masa said, “because the teams were afraid that this would increase the time needed to create each project portfolio.”

However, creating the Business Model Canvases ended up saving time overall; the act of creating the BMCs collaboratively meant that the teams actually better understood each project and were able to eliminate irrelevant projects early on. The total number of projects in the portfolio of each department was reduced by 30% to 45%, Masa reports, so the total number of projects to be sent to the board was considerably smaller than in past years.

To make sure the person was focused on the most important needs of the business, each project was classified as either strategic or progressive during the second and third week of November. The progressive projects remained under the decision-making control of the departments while the strategic projects were elevated to be used in Decision Engine under the belief that collaborative prioritization among the department heads would produce the best overall choices for the company.

Preparation

In the second week of December, each department generated a spreadsheet that included each project name, a one-paragraph description, and one paragraph indicating its benefits and compromises.

What Happened When: The overall project timeline and deliverables.
What Happened When: The overall project timeline and deliverables.

“I used these spreadsheets to prepare the three-round Decision Engine tournament,” said Masa. “I gave a copy of the list to all the managers and the board who were to participate, three days prior to the tournament for them to read and start getting acquainted with all the projects. In hindsight, I should have given them more time, but the schedule didn’t allow it.”

The day prior to the tournament, Masa organized two activities. First, the department managers and the board gathered together for a set of presentations by each department on its proposed projects. Each project was allotted 5-minutes (3 minutes for presentation and 2 minutes for Q&A). Second, everyone participated in a practice session using the online platform, Decision Engine, using dummy data to ensure everyone was comfortable with the platform and the game mechanics. “I wanted them to be able to focus entirely the prioritization activity,” reports Masa.

More Data

Masa also added two new elements to the process to gather even more data. The data analysis done after a Decision Engine forum typically compares the exhaustive data gathered by the online system (chats, bids, purchases etc.). Sometimes, the producers will also assign observers to work with the facilitator to record notes on participant behavior, which is very valuable information that influences the study for better results. In this case, Masa decided to add:

[ordered_list]

  1. Video and audio recordings of the sessions, and
  2. Heuristics based on fundamentals of Bayesian Statistics, to weigh variables taken from game observation such that applying the corresponding algorithm together with the game results would provide a better prioritization.

[/ordered_list]

Show Time

Room preparation began one hour in advance. In addition to Masa, the facilitation team included two volunteers with experience in high collaboration dynamics. One volunteer handled logistics, the other audio/video recording.

PrunetheTree
The team also used the collaboration framework “Prune the Product Tree” to collaboratively prioritize projects in the overall portfolio.

“We placed three session tables so that I could monitor all of them at the same time from a central table where I had 3 computers set up to facilitate the sessions,” detailed Masa. The team also had high-quality video and audio equipment set up to record each table. And they posted Prune the Product Tree posters on one wall, with large sticky notes printed with all the projects titles.

Everyone was on time and when the tournament began, all tables prioritized the first 50% of the projects in around 50 minutes.

“This first session started a bit slow,” Masa relates, “mostly due to discussion about the projects, and fortunately not due to the platform or game mechanics, demonstrating the benefit of the practice session done the previous day.”

The participants had a 15-minute coffee break at the end of the first session, so that Masa could set up the second forum. The participants then prioritized the remaining 50% of the projects in only 40 minutes.

“At that point I had to take the results of both games from all three tables and extract the top 10 projects to run the third session,” Masa reports. “We didn’t waste the time, however. While I set up the next set of forums, the two volunteers facilitated a Prune the Product Tree forum with all the participants to prune the entire project portfolio. I was ready to run the final prioritization session by the time they were done with the trees.”

The last Decision Engine forum took less than 30 minutes to complete, and all participants were able to leave earlier than scheduled. According to Masa, their familiarity with the projects was a huge contributor to more effective and proactive discussions. The discussions were also shorter because they focused on the value of the projects, rather than on understanding them.

The day-of agenda for the in-person prioritization.
The day-of agenda for the in-person prioritization.

Analysis

Masa collected the video and audio recordings, and the Prune the Product Trees (thus pruned!) and returned to his hotel room to begin analysis.

“This was a very involved activity,” said Masa. “I had to listed to every recording very carefully and map the information onto relevant variables to apply my algorithm. This was rather dynamic, since the variables emerged from the observation itself rather than being pre-determined, but this made it more effective.”

“I also added the results of Prune the Product Tree as a variable. Criteria included aspects such as the order in which projects were being discussed and purchased, the level of interest, amount of participation and other for a total of 15 variables.”

Masa reports that the analysis consumed the better part of two days. Once the data mapping was done, he ran the algorithm over the data. “I was very pleased with the results, because with the exception of one project, all were in agreement with what I had learned and observed during the past weeks. There was no bias since I didn’t participate on the games, and the data feeds were based on the observation captured by the cameras, microphones and the Prune the Product Trees.”

Masa used the one project that was in a higher priority than expected as a point of verification by reviewing all the data related to it. He found that the data effectively gave the project higher ranking. He then proceeded then to write the full report.

The results

Masa met with the team who helped him organize the Decision Engine tournament first, and they were amazed by the results and pleased with his explanation. Masa reports, “They were also surprised by the same project that I had surprised me. But they too agreed based on the data that its higher priority was correct.”

“The low esteem, so to speak, towards that project was because it wasn’t a sexy project. So while most people didn’t care for it, it absolutely needed to be done because it had to do with external governance.”

The next step was to present the results to the board. They were very impressed by the quality of the results, the process itself, the fact that the entire process took less than three weeks, the reduced number of projects and the already obvious economic benefit that was taking place.

The department heads and those who participated in the prioritization were also very pleased. The teams that generated the business model canvases and their department’s portfolio, also related to Masa that the experience was fun and helped them truly understand the projects.

“The decision makers said that it was the first time in the history of the company that they truly understood all the projects, and truly collaborated,” said Masa. They even gave higher priority to projects that weren’t their own; whereas in previous years, it was a battle to defend their own projects.

Moral of the story? Using Decision Engine and collaborative prioritization to prioritize their annual project portfolio brought the best out in all of them.


Budget Puzzles Considered Harmful

In my last post about Participatory Budgeting I discussed why surveys suck when used as a tool to understand budget priorities. But there is game-related evolution of surveys, so-called “budget puzzles”, that are even more harmful than surveys because they create intense feeling of despair and harden political opinion. In an era of increasingly partisan politics, budget puzzles are making things worse, not better. What’s especially sad about this is that it appears to be the exact opposite of the goals of the organizations who are promoting budget puzzles. In this post, I’ll elaborate on why budget puzzles are considered harmful and show how collaborative participatory budgeting is the superior approach.

San José Participatory Budgeting.
San José, CA Participatory Budgeting. Messier than a solo puzzle? Yes – and worth it!

Budget Puzzles in Action

I define a budget puzzle as an interactive simulation in which a solo player strives to complete the typically nearly impossible task of balancing a city, state or national budget.

An example of a budget puzzle is the New York Times Budget Puzzle, in which you attempt to balance the national budget by considering various combinations of spending reductions and revenue increases. Spending reductions are grouped in areas such as Domestic Programs and Foreign Aid, Military Spending, Health Care, Social Security, Existing Tax Reforms, while revenue increases (which are always fees or taxes) are identified as modifications or new choices.

Let’s consider three admittedly broad approaches to trying to solve the puzzle: one emphasizing what might be considered stereotypically conservative choices, another more liberal, and third a balanced mix of choices that attempts to affect every area of the budget. In this first pass, I’ll try and keep the choices “moderate” and explore the results. If possible, I recommend that you try the puzzle yourself before reading further.

[table]

[table_row]

[table_cell_head] Conservative [/table_cell_head]

[table_cell_head] Liberal [/table_cell_head]

[table_cell_head] Balanced [/table_cell_head]

[/table_row]

[table_row]

[table_cell_body] As a moderate conservative, I see that capping medicare growth, raising the age for social security, changing how we measure inflation and enacting medical malpractice reform saves about $71B, leaving me $347B over budget.  This has me thinking hard about cuts to military spending, but I don’t make them.[/table_cell_body]

[table_cell_body] As a moderate liberal, I too might raise the age for social security, but I’m going to focus on the military, reducing nuclear arsenals, navy and air force fleets, and troop levels. This saves nets me $102B, leaving me $306B over budget. [/table_cell_body]

[table_cell_body] As a moderate who believes that all areas of the budget must be reduced, I make a few choices in every area. I cut some domestic programs, reduce the size of the federal government, raise social security and medicare eligibility, and so forth. I also considered various tax increases. In my experiment I was able to save $173B – less than half of the $418B I need to save. [/table_cell_body]

[/table_row]

[/table]

[separator type=’transparent’ color=” thickness=” up=” down=”]

Curiously, the moderate approach generated the best results! Ultimately, though, every approach failed. None of the “moderate approaches” was able to get the job done. Now, you can argue that this is OK — that the benefit of interacting with the budget puzzle was to get a sense of the magnitude of the problem and how hard it will be to solve it.

The problem is that most will have the impulse to try again. After all, thanks to video games, we’re used to “failing”, getting a new life, and trying all over again. According to video game designers, this is (always) good! I learned something even though I failed.

I was given the task to balance the budget, so dammit, I’m going to try again. And since I’m a solo player, with no need to justify my ideas or opinions with anyone else, and no requirement to actually think about the feasibility of the choices I’m making, I’m going to solve this puzzle.

Budget Puzzles Harden Political Will

What I learned is that being moderate isn’t going to work. I have to be extreme. Hey, that’s OK, right? It is just a puzzle and I’m not really doing anything that matters because I’m “playing a game”. That makes a bunch of choices easier.

[table]

[table_row]

[table_cell_head] Hardened Conservative [/table_cell_head]

[table_cell_head] Hardened Liberal [/table_cell_head]

[table_cell_head] Balanced [/table_cell_head]

[/table_row]

[table_row]

[table_cell_body] As a hardened conservative, I start by choosing every possible savings associated with both Health care and Social Security. This doesn’t even get me half of the way to my goal, so I choose every possible cut in Domestic Programs and Foreign Aid. Now I’m making real progress! I’m just over half. So, I keep going! I add a National Sales Tax. I don’t fully reach my goal without raising taxes, so I grudgingly accept that I can save $323B by being a great conservative. And if my zeal for solving the puzzle overtakes me I might even raise a few taxes. [/table_cell_body]

[table_cell_body] As a hardened liberal, I start by cutting all of the military programs I can and raising taxes on the rich. Ha! Just this gets me to $316B in saving! I raise a bunch more taxes and let certain taxes expire and I get the magical hit of dopamine that tells me I’ve solved the puzzle. [/table_cell_body]

[table_cell_body] There really no need to try a balanced approach. I just randomly select a bunch of stuff to see which combinations of choices produce the right result, with no genuine investment in the outcome. [/table_cell_body]

[/table_row]

[/table]

 

Of course, all of this work produces an epic #fail: None of these choices could ever be implemented. More importantly, in our political system no one person gets to make these decisions. Solving the Budget Problem requires collaboration, negotiation, listening not just discussion.

After a solo attempt at solving the problem, the player leaves with hardened positions and is almost certainly less willing to engage in the collaborative dialogue and shared actions and compromises that are so desperately needed in today’s political landscape.

Winning the budget puzzle means losing the political process.

Conteneo’s Collaborative Budgeting vs Budget Puzzles

Our approach to Participatory Budgeting is neither a survey or a puzzle. Our approach is real-time, collaborative budgeting in which small groups of five to eight people work together to make choices that impact a budget.

Like our work in San José in 2011, sometimes these choices are not capable of fully balancing a budget in just one year. But, like the collective work done by San José over many years, these choices can create a path to a balanced and sustainable budget.

This table will help you consider the differences between collaborative budgeting, surveys and puzzles. Note that while in many cases the goals are similar, the process of trying to reach these goals can create exactly the opposite of the intended result.

[table]

[table_row]

[table_cell_head] Dimension [/table_cell_head]

[table_cell_head] Collaborative Budgeting [/table_cell_head]

[table_cell_head] Budget Survey [/table_cell_head]

[table_cell_head] Budget Puzzle [/table_cell_head]

[/table_row]

[table_row]

[table_cell_body] Producer Goals?[/table_cell_body]

[table_cell_body] Develop data that elected officials can use to take action. Not just priorities, but the reasons behind the priorities and the conditions of acceptance for proposed actions.[/table_cell_body]

[table_cell_body]Identify priorities of the public. [/table_cell_body]

[table_cell_body] Educate the public.[/table_cell_body]

[/table_row]

[table_row]

[table_cell_body] Participant Goals?[/table_cell_body]

[table_cell_body] Collaborate with other citizens to solve hard problems. [/table_cell_body]

[table_cell_body]Express my priorities. [/table_cell_body]

[table_cell_body] Solve the puzzle.[/table_cell_body]

[/table_row]

[table_row]

[table_cell_body] How do you “win”?[/table_cell_body]

[table_cell_body] You win when you’ve found what organizational theorists refer to as an “equifinal meaning”. This isn’t consensus, per-se, but an agreement on a course of action. [/table_cell_body]

[table_cell_body]Take the survey – I can express myself! [/table_cell_body]

[table_cell_body] Balance the budget.[/table_cell_body]

[/table_row]

[table_row]

[table_cell_body] What is the impact?[/table_cell_body]

[table_cell_body] Considerable! Collaborative Participatory Budgeting directly affects the budget of a city or other governmental institution. [/table_cell_body]

[table_cell_body]Moderate. Survey data helps elected officials make decisions, but method bias leads to less actionable results. [/table_cell_body]

[table_cell_body] None – which means being completely silly is just fine.[/table_cell_body]

[/table_row]

[table_row]

[table_cell_body] Negative Unintended Consequence [/table_cell_body]

[table_cell_body] None. [/table_cell_body]

[table_cell_body] Citizens don’t explore the causal effects of their choices or build relationships with others. [/table_cell_body]

[table_cell_body] As shown in this post, puzzles harden political will and increase negative, partisan politics.[/table_cell_body]

[/table_row]
[/table]

The choice is clear: while surveys and puzzles may be well-intentioned, collaborative, real-time participatory budgeting the superior approach to generating actionable results.

 


Getting Through Awkward to Awesome

By Luke Hohmann

We’re just a few days away from releasing Idea Engine 2.0, the most significant release ever for our platform that supports visual collaboration frameworks like Speed Boat, Prune the Product Tree, and many other agile and Gamestorming games. Our development team has rewritten the entire user interface, and over the next few weeks we’re going to have a lot of fun introducing its many improvements.

The experience of creating Idea Engine 2.0 has me reflecting on a mountain of important management thinking, ranging from Lean Startup and Minimum Viable Products to Value Proposition Design and Crossing the Chasm. And because I’m a dad with two teenage boys and two soon-to-be teenage girls, I’m also reflecting on that awkward-to-awesome stage known as adolescence.

[separator type=’transparent’ color=” thickness=” up=” down=”]

The Early Years

When we first introduced Idea Engine in 2009, it was with one framework, Prune the Product Tree and a very limited set of functionality. So limited that I often referred to Idea Engine 1.0 as the best example ever of a Minimum Viable Product. However, we continued to enhance the product, and with care it grew from infancy to childhood, helping more than 50,000 teams solve problems in new ways.

And while Idea Engine 1.0’s growth was really awesome, Alexander Osterwalder’s brilliant work, Value Proposition Design*, shows us that even when you have a growing product and a product-market fit, changing needs, competitive forces and maturing customers can create new demands. What was once “great” becomes “not good enough”. And if you’re not more than good enough, you’re just not going to cross the chasm.

[separator type=’transparent’ color=” thickness=” up=” down=”]

The Awkward Phase

In the case of Idea Engine 2.0, it was as if my kids’ dentist noticed that their teeth weren’t growing in quite as straight as needed and that corrective action was needed. In a word, Idea Engine needed braces to get back on track.

Idea Engine’s braces weren’t entirely visible. But they were needed, because under the covers, Idea Engine needed a complete overhaul to take advantage of modern Web standards and touch-first devices like iPads. So, bending a few metaphors, and hopefully a few smiles, our Development team went to work, adding braces to Idea Engine 1.0, overhauling and aligning, rewriting, testing and rewriting again.

Next week, we get to take our braces off, and share with you all of the ways in which Idea Engine 2.0 is making us smile. We suspect that once you experience Idea Engine 2.0, you’ll be smiling too!

[separator type=’transparent’ color=” thickness=” up=” down=”]

Launch

You can begin to experience the new Idea Engine next week with our soft launch. We’ll cast off the retainers (Couldn’t resist!) and officially launch Idea Engine 2.0 at the Collaboration Confab on February 19. In the meantime, we will be capturing all of the ways Idea Engine 2.0 makes you and us smile and share them at the launch event.

 


* The Value Proposition Design book (Wiley) includes and recommends using Innovation Games as part of its recommended process.

**Photo credits: Monica Y. Garza, Unsplash.com, Helen Bae