Case Study: How to Take the Pain out Portfolio Prioritization

Ed. Note: This is the story of how Decision Engine and the Collaboration Framework known as “Buy a Feature/Budget Games” transformed the fourth largest telecommunications company in the world. 

MasaMaeda

Masa K. Maeda, CEO of Valueinnova.

In June 2014, Masa K. Maeda, CEO of Valueinnova, Playcamp organizer and Conteneo Certified Collaboration Instructor, began work on an Agile Transformation project at the Ecuadorean office of the fourth largest telecommunication company in the world. As you’d expect, this company has a corporate presence in each country where it offers services and among all of its offices, the Ecuadorian headquarters was considered the most innovative by the senior leadership team.

“The agile transformation began with a very positive impact,” Masa relates, “spreading from 34 people in one department to more than 200 people in eight departments in only six weeks. This happened despite the fact that original contract was for the transformation of just one department.”

“The key to such an accelerated rate of adoption,” Masa continues, “was the ubiquitous introduction and widespread use of high collaboration frameworks (a.k.a. “serious games”) in the teams and at most levels of decision making.”

This initial success gave Valueinnova the opportunity to propose to the general manager that the company use Decision Engine and the collaboration framework “Buy a Feature/Budget Games” to prioritize the company’s 2015 project portfolio, and Valueinnova’s proposal was accepted.

The Backstory

The company’s typical project portfolio prioritization process would begin in October and be complete in December. Each of the company’s twelve departments first prioritized its own project portfolio, which was comprised of 10 to 15 project proposals. The set of twelve prioritized project lists were then handed to a board led by the general manager.

“The board would then go through the painstaking and time-consuming task of merging all those projects to generate one project portfolio of around 140 projects!” Masa relate. “They also preserved the order of projects from each department. No project proposed by any department was rejected, save rare exceptions.”

The issues with the original process were:
[unordered_list style=’circle’ animate=’no’]

  • Resource and time consumption: Many employees and decision makers were involved for too long—they had to give up a good portion of their daily activities during the three month period.
  • Quantity and quality repercussions: Because the company didn’t make hard prioritization choices, they ended up with too many projects, causing some projects to be delivered late due to insufficient resources and other projects to be delivered with poor quality due to cutting corners.
  • Local optimization: Since each department did its own project filtering, the board rarely rejected any projects, resulting in green-lit projects that had little relevance to the company’s bottom line. This localized optimization problem meant that some departments which should have been given more resources to grow faster were starved of their potential.
  • Silo mentality: Each department focused on its own projects without knowledge or interest in the projects from other departments. This is also why the board only merged the departments’ portfolios and did no filtering.
  • Failing economy: All the issues above ultimately had a negative impact on the overall profitability and economic viability of the company.

[/unordered_list]

“By using the collaboration framework “Buy a Feature/Budget Games” and the online prioritization platform Decision Engine, we sought to minimize—and possibly eliminate—those issues,” reported Masa.

In the Beginning

The first step was to ask all 12 departments to create a Business Model Canvas (BMC) for each project that was to become part of its proposed portfolio.the-business-model-canvas-shadow-hero

“There was some hesitation,” Masa said, “because the teams were afraid that this would increase the time needed to create each project portfolio.”

However, creating the Business Model Canvases ended up saving time overall; the act of creating the BMCs collaboratively meant that the teams actually better understood each project and were able to eliminate irrelevant projects early on. The total number of projects in the portfolio of each department was reduced by 30% to 45%, Masa reports, so the total number of projects to be sent to the board was considerably smaller than in past years.

To make sure the person was focused on the most important needs of the business, each project was classified as either strategic or progressive during the second and third week of November. The progressive projects remained under the decision-making control of the departments while the strategic projects were elevated to be used in Decision Engine under the belief that collaborative prioritization among the department heads would produce the best overall choices for the company.

Preparation

In the second week of December, each department generated a spreadsheet that included each project name, a one-paragraph description, and one paragraph indicating its benefits and compromises.

What Happened When: The overall project timeline and deliverables.

What Happened When: The overall project timeline and deliverables.

“I used these spreadsheets to prepare the three-round Decision Engine tournament,” said Masa. “I gave a copy of the list to all the managers and the board who were to participate, three days prior to the tournament for them to read and start getting acquainted with all the projects. In hindsight, I should have given them more time, but the schedule didn’t allow it.”

The day prior to the tournament, Masa organized two activities. First, the department managers and the board gathered together for a set of presentations by each department on its proposed projects. Each project was allotted 5-minutes (3 minutes for presentation and 2 minutes for Q&A). Second, everyone participated in a practice session using the online platform, Decision Engine, using dummy data to ensure everyone was comfortable with the platform and the game mechanics. “I wanted them to be able to focus entirely the prioritization activity,” reports Masa.

More Data

Masa also added two new elements to the process to gather even more data. The data analysis done after a Decision Engine forum typically compares the exhaustive data gathered by the online system (chats, bids, purchases etc.). Sometimes, the producers will also assign observers to work with the facilitator to record notes on participant behavior, which is very valuable information that influences the study for better results. In this case, Masa decided to add:

[ordered_list]

  1. Video and audio recordings of the sessions, and
  2. Heuristics based on fundamentals of Bayesian Statistics, to weigh variables taken from game observation such that applying the corresponding algorithm together with the game results would provide a better prioritization.

[/ordered_list]

Show Time

Room preparation began one hour in advance. In addition to Masa, the facilitation team included two volunteers with experience in high collaboration dynamics. One volunteer handled logistics, the other audio/video recording.

PrunetheTree

The team also used the collaboration framework “Prune the Product Tree” to collaboratively prioritize projects in the overall portfolio.

“We placed three session tables so that I could monitor all of them at the same time from a central table where I had 3 computers set up to facilitate the sessions,” detailed Masa. The team also had high-quality video and audio equipment set up to record each table. And they posted Prune the Product Tree posters on one wall, with large sticky notes printed with all the projects titles.

Everyone was on time and when the tournament began, all tables prioritized the first 50% of the projects in around 50 minutes.

“This first session started a bit slow,” Masa relates, “mostly due to discussion about the projects, and fortunately not due to the platform or game mechanics, demonstrating the benefit of the practice session done the previous day.”

The participants had a 15-minute coffee break at the end of the first session, so that Masa could set up the second forum. The participants then prioritized the remaining 50% of the projects in only 40 minutes.

“At that point I had to take the results of both games from all three tables and extract the top 10 projects to run the third session,” Masa reports. “We didn’t waste the time, however. While I set up the next set of forums, the two volunteers facilitated a Prune the Product Tree forum with all the participants to prune the entire project portfolio. I was ready to run the final prioritization session by the time they were done with the trees.”

The last Decision Engine forum took less than 30 minutes to complete, and all participants were able to leave earlier than scheduled. According to Masa, their familiarity with the projects was a huge contributor to more effective and proactive discussions. The discussions were also shorter because they focused on the value of the projects, rather than on understanding them.

The day-of agenda for the in-person prioritization.

The day-of agenda for the in-person prioritization.

Analysis

Masa collected the video and audio recordings, and the Prune the Product Trees (thus pruned!) and returned to his hotel room to begin analysis.

“This was a very involved activity,” said Masa. “I had to listed to every recording very carefully and map the information onto relevant variables to apply my algorithm. This was rather dynamic, since the variables emerged from the observation itself rather than being pre-determined, but this made it more effective.”

“I also added the results of Prune the Product Tree as a variable. Criteria included aspects such as the order in which projects were being discussed and purchased, the level of interest, amount of participation and other for a total of 15 variables.”

Masa reports that the analysis consumed the better part of two days. Once the data mapping was done, he ran the algorithm over the data. “I was very pleased with the results, because with the exception of one project, all were in agreement with what I had learned and observed during the past weeks. There was no bias since I didn’t participate on the games, and the data feeds were based on the observation captured by the cameras, microphones and the Prune the Product Trees.”

Masa used the one project that was in a higher priority than expected as a point of verification by reviewing all the data related to it. He found that the data effectively gave the project higher ranking. He then proceeded then to write the full report.

The results

Masa met with the team who helped him organize the Decision Engine tournament first, and they were amazed by the results and pleased with his explanation. Masa reports, “They were also surprised by the same project that I had surprised me. But they too agreed based on the data that its higher priority was correct.”

“The low esteem, so to speak, towards that project was because it wasn’t a sexy project. So while most people didn’t care for it, it absolutely needed to be done because it had to do with external governance.”

The next step was to present the results to the board. They were very impressed by the quality of the results, the process itself, the fact that the entire process took less than three weeks, the reduced number of projects and the already obvious economic benefit that was taking place.

The department heads and those who participated in the prioritization were also very pleased. The teams that generated the business model canvases and their department’s portfolio, also related to Masa that the experience was fun and helped them truly understand the projects.

“The decision makers said that it was the first time in the history of the company that they truly understood all the projects, and truly collaborated,” said Masa. They even gave higher priority to projects that weren’t their own; whereas in previous years, it was a battle to defend their own projects.

Moral of the story? Using Decision Engine and collaborative prioritization to prioritize their annual project portfolio brought the best out in all of them.


Budgets Are Not Broccoli (or, Why Surveys Suck)

In my last post on the San José Budget Games I promised a brief overview of why Surveys SUCK and why collaborative games are better. This expands on my Agile 2015 keynote and will help the Participatory Budgeting community in creating even more participation for our sessions.

Let’s start with a story. I was in Paris this week, teaching a master class on serious game design and attending Playcamp Paris. Produced with our partner, Raphael Goumot, a Black Belt CCA and owner of creagile.fr, we spent the entire evening discussing Participatory Budgeting, our Budget Games and our Budget Investor.

Through an amazing coincidence, Tarang Patel (an Agile specialist from Adobe and another Black Belt CCA) was in Paris. Tarang has personally facilitated San José Budget Games sessions in 2013 and 2014 and told this story:

[blockquote width=’100′ mark=’grey’]

What is amazing about collaborative prioritization, in which citizens have to work together to purchase projects, is how collaboration can change the minds of the other participants. In 2014 our table was not interested in funding any items. And yet, near the end of the session, a girl from the youth commission who strongly believed in addressing what she felt was an increasing problem homelessness, invested a substantial sum of her money on a project designed to address homelessness. The other participants at her table really took notice and started to explore the issue. By the time they were finished, the table had changed their minds and joined together to fund this project.”

[/blockquote]

This story is not unique. As near as we can tell, every single collaborative forum changes the opinions of the participants. Here is another story, this time from Laura Richardson:

[blockquote width=’100′ mark=’grey’]

At our table one woman from an affluent neighborhood started the negotiations by purchasing code enforcement. She wanted to make sure the City was looking as good as possible. She changed her mind, however, when a mother from a less affluent part of the city described the dangers her children faced from gang violence — just walking to school wearing the wrong color jacket could leave her child harassed, beaten, or even worse. The emotional impact was visible because both woman were crying by the time the ‘negotiations’ were finished. The table quickly aligned on purchasing projects to designed to address gang violence.”

[/blockquote]

We see similar outcomes in our platforms when used for prioritizing features in product roadmaps and project portfolios inside corporations: collaboration provides a forum for understanding complex issues and making better choices. Here is a story from my own experience in helping VeriSign prioritize a set of potential customer projects:

[blockquote width=’100′ mark=’grey’]

One of the projects under consideration was a self-service support website, in which customers could form a community to post questions, share best practices and help each other configure and use VeriSign products. In every forum junior customer service agents would purchase this project because it promised to make their lives easier. And in every forum a more senior and experienced agent would gently explain that this project would actually increase the amount of work because every comment by a forum member would have to be reviewed and verified by a VeriSign employee. The reasoning was simple: VeriSign provides complex, sophisticated security solutions. Hackers and other nefarious types could provide incorrect advice to make it easier to exploit a website.”

[/blockquote]

But Why Broccoli?

Let’s compare these stories about shared prioritization of a budget with broccoli. I happen to like broccoli. A lot of people don’t. And no matter how much time I spend trying to convince you that you should like broccoli, I’m not going to change your mind. And no matter how much time you spend trying to convince me that I shouldn’t like broccoli, you’re not going to change my mind.

Broccoli isn’t a budget. At one level, it doesn’t actually matter if you like broccoli, or you don’t like bananas. Conversations on these topics aren’t going to change your perspective and they certainly aren’t going to affect your job or your city.

[blockquote width=’100′ mark=’grey’]

This is why surveys suck when you’re dealing with complex issues like corporate or city budgets. The results of the survey are subject to change once you start talking about the items with other people.

[/blockquote]

This change is what draws people together instead of pushing them apart. In my next post, I’ll outline how surveys evolve from being something that just plain sucks to being something that is downright harmful.


Collaboration Framework Spotlight: Product Box to Science Fair.

Necessity may be the mother of invention, but fexibility is the progenitor of new collaboration frameworks, especially in in-person forums. Even with our long experience with live events and logistics, occasionally things go awry. Shipments are late or supplies we thought we definitely needed are sold out and unavailable. When faced with logistical snafus, Collaboration Architects excel as transforming problems into solutions.

Product Box to Science Fair

Protegra’s Terry Bunio transformed Product Box to Science Fair by using Tri-fold posters.

Protegra’s Terry Bunio writes on the Protegra company blog about just such an issue and how he was able to transform missing Product Box supplies into a new collaboration framework he calls, “Science Fair,” for an Agile Winnipeg User Group meeting.

Science Fair

The birth of Science Fair came from missing supplies (and the creative use of others.) For Product Box, we use white literature mailers, commonly found in most office supply stores in the U.S. Outside of the 50 states, however, we have to be flexible as they are often out of stock. As Terry writes, “I was hoping that the lack of white cereal-sized boxes was only temporary at Staples. Nope. They were nowhere to be found.”

Staples did have a selection of tri-fold display boards, the kind used by kids all over the world in science fairs. And Terry found a large selection of “animal stickers.” Using the supplies that were available, Terry had his teams create science fair “posters,” instead of Product Boxes, selling their bosses on the value of Agile.

product box takeout

Restaurant Takeout boxes are another common substitution when the typical white boxes can’t be found.

Terry writes that the new framework garned some unexpected benefits, including:
[unordered_list style=’circle’ animate=’no’]

  • More real estate or physical space for buiding out your argument.
  • Additional metaphor for talking about the problems, created through the use of animal stickers and the science fair concept.

[/unordered_list]

Have you had to adapt or change or create a new Innovation Game collaboration framework out of necessity? We’d love to hear about it.