Introducing the OKR ScoreCard

So let’s assume we’re using OKRs now. We’ve come up with some Objectives and tried to find meaningful measures for our Key Results that we think will help us ensure we’re going in the right direction.

It’s important to remind ourselves that OKRs are just a tool. Using an ORK framework doesn’t necessarily ensure that we’re getting any tangible benefits over any other approach.

For example, the next two OKR examples for our imaginary TownTalk ™ product both provide an objective and measurable KRs against it, but do both really unlock the same value?

OKR 1

Objective: Deliver against the TownTalk Product Requirements Document

KRs:

– Go from 0 to 100 mandatory features complete

– Go from 0 to 20 desired features complete

– Package is released through Cognidox

OKR 2

Objective: TownTalk makes highstreets more vibrant and social than they has ever been, so that businesses and people are healthier.

KRs:

– TownTalk users highstreet visits go from 1 to 10 per month

– TownTalk users increase from 100 to 1000

– TownTalk users report a ‘community interaction’ score of over 8/10

I hope it’s pretty clear that OKR 1 has a lot to be desired (and if it doesn’t make you want to keel over with boredom, I don’t know what will). So much so that one might wonder what value an OKR framework really brings.

However, I have observed many more OKR 1 style OKRs in my career than OKR 2 style. I suspect there are many reasons for this, but one of the most compelling might be just about how much effort needs to go into the OKR creation in the first place. I created both of these OKRs just from the top of my head and I can honestly say that OKR 1 took me about 2 minutes to create whereas OKR 2 took about 20 minutes with several refinements – and that was without the constraints of reality or any consideration/agreement around other contributors ideas.

If OKRs are not treated as first class citizens, or being given the priority, time and focus required to develop and review them, they can easily become misused and stop providing the benefits you’re looking for.

Does this all mean that OKR 1 is of zero value? Do all OKR attempts fall into categories of “bad” or “good”?
I’d suggest not. There are several ways in which OKRs can provide value, and the more of those ways are enabled, the better. Having said that, sometimes we’re constrained by what we can measure, or we’re on a journey as a company and don’t want to overwhelm teams with a sudden, drastic change.

Wouldn’t it be nice if we had some sort of scoring mechanism for how “mature” an OKR is?

Put your OKRs on a maturity scale

While I was running an OKRs CoP at work, we identified that being able to ‘score’ OKRs on a scale would allow us to try and make ours better over time. There didn’t seem to be anything on the internet that provided what we wanted, so I came up with something the group could review and refine.

This is how it works:

Objective

For this Objective, out of ten, give a score for each of the following and then sum together:

  • Is inspiring to me
  • Represents an important impact in the world
  • I understand why

For this Objective, apply the multiplier in brackets next to the answer you think is most accurate:

  • Represents value for the business
    1. No Value (0)
    2. Indirect Monetary Value (1)
    3. Direct Monetary Value (2)
  • Defines
    1. What (0)
    2. How (0)
    3. Where (1)

Key Result

For this Key Result, out of ten, give a score for each of the following and then sum together:

  • Is “realised” as opposed to “delivered”
  • I would consider reaching this as a significant aspect of the objective complete

For this Key Result, apply the multiplier in brackets next to the answer you think is most accurate:

  • Is numerical
    • Yes (2)
    • No (1)
  • Measurability
    • Is measurable now (2)
    • Could be measured with some work (1)
    • Can not be measured directly (0)
  • Represents a change in customer behaviour
    • Yes (2)
    • No (1)
  • Likely distribution of movement over OKR lifetime
    • Even distribution (2)
    • Front or back loaded (0.5)
    • Single event (0)
    • After lifetime (0)
  • We (the group working towards the OKR) can think of more than one set of features or deliverables that could move the needle on this KR
    • Yes (1)
    • No (0)

Worked example

Purely for illustrative purposes, I’ll run through the application of the scorecard across the two OKR examples earlier in this post using my own subjective scoring. In the workplace, this might be better done collaboratively using dot-voting or similar & taking an average.

OKR 1

Objective: Deliver against the TownTalk Product Requirements Document

I struggle to see how anyone could find this objective inspiring, so that’s a solid zero.

I scored a 1/10 for ‘important impact to the world’, which is pretty generous – it’s not describing an important impact.

I don’t understand ‘why’ from the OKR statement at all, so that gets a zero.

Moving on to the multipliers, I chose ‘Direct monetary value’ which is, again, generous and assuming that we have customers lined up for this thing.

Finally the statement itself does not say ‘what’ should be created or ‘how’. I suspect that the PRD says what and, possibly, how… but let’s be ultra generous and interpret it as saying we want to get to a place ‘where’ our product is delivered.

This makes our grand total Objective score 2/60. Ouch.

I’m lumping the first two KRs together as they’re fundamentally the same.


KR – Go from 0 to 100 mandatory features complete &

KR – Go from 0 to 20 desired features complete

Completed features are absolutely ‘delivered’, not ‘realised’. So this gets a zero.

These KRs would represent a significant aspect of the objective (terrible though it is) complete, so full scores here 10/10.

The KRs are numerical, full scores there. I’ll assume this feature factory has it’s feature delivery process down and there’s little to no work to measure features delivered, full scores here again.

These KRs do not represent a change in customer behaviour, but fortunately this isn’t a zero multiplier, so we’re safe.

I’ll assume the delivery of features over time is likely to be fairly consistent, not front or back loaded and certainly not a one time event, or after the lifetime of the OKR, so another good score.

Here’s where things fall over though. Can we think of more than one set of features or deliverables that could move the needle here? Of course not as it’s all pre-proscribed. This means we get a zero multiplier and our KRs are dead. Ouch again.

KR – Package is released through Cognidox

I won’t go through all the reasoning for this one, suffice to say that it’s a one time event and that’s another KR killer in this score sheet.

OKR 2

Objective: TownTalk makes highstreets more vibrant and social than they have ever been, so that businesses and people are healthier.

I find this pretty inspiring, given the years-long decline of the highstreet. TownTalk wants to innovate to reverse this trend and positively impact businesses and peoples mental health. 👍
I think this absolutely represents an important impact in the world. 7/10

The objective wording explains ‘why’ in the “so that” component. So I understand why pretty well. 9/10

For the multipliers – if TownTalk is making businesses and people healthier and there’s a business model behind that, then I’d argue the objective represents direct monetary value.

The objective does not describe what to build or how to build it – it does describe where we want to be, so full marks there too.

This gives us a total score for our Objective of 48/60!

KR -TownTalk users highstreet visits go from 1 to 10 per month

You ‘realise’ an increase in highstreet visits, you certainly don’t deliver them. 10/10

Do increasing user visits to the highstreet represent a significant aspect of making the highstreet more vibrant and social? Absolutely yes. This could be 10/10 but I’ll thrown caution to the wind and give an 8.

It’s numerical, we’ll assume it’s measurable as the business model must rely on knowing, it is a change in customer behaviour, it’s probably going to be distributed fairly well and there is huge scope for features and deliverables that can move the needle.

This KR scores a healthy 144/160!

KR – TownTalk users increase from 100 to 1000

You ‘realise’ an increase in users. 10/10

Although I think increased users do represent a significant aspect of the objective complete, the numbers are nothing close to what we’d need to reach our objective… Perhaps we’re playing a bit safe? 2/10

It’s numerical, we’ll assume it’s measurable as the business model must rely on knowing, it is a change in customer behaviour, it’s probably going to be distributed fairly well and there is huge scope for features and deliverables that can move the needle.

This KR scores a reasonable 96/160!

KR – TownTalk users report a ‘community interaction’ score of over 8/10

It’s ‘realised’, it’s a significant aspect of our objective, 10/10 for both.

It’s numerical, we’ll assume it might take some work to measure it as we don’t have a “current” value, it is a change in customer behaviour, it’s probably going to be distributed fairly well and there is huge scope for features and deliverables that can move the needle.

So we have 80/160 on this one. It seems low for a KR that nailed both the top two questions. Perhaps the “could be measured with some work” multiplier is having an unreasonably large impact.

Improvements moving forward

This is my first attempt at an OKR scorecard and I hope there’s some value in the idea at least. I’m pretty sure the execution can be improved. For example:

  • Should the concept of “monetary value” be removed from objective scoring?

If we recall the INTRAC results chain and how that relates to Objectives (see “Focus on outcomes with OKRs” blog post) then we’re talking impacts. If the impact of some medical treatment we’re developing is that fewer people die in some medical intervention – should we mark that impact down because it doesn’t represent direct monetary value to the business?

  • Combined KRs cover all important aspects of the objective?

We currently only consider each KR in isolation, perhaps there’s value in scoring all KRs for any particular Objective as a group. For example, perhaps there’s a question of whether the combined KRs are missing something fundamental that might take the team off course.

  • Qualify each question and how it relates to the benefits of using OKRs

What benefit are we trying to assess with each question? OKRs provide several benefits including empowerment, alignment, measuring progress and a focus on value amongst others. I think one of the next things to do on this score sheet is to represent how each question being asked relates to these values.

Some OKR attempts may do very well at measuring progress and alignment, but completely lack empowerment and value focus. It would be useful to see where our attempt falls short.

  • Review why multipliers are multipliers

Initially, there was good reason for this. I wanted a way to completely zero out the OKR score if the structure removed one of the fundamental values of using OKRs. E.g. Having and objective of “build this thing” giving no empowerment, or a KR that represented a single event and therefor provided no progress indication.

However, multipliers have huge impacts across the board and, as with the last KR we looked at, halving the possible score because it took some work to make measurements seems overly harsh.

The sheet

Finally, here’s the sheet, if you feel like using it to do your own scores.

Thanks for reading!