Measurement
This page gives an overview of measurement and review methods used for the evaluation of a knowledge management programme. It covers:
- AAR (After Action Review) - a project or project milestone review process
- From Baseline to Benchmarking - assessing your overall KM progress against your initial baseline
- Benefits Realization - evaluating if your achieved the benefits anticipated
- Intellectual Capital Measures - for those who want to take measurement to a new strategic level.
AAR (After Action Review)
Every project management methodology includes a process for reviewing outcomes. A specific KM approach is that of the After Action Review, first developed by the US Army. It is a structured question and answer session which addresses the following four questions:
- What should have happened?
- What actually happened?
- Why were there differences (from plan to actual)?
- What lessons can we take forward to next time?
An AAR session should involve all key players and take place in a non-threatening environment. Thus, if people admit to mistakes on their part, it should not be held against them, but treated as a learning experience. I usualy find that these take between 1.5 and 2 hours, and I also find it useful to pre-send a list of questions we cover, have written answers from those who cannot attend, and also record the conversations so that nothing is filtered out when writing it up.
You'll find a sample of the question list (PDF) that can be customized for a given situation.
From Baseline to Benchmarking
The knowledge audit and KM assessment you carried out earlier act as a baseline to check progress. At the time you may have set yourself some target results for each of the dimensions of the KM Quick Assessment. By using the same measures year-on-year you get a solid picture of progress. You could also try, more informally, where your people feel you are on the KM maturity curve.
But you can take benchmarking further by evaluating your KM performance against that of external organizations. However, unless they are using the same assessment method direct comparisons are difficult. What you can do, though, is to informally share your experiences and best practices. This is often done through professional KM networks and is particular valuable for organizations that do not directly compete with each other.
Benefits Realization
When you made your business case, you suggested benefits that could be achieved through KM. Now is the time to assess whether those benefits have been realized. By their very nature many of the benefits are not easily quantifiable, so you will probably need a short questionnaire asking respondents to rate achievement on a scale of 1-5 (say), or bring in an external objective assessor. A useful exercise would be to get key people from business units in your KM network to a workshop to revisit the benefits tree you (may have) produced at the start of your initiative.
In a survey by KPMG in 2000, respondents gave the top reason that benefits were not realized as follows:
- Poor communications (20% of respondents)
- Not part of everyday practice (19%)
- Lack of time / too complex (18%)
- Lack of training (15%)
- Users not see personal benefit (i.e. the "what's in it for me" syndrome) (13%)
- Unsupportive senior management (8%)
- Technical problems (7%).
Although we are unaware of more recent surveys, these figures reflect our own experiences over the last decade or so, where reasons 1 and 3 feature the most. In general, you will find that some benefits have not materialized as expected, but in contrast there may be unexpected benefits. Typical of these are better working relationships and understanding between people in different departments, and a greater awareness of by individuals of how they can become more effective through managing their own information and knowledge better.
Intellectual Capital Measures
measurement is recognized as one of the trickiest areas of knowledge management. However, "you can only manage what you can measure" is the adage to go by. Quite a few organizations are using measures of intellectual capital to supplement traditional financial measures. This is a vast subject, and one for which the author wrote a 525-page report in 1999!
Although that report reviewed over 30 different IC measurement methods, one that he particularly likes is the Danish matrix. Here is the author's version of it, with (in brackets) the author's terminology for the different categories of measure.
| What Is (Resources) |
What Is Done (Recipes) |
What Happens (Results) |
|
|---|---|---|---|
| Human Resources | |||
| Customers | |||
| Processes | |||
| Technology |
You will need to create a few key indicators for each cell, and devise a way of collecting the data. As with most measurement systems, the output is often less important than the discussion that goes into it. We often hear of 'ah ha' moments as discussion on the indicators leads those involved to get a better understanding of the role of knowledge in their organization's performance.
On a Practical Note
If you try to do all the types of evaluation mentioned above at once, you will find two things. First, it will take more of your time and resources to carry out than you had planned for. Secondly, staff may become irritated by demands on them for input, e.g. completing questionnaires, attending review meetings. Therefore, make sure you pick the most relevant things to do. Consider alternating different approaches or different stakeholder representatives at different times. Also, make good use of your KM champions throughout your organization.
Last updated: 19th March 2011
