Why self-assessment is (probably) fundamentally flawed (part two)

This is part two of my case against the use of self-assessment, as a reliable exclusive means of measuring staff performance.

The Latin maxim “ignoramus et ignorabimus”, meaning "we do not know and will not know", stood for a position on the limits of scientific knowledge in the nineteenth century.

In September 1930, mathematician David Hilbert pronounced his disagreement in a celebrated address to the Society of German Scientists and Physicians, in Königsberg:

“We must not believe those, who today, with philosophical bearing and deliberative tone, prophesy the fall of culture and accept the ignorabimus. For us there is no ignorabimus, and in my opinion none whatever in natural science. In opposition to the foolish ignorabimus our slogan shall be: We must know - we will know!”

In the late twentieth century, Ex-United States Secretary of Defense, Donald Rumsfeld, whilst defending his country's position on the Iraq war, made the following (now infamous) statement; “There are ‘known knowns’. These are things we know that we know. There are ‘known unknowns’. That is to say, there are things that we now know we don’t know. But there are also ‘unknown unknowns’. These are things we do not know we don’t know”.
 
There are four recognised stages of competence.

1) Unconscious Incompetence
The individual neither understands nor knows how to do something, nor recognizes the deficit, nor has a desire to address it. The person must become conscious of their incompetence before development of the new skill or learning can begin.

2) Conscious Incompetence
Though the individual does not understand or know how to do something, he or she does recognize the deficit, without yet addressing it.

3) Conscious Competence
The individual understands or knows how to do something. Demonstrating the skill or knowledge requires a great deal of consciousness or concentration.

4) Unconscious Competence
The individual has had so much practice with a skill that it becomes “second nature” and can be performed easily. He or she may or may not be able to teach it to others, depending upon how and when it was learned.

"The Invisible Gorilla" experiment is one of the most famous psychological demo's in modern history. Subjects are shown a video, about a minute long, of two teams, one in white shirts, the other in black shirts, moving around and passing basketballs to one another. They are asked to count the number of aerial and bounce passes made by the team wearing white, a seemingly simple task. Halfway through the video, a woman wearing a full-body gorilla suit walks slowly to the middle of the screen, pounds her chest, and then walks out of the frame. If you are just watching the video, it’s the most obvious thing in the world. But when asked to count the passes, about half the people miss it. It is as though the gorilla is completely invisible.
(http://www.theinvisiblegorilla.com/gorilla_experiment.html).

In his popular KM blog, Nick Milton (http://www.knoco.co.uk/Nick-Milton.htm) writes in detail about the impact of this experiment and picks up on a number of key trends discussed in the book of the same name, authored by Christopher Chabris and Daniel Simons (the guys behind the original experiment).

The subtitle of the book is "ways our intuition deceives us", and the authors talk about a number of human traits (they call them illusions) which we need to be aware of in Knowledge Management, as each of them can affect the reliability and effectiveness of Knowledge Transfer.

To paraphrase Milton, the illusions which have most impact on Knowledge Management are;

• The illusion of memory
• The illusion of confidence
• The illusion of knowledge

Our memory of events fades over time, to the point that even firm documentary evidence to the contrary doesn't change what we remember. The implication is that if you will need to re-use tacit knowledge in the future, then you can't rely on people to remember it. Even after a month, the memory will be unreliable. Details will have been added, details will have been forgotten, the facts will have been rewritten to be closer to "what feels right".

Tacit knowledge is fine for sharing knowledge on what's happening now, but for sharing knowledge with people in the future then it needs to be written down quickly while memory is still reliable.

Without a written or photographic record, the tacit memory fades quickly, often retaining enough knowledge to be dangerous, but not enough to be successful. And as the authors say, the illusion of memory can be so strong that the written on photographic record can come as a shock, and can feel wrong, even if it’s right.

Any approach that relies solely on tacit knowledge held in the human memory can therefore be very risky, thanks to the illusion of memory.

The illusion of confidence represents the way that people value knowledge from a confident person. This would be fine if confidence and knowledge go hand in hand, but in fact there is almost an inverse relationship. A lack of knowledge is, instead, allied to overconfidence. Lack of knowledge leads to confidence, which leads to you being seen as knowledgeable.

Each chess player is given a points rating based on their competition results, which is in fact a very effective and reliable measure of their ability. Yet 75% of chess players believe they are underrated, despite the evidence to the contrary. They are overconfident in their own ability.

In studies of groups of people coming together to solve a maths problem, you would expect the group to defer to the person with the greatest maths knowledge, wouldn't you? In fact, the group deferred to the most confident person, regardless of their knowledge. In trials, in 94% of the cases, the final answer given by the group is the first answer suggested, by the most confident person present, regardless if whether it is right or wrong.

In a Harvard study of confidence vs knowledge in a trivia test, they certainly saw overconfidence in individuals - people were confident of their answer 70% of the time, while being correct only 54% of the time! When people were put together in pairs, the counterintuitive outcome was that the pairs were no more successful than the individuals, but they were a lot more confident! When two low-confidence people were put together, their overall confidence increased by 11%, even though their success rate was no higher than before.

The Illusion of Knowledge is behind the way we overestimate how much we know. The authors refer to how people think they know how long a project will take, and how much it will cost, despite the fact that projects almost always overrun in both cost and time. "We all experience this sort of illusory knowledge, even for the simplest projects" they write. "We underestimate how long they will take or how much they will cost, because what seems simple and straightforward in our mind typically turns out to be more complex when our plans encounter reality. The problem is that we never take this limitation into account. Over and over, the illusion of knowledge convinces us that we have a deep understanding of what a project will entail, when all we really have is a rough and optimistic guess based on shallow familiarity"

"To avoid this illusion of knowledge, start by admitting that your personal views of how expensive and time-consuming your own seemingly unique project will be are probably wrong. If instead, you seek out similar projects that others have completed, you can use the actual time and cost of these projects to understand how long yours will take. Taking such an outside view of what we normally keep in our own minds dramatically changes how we see our plans"

If we are unaware of these 3 illusions, we can feel confident in our knowledge, based on our memories of the past, without realising that the confidence is false, the knowledge is poor, and the memories are unreliable and partially fictitious. Awareness of these illusions allows us also to challenge the individual who confidently declares "I know how to do this. I remember how we did it 5 years ago", because we recognise the shaky nature of confidence, knowledge and memory.

A natural human tendency is that we tend to think that we know more than we do and that
we are better than we are. We suffer from what psychologists call the “Lake Wobegon effect”.
Based on Garrison Keillor’s fictional town where “all the women are strong, all the men are good-looking and all the children are above average.” According to the author’s own survey, 63% of Americans consider themselves more intelligent than the average American.

In contrast, 70% of Canadians said they considered themselves smarter than the average Canadian. In a survey of engineers 42% thought their work ranked in the top 5% among their peers. A survey of college professors revealed that 94% thought they do ‘‘above average’’ work – a figure that defies mathematical plausibility! A survey of sales people found that the average self-assessment score (for sales demos) was 76%. The average % of demo's that achieved objectives (for the same group) was 57%. The list goes on..

So, in summary, any strategy for capturing user skills data, which relies solely on an individual's ability to self-rate themselves on a given subject, is simply doomed to fail. I leave the last word to David Dunning; “In essence, our incompetence masks our ability to recognize our incompetence”.

R

Why self-assessment is (probably) fundamentally flawed (part one)

A while back, we wrote a paper about the pro's and cons of using self assessment, as an exclusive means of gaining useful corporate intelligence and capturing management metrics for staff performance.

Over the years, many AEC firms have confidently stated, 'We don't need independent skills testing, we already know how good our teams are'. When one enquires further, what they actually mean, is that they sent out a user survey, asking staff to rate themselves (usually out of 5) on a range of different skills topics, including AutoCAD, Revit, BIM, etc. What they end up with is a spreadsheet (why is it always a spreadsheet?) with a list of names down one side, a list of skills categories across the top - and a sheet filled with 3's and 4's. Why 3's and 4's, I hear you ask? Simply because people don't want to risk the personal penalties that might go along with admitting they're a 1 or a 2. And conversely, they don't want to stick their head above the parapet by admitting to a 5 (even if they are a 5) because this can cause all sorts of new issues (more work, more people pestering them for answers to the same questions, you get the picture). So it's 3's and 4's all the way.

Congratulations XYZ Engineers, you have your completed spreadsheet, so you are now totally self-aware, as an organization. (Not really). And the real rub here is that, more often than not, people have no clue how good they are, relative to the rest of the team, or wider industry!

So, we decided to explore this concept in greater detail. Here's the evidence..

Let us begin with a story. A few years back, NY Times Online posted a series of articles by filmmaker Errrol Morris. He tells the tale of Bank robbery suspect McArthur Wheeler, who was recognized by informants who tipped detectives to his whereabouts after his picture was telecast one Wednesday night, during the Pittsburgh Crime Stoppers segment of the 11 o’clock news. At 12:10 am, less than an hour after the broadcast, he was arrested. Wheeler had walked into two Pittsburgh banks and attempted to rob them in broad daylight.

What made the case peculiar is that he made no visible attempt at disguise. The surveillance tapes were key to his arrest. There he is with a gun, standing in front of a teller demanding money. Yet, when arrested, Wheeler was completely disbelieving. “But I wore the juice,” he said. Apparently, he was under the deeply misguided impression that rubbing one’s face with lemon juice rendered it invisible to video cameras.

Pittsburgh police detectives who had been involved in Wheeler’s arrest explained that Wheeler had not gone into “this thing” blindly but had performed a variety of tests prior to the robbery. Although Wheeler reported the lemon juice was burning his face and his eyes, and he was having trouble (seeing) and had to squint, he had tested the theory, and it seemed to work. He had snapped a Polaroid picture of himself and wasn't anywhere to be found in the image.

There are three possibilities:
(a) the film was bad;
(b) Wheeler hadn’t adjusted the camera correctly; or
(c) Wheeler had pointed the camera away from his face at the critical moment when he snapped the photo

Pittsburgh Police concluded that, 'If Wheeler was too stupid to be a bank robber, perhaps he was also too stupid to know that he was too stupid to be a bank robber - that is, his stupidity protected him from an awareness of his own stupidity.'

Now, this sorry tale might have been just another footnote in history, were it not for the fact that it came to the attention of David Dunning, a Cornell professor of social psychology. After reading this story in 1996, Dunning wondered whether it was possible to measure one’s self-assessed level of competence against something a little more objective – say, actual competence.

Over the next 3 years, Dunning (assisted by colleague Justin Kruger) undertook a major academic study and, in 1999, published the paper, “Unskilled and Unaware of It: How Difficulties of Recognizing One’s Own Incompetence Lead to Inflated Self-assessments”.

Dunning’s epiphany was; “When people are incompetent in the strategies they adopt to achieve success and satisfaction, they suffer a dual burden; not only do they reach erroneous conclusions and make unfortunate choices, but their incompetence robs them of the ability to realize it. Instead, like Mr. Wheeler, they are left with the erroneous impression they are doing just fine. In essence, our incompetence masks our ability to recognize our incompetence”.

Dunning & Kruger also quote the “above-average effect”, or the tendency of the average person to believe he or she is above average, a result that defies the logic of statistics. Participants scoring in the bottom quartile on tests grossly overestimated their performance and ability. Although test scores put them in the 12th percentile they estimated themselves to be in the 62nd.

Conversely, Because top performers find the tests they confront to be easy, they mistakenly assume that their peers find the tests to be equally easy. As such, their own performances seem unexceptional. In studies, the top 25% tended to think that their skills lay in the 70th–75th percentile, although their performances fell roughly in the 87th percentile.

Dunning and Kruger proposed that, for a given skill, incompetent people will:

  tend to overestimate their own level of skill;
  fail to recognize genuine skill in others;
  fail to recognize the extremity of their inadequacy;
  recognize and acknowledge their own previous lack of skill, if they can be trained to substantially improve.

As a follow up study, “Why the unskilled are unaware: Further explorations of (absent) self-insight among the incompetent”, was published in 2006. (David Dunning, Justin Kruger, Joyce Ehrlingera, Kerri Johnson, Matthew Banner).

In part two, we'll take a look at the 4 stages of competence - and how the combined illusions of memory, confidence and knowledge can impact on a firms' knowledge management strategy.

R

Structured Knowledge Management Using Skills Assessments - White Paper

The team at A 2 K, led by ANZ Consulting Manager, Sean Twomey, have put together an excellent, thought-provoking paper which examines knowledge assessment and the benefits and process of using skills assessment for benchmarking in the field of design technology.

The paper has a particular focus on the CAD/BIM sides of design and engineering companies, but the process applies to all facets of business. The examples referred to in this paper are based on
Autodesk environments and processes (e.g. CAD and BIM), but are also applicable to Adobe, Graphisoft,
Bentley and similar packages.

In his detailed analysis, Twomey looks at the process of analysing and harnessing assessment data to calibrate goals, to build major improvements and measure progress, with one of the most practical and visible benefits of knowledge management being the ability to facilitate decision-making capabilities. He also considers the challenge of overcoming the innate fear of assessments.

Here's a link to download the white paper.

R

Daisy-chain

This month, we are introducing a change to the way assessments are authored, based on the idea of linking separate modules in a 'daisy-chain', to create a longer test.

In essence, the KS system breaks down like this;

- A test is one or more modules linked together.
- A module is comprised of typically 4-6 questions, covering the same topic, i.e. Families in Revit or UCS in AutoCAD, etc.
- A question tests an individual's understanding of a particular software feature or command.

Questions and modules have training tag keywords (meta tags) associated with them, which makes training needs analysis, post-assessment, much more relevant.

Questions can be "task" based or "knowledge" based.

A task based question has files associated with it. The user has to do something to the file, in order to work out the answer, as follows;

- Task (interrogate) - the user has to open the file and find the answer, but they don't have to modify or change the file in any way.
- Task (modify) - the user has to open and modify the file, to work out the correct answer.
- Knowledge - this type of question does not require a file to be associated with it.

Test administrators can present a single module as a test, provided they create a new test from the module first.

A 'daisy-chain' happens when an administrator links a series of modules together, to create a longer test.

This concept allows us to create a wide range of test material, in bite-sized blocks, which map closely to corresponding training and learning material.

Daisy-chains go live next month.

R

KA Connect - a new kind of Tribe

Just back from an enjoyable trip to Chicago, where we sponsored and presented at the inaugural KA Connect, knowledge management conference.

Thanks to Chris P and the team at Knowledge Architecture for hosting what was the most enjoyable and useful conference I've been to in a long time.

What set KA Connect apart from most events was the format. Taking Seth Godin's theme of creating a Tribe (http://www.sethgodin.com/) and combining it with Derek Sivers' 'Shirtless Dancing Guy' discussion on TED.com about how to start a movement;(http://www.ted.com/talks/derek_sivers_how_to_start_a_movement.html?awesm=on.ted.com_8EPn).

The event, over 2 days, zipped along at a fair old pace. With 36 presenters, covering a range of KM angles, there was a smorgasbord of good ideas.

Particular highlights for me were;

- John Moebes from Crate & Barrel, discussing a new process for design-bid-build. This is the future of construction clients.

- Justin Quimby from Quimby Heavy Industries, with insights on how the AEC industry can learn lessons from the gaming sector.

- Michael Kilkelly from Gehry Partners, showcasing the way model-making impacts on their high-end designs (plus a bit of AutoCAD :) )

- Darren Rizza from Satellier, on why it's OK to whack a sacred cow with a stick once in a while!

- Pete Walsham from Axomic, comparing most architects to peasants (from a digital asset management pov).

- John Soter from ZweigWhite, with some parables about monkeys.

- Laurie Dreyer from Anshen + Allen, who brought the house down with her 'dinosaur lessons' and hilarious take on life from a HR & Learning professional's perspective.

Chicago was a perfect venue for this event. The Architecture is breathtaking, the weather was surprisingly temperate and the evenings were a blast (thanks in part to some rather stunning Russion Imperial stout!).

Next year's KA Connect event is already being lined up to take place in San Francisco. Keep an eye out on http://www.ka-connect.com/ for news of what's happening next. This is the beginning of a Tribe which is going to bring tremendous value to the AEC sector..

R

KnowledgeSmart sponsors KA Connect Conference

Our friends at Knowledge Architecture are organising and hosting the first knowledge management conference for the AEC industry - KA Connect - and KnowledgeSmart are proud to be sponsoring the event.

This is a fascinating conference, with a superb lineup of leading AEC figures presenting on a range of topics. With a general theme of, 'Short Talks, Big Ideas', this promises to be one of the key events of our calendar year. With influences like TED (http://www.ted.com/) and marketing guru Seth Godin (http://www.sethgodin.com/), KA-Connect promises to be breath of fresh air.

A format tailor-made to stimulate fresh ideas and original thinking, boasting a blend of blue sky talks with Pecha Kucha (20 slides, 20 seconds per slide), expert panel discussions, video and breakout groups.

The speaker lineup is diverse, sharing emerging trends, new ideas, best practice and lessons learned for building knowledge-driven firms. The goal is to empower leaders in AEC firms to engage in, 'the process of systematically creating, capturing and sharing knowledge assets to leverage (yourself and) your organization'.

The conference organiser is Knowledge Architecture (http://www.knowledge-architecture.com/), a knowledge management and information systems consultancy, based in San Francisco.

The show takes place in Chicago on April 8 & 9, at theWit Hotel. (http://www.ka-connect.com/). Maybe we'll see you there.

Rory

KnowledgeSmart goes live

Well, we rolled out the new tools last week and so far, so good! It's a new direction for us in terms of the core approach to skills testing. The new web based platform allows greater flexibility in the way firms can deploy test modules and capture knowledge across a wide range of topics.

We're leaving a beta label on the system for a few more weeks; it's fully operational, but we welcome user feedback on any issues, bugs, missing data sets, or indeed anything that looks like it doesn't belong! Our next priorities are to add some more functionality to the admin dashboard; this includes some content management tools (for when question libraries get to a certain size), some file management tools (for marrying data sets to test questions in a variety of ways) and some general usability items, which have been suggested during pre-launch testing.

In addition, we have a rolling programme of new test material creation. Currently under development include the following libraries; Rhino for Architecture, AutoCAD & MicroStation for occasional users, Bentley Structural Modeller, MicroStation 3D fundamentals, Vectorworks, AEC Revit Standards, Ecotect fundamentals, Bentley InRoads and Civil 3D. These will all be making an appearance over the next few months.

KnowledgeSmart Enterprise customers will automatically see new library modules appear in their dashboard, as soon as they go live. The next challenge is to pull the live test results data into an online skills matrix tool, which administrators can use to display a range of useful metrics. Our intention is for this tool to dovetail into existing intranet environments, feeding into training and project resourcing activity, fully searchable, helping firms to ID the right people, with the right skills, at the right time.

Part of the challenge the new tools present is the variety of options available to admins when deciding just what test types to present to users. In the past, we have provided tests in any colour the customer could choose (so long as they chose black!). Now they really do have choice - and lots of it! The new 'Build Your Own' (BYO) feature enables firms to create their own test material from scratch, using an intuitive online authoring wizard to guide them through the process. AEC firms can capture knowledge across multiple topics, whether that be technical software ability, understanding and adherence to in-house standards, discipline specific knowledge, project-specific information, or indeed anything at all!

On the subject of Standards, we have partnered with the team at Evolve Consultancy to create test content for AEC (UK) CAD Standards for Layers/Levels. Next week, we'll be meeting the team at the US National CAD Standards committee to discuss a skills test for US CAD Standards.
We're running a series of web meetings over the next few weeks, to demonstrate the new system and tools on offer. To book a session, email info@knowledgesmart.net with 'Webinar' in the subject line and we'll schedule you in.

We have an ambitious programme of marketing and promotional activities planned between now and the Summer, both at home and in the US. We're sponsoring the Bentley Community UKI Winter Workshop in Oxford this month (http://www.bentleyuser.org/). We're presenting at the first WAN HR Forum in London on March 18th. And soon after that, we're heading off to Chicago, where we are sponsors at the inaugural KA-Connect knowledge management conference (http://www.ka-connect.com/).

Hope to see you online or in person in the coming weeks!

Rory

CAD & BIM - State of the Nation(s) article

Happy New Year. Towards the end of last year we held a series of (taped) discussions with the guys at Evolve (http://www.evolve-consultancy.com) and Robert Green (http://www.cad-manager.com) , which culminated in a published article looking at the big picture of CAD training, CAD management, BIM, and a whole lot more.

Credit to Karen for her amazing editing skills and to Nigel for making a fair dent in his red wine collection, as the evenings wore on! :)

The results were published in a variety of places, but most recently, here: http://www.itshowcase.co.uk/News/2009-05-12/Is-your-business-smart-enough-An-interview-on-CAD-usage/.

So, we're finally out of 2009 (hooray!). 2010 is looking much more encouraging, with the economy starting to drag itself off the floor. 'Bout time!

February sees the rollout of the new KnowledgeSmart web-based testing tools. If you'd like a demo, drop a line to info@knowledgesmart.net, with 'DEMO' in the subject line, and we'll pencil you in.

Plenty more to follow over the coming weeks..

Rory

One door closes...

...another opens, as the saying goes.

This week, we pass another milestone on the winding road of CAD & BIM skills testing. The much loved CADsmart brand is being superseded by our new name - KnowledgeSmart.

Welcome to our new blog! (Our old blog can still be found at http://cadsmart.blogspot.com/).

All of our existing CAD skills tools are still available, but we are embarking on a new journey in January, with the introduction of our web-based testing platform.

More and more, we find ourselves involved not just in the traditional realm of CAD skills testing, but also straying into the wider field of learning and knowledge management.

Over the coming months, we'll be broadening the range of design and engineering software applications which can be assessed, whilst at the same time developing a community resource for building even more test content. And we have some exciting plans for our friends in the Revit community coming up in the Spring. Watch this space!

We look forward to the next phase of the journey..

Rory