AEC IT Leaders Roundtable

KnowledgeSmart will be showcasing the latest release of our skills assessment tools for CAD, BIM, Engineering & Analysis software tools, at this year's AEC IT Leaders Roundtable.  The meeting takes place in San Jose, California, on 14 & 15 April.

KS CEO, Rory Vance, will be talking about the benefits of capturing performance metrics for a range of design and engineering software titles - and the shortcomings of firms who rely too heavily on self-assessment, as a means of identifying training needs for their teams.

For more information, go to: http://sites.google.com/site/aecitleadersorg.

New charting options

This month sees the launch of the latest release of the KS tools, with a host of new features to be found in the admin dashboard.

One of the key updates relates to the range of charting and management reporting options now available.

Here's a brief summary..

Search & Group Results
The first task for an admin, is to carve up the raw results data into more manageable blocks.  There are a range of searching and grouping options available, including; score ranges, date ranges, elapsed time, test topic, user names, training keywords, plus 5 additional custom datafields.

Performance Scatter
This chart displays group results in performance quartiles.  The upper left quadrant (Q1) contains results, where the users completed their assessment accurately and fast.  The upper right quadrant (Q2) shows results which have high accuracy, but slower completion times.  Bottom left (Q3) represents lower scores, but in a fast time.  Lastly, the bottom right quartile (Q4) shows test scores which are inaccurate and slow.  Hint: you don't want your core users anywhere near quartile 4!

Training Requirements

This chart highlights training topics for a given group of test results. The logic analyzes all of the results for a group, references the training tags assigned to questions presented during a test and lists those tags in priority order.
Red indicates the tasks which have been answered incorrectly by most people in a given group. Orange is the next highest priority, followed by Yellow. Green training tags are the topics which have been answered correctly by most of the group, so represent the least urgent issues.
For example, 10 people are presented with one or more questions, which include the tag, ‘Lines’. If 7 of the 10 people answer one or more ‘Lines’ questions incorrectly, then the keyword, ‘Lines’, will be flagged at 70% and appear in Red on the chart.

Question Performance

This chart looks at how each individual question in a library has performed, in any given test. The logic analyzes all of the results for each test and presents an aggregate percentage score for each question, divided by the total number of results for that test on the account. 

For example, 10 people answer a question called ‘Lines’. If 7 of the 10 people answer the question 100% correctly, 1 person scores 50% and the other 2 score 0%, then the question will score an aggregate of 75% and appear in Yellow on the chart. 

Group Scores
This chart displays user performance for any given group, in descending order. The X-axis shows the % score attained and the Y-axis displays user names.

Group Comparisons
This chart allows firms to compare performance, from one group to another, across (up to) 9 sets of data at a time.  Group vs group comparisons can be used to compare a range of results data.  For example; pre and post-training performance, different project teams, firm offices from different geographic locations, data representing different job titles or industry disciplines, in-house data vs interview candidates, and so on.

Global Comparisons

This chart allows AEC firms to compare in-house performance against a wider set of anonymous industry aggregate results data, for selected tests. Currently, we are publishing results for; AutoCAD 2D, MicroStation 2D and Revit Architecture.  When some of our other test titles accumulate a sufficient body of results data, we will publish additional benchmark figures.

This release is scheduled for go-live mid-March.  The new range of charting and reporting options should make identifying skills gaps and training needs much easier for CAD & BIM administrators and training and learning development professionals.

We'll be revisiting charting options again later in the year, so if you have any additional requests for chart styles and reports, please let us know and we'll add them to the Summer/Fall roadmap.

R

Question Types

When creating skills assessments, it is important to include a range of question styles, so the user experience is varied and engaging.

Here is a list of different question types, which commonly feature in most skills assessments:

Multiple-choice questions usually have up to eight answer choices with only one correct answer.

Pick-list or Check box questions usually have up to ten answer choices. Two or more of these answer choices can be correct.

True or False questions have two answers, True and False, one of which is correct.

Fill in the blanks questions have statements with blanks in them. The test taker must type text in each blank as the answer.

Short answer or Free text questions require the test taker to type in an answer to the question.

Order list answers must be selected in a specific order or sequence.

Point and click or Hot spot questions contain a picture or some other form of media. The test taker must select a point on the picture or image.

Numeric question types are similar to Short answer questions but require numeric instead of text answers.

Likert questions allow you to design a question based on a scale of answers, i.e. 1-5 or 1-10. (NB this type of question is better suited to surveys than straight assessments).

Matching questions have a content area and a list of names or statements (List A) which must be correctly matched against another list of names or statements (List B). List B usually includes a number of 'distractor' answers.

Essays are useful for fact-based or critical thinking test items. Answers can be a word, phrase, sentence, paragraph or lengthy essay. Essay questions are scored manually.

For capturing skills gaps across a range of design and engineering software applications, KnowledgeSmart typically uses the following types:

• Free text
• Multiple-choice
• Pick-list
• Order list
• True or False

Following on from a series of customer review meetings, we plan to include some new question types in the KS system this year:

• Fill in the blank
• Hot spot
• Matching

We'll also look into whether firms would like added flexibility on the free text questions.  Up until now, we have gone with the approach that an answer is either right or wrong.  However, it is possible to tweak the marking routines, so that a test taker can receive credit for entering one of multiple variations/spellings/etc. of the answer.

One of the main advantages of our system is the automated marking routine.  For this reason, we avoid using essay-style questions in our assessments.  We combine Free text and Numeric question types in the KS system.

R

Bloom's Taxonomy

In creating practical skills assessments for AEC firms, we try to offer a blend of question styles and types.  Some questions are task based (i.e. the user needs to physically perform a task, usually relating to a particular technical software application, in order to figure out the right answer).  Some questions are knowledge, or theory, based (i.e. the user should be able to answer the question without referring to any other resources, or opening up a software program).

Before we examine question types, however, it might be helpful to take a look at how people learn - an accepted definition of the cognitive process.

"Taxonomy" is a system of categorizing and organizing.  'Bloom's Taxonomy' is a system created to improve testing precision by categorizing cognitive functioning into different levels. Appropriate questions could then be developed to assess the desired level. Psychologist Benjamin Bloom developed this system at the University of Chicago in the late 1940s.  His goal was to improve the quality of discussion of educational goals among teachers, administrators, and researchers.

Bloom's levels are, in ascending order:

1. Knowledge 
2. Comprehension 
3. Application 
4. Analysis 
5. Synthesis 
6. Evaluation

In this case, the taxonomy is hierarchical; each level is subsumed by the higher levels. In other words, a student functioning at the "application" level has also mastered the material at the "knowledge" and "comprehension" levels.

Let's take a brief look at each stage - and what it means in terms of a learner's level of understanding..

Knowledge 
Recall content in the exact form that it was presented. Memorization of definitions, formulas, or procedures are examples of knowledge-level functioning.
Types of action: List, define, label, identify, name.

Comprehension 
Restate material in their own words, or can recognize previously unseen examples of a concept.
Types of action: Describe, associate, categorize, summarize.

Application 
Apply rules to a problem, without being given the rule or formula for solving the problem.
Types of action: Apply, calculate, illustrate, solve.

Analysis 
Break complex concepts or situations down into their component parts, and analyze how the parts are related to one another.
Types of action: Analyze, compare, separate, order, explain.

Synthesis 
Rearrange component parts to form a new whole.
Types of action: Combine, modify, rearrange, "what-if".

Evaluation
Evaluate or make judgments on the worth of a concept, object, etc. for a purpose.
Types of action: Assess, decide, grade, recommend, explain, judge.

In my next blog, we'll take a look at the different types of question, which typically make up a skills assessment.

R

KnowledgeSmart user group meeting

We're organising our next user group meeting in London next month.

Here are the details:

Date: Tuesday 15 March

Time: 9am – 1pm (working lunch included)

Venue: Stirling Room, RIBA, 66 Portland Place, London

Cost: Free of charge

Places are filling up rapidly - and we always end up with a waiting list for these sessions - so if you'd like to come along, you'll need to let us know in the next few days.

We have room for about 30 people in total and, as of this afternoon, we have 10 places remaining.

The agenda on the day will be pretty flexible, with plenty of scope for open discussion. From our end, the main talking points will be:

KS system tools update – what’s new?

KS wish list – tell us what else you’d like to see in the next 3-6 months..

Training links – marrying the KS training outputs with appropriate learning material

The impact of BIM on technical software in AEC firms

A closer look at testing Revit and Civil 3D skills

Writing your own assessment material – do’s and don’t’s

Plus whatever else comes up in open debate!

Hope to see you there!

R

Assessing Revit skills

We have a rolling program of new test content creation; our aim is to double the size of the KS library this year, depending on how things go, economically speaking. So far, so good in 2011.

It will come as no great surprise to learn that Revit tops the list of customer requests, so we have a few ideas for new content coming up in the next 3-6 months.

That's not to say the existing Revit test library material isn't looking good already! We have some excellent RAC questions, authored by leading Revit protagonists Paul Aubin, Paul Woddy and Chris Senior. We have a comprehensive RST library, written in partnership with Thomas Weir. And our RMEP modules are coming along nicely, too, with the help of US based author, Tony Tedder.

It's just that people want more! We're happy to comply, so here's what we have in mind..

Revit Families - a module which looks specifically at creation of Revit families. We'll probably split this into 2 sections; one for general families and one for discipline specific content.

Revit process - a module covering the practical aspects of using Revit on a project and the impact of working in a team environment.

RAC 2012 - more basic test content for Revit Architecture 2012, linking to corresponding training courseware from authors White Frog.

RMEP 2012 - more basic test content for Revit MEP 2012, linking to corresponding training courseware from authors White Frog.

RST 2012 - more basic test content for Revit Structure 2012, linking to corresponding training courseware from authors White Frog.

Timescales are not set in stone, but work is underway on the families and process modules. We'll start the new 2012 modules in March, once the code for the new releases from Autodesk is more or less frozen! We're aiming to have much of the new library material in place by the Summer.

Last, we have some intermediate level material planned for all 3 disciplines, during the Summer/Fall period.

R

Assessing AutoCAD Civil 3D skills

Another new addition to the KS test library went live this week - AutoCAD Civil 3D 2011, module 1.

Now I'm not a believer that you can reliably create a 'general' Civil 3D test, so we're taking a slightly different approach with this one.

Because the KS tools offer AEC firms a tremendous amount of flexibility, in the way they can customise test content, we are giving firms the final say on what modules their version of the C3D test will contain. The daisy-chain feature, which allows admins to link a series of individual modules together to form a longer test, means that firms can mix & match the C3D content in a wide variety of ways.

So how are we approaching this task? Well, first, we're working with leading C3D author, Eric Chappell, to create 4 overall test modules, covering a range of C3D tools and commands. Each module will contain c.25 test questions, with the usual mix of task-based and knowledge-based exercises.

These are the main topics, which will be covered across the 4 modules:

• Surfaces
• Alignments
• Profiles
• Points
• Sections & quantity take offs
• Assemblies & Subassemblies
• Corridors & intersections
• Parcels
• Grading
• Pipes & pipe networks
• Plan production
• Data exchange
• Rendering
• Survey
• Hydraflow

Module 1 is now available, covering Profiles, Alignments and Surfaces. Work has already begun on module 2, which will be finished in March. Module 3 will follow in April/May. The last module will be written in the summer; we'll review progress to date at that point and take feedback on any remaining features which haven't been addressed in modules 1-3.

Enterprise customers will see the new C3D content appear in their accounts this week. As always, we welcome your feedback on the material and also any topics you'd like to see covered, which are not mentioned in the summary list above.

R