The KnowledgeSmart Blog

Demonstrating Professional Credentials

2012-01-09T12:24:00.000-08:00

A common theme, here at KnowledgeSmart, is that of skilled professionals being able to demonstrate their capability and credentials, whatever the subject. And it is a topic which is being discussed at the highest levels, here in the UK, as 2012 rolls around.

Back in 2009, I posed the (rather tongue-in-cheek) question; what do bus drivers and welders have in common that CAD & BIM professionals don't? The answer is simple; bus drivers and welders need to demonstrate their professional credentials on a regular basis throughout their careers, in order to be allowed to do their job.

Bus drivers (and lorry drivers) have to pass a 'certificate of professional competence' in order to drive their vehicles in the UK. The CPC was developed as a requirement of a European Directive, designed to improve the knowledge and skills of professional LGV and PCV drivers throughout their working life. There are two parts to the legislation:

- An Initial Qualification that must be achieved by new (LGV and PCV) drivers along with their vocational licence to enable them to use their licence professionally.
- Periodic Training, which involves all professional drivers undertaking 35 hours of training every 5 years.

Meanwhile, welder certifications are specially designed tests to determine a welder's skill and ability to deposit sound weld metal. The tests consist of many variables, including the specific welding process, type of metal, thickness, joint design, position, and others. Most certifications expire after a certain time limit, and have different requirements for renewal or extension of the certification. Once a welder passes a test (or a series of tests) their employer will certify the ability to pass the test, and the limitations or extent they are qualified to weld, as a written document (welder qualification test record, or WQTR).

So if the driving industry has a recognised benchmark for performance and if the welding industry won't allow its members to pick up a blow-torch without first demonstrating they have the requisite skills, then why on earth does the AEC industry let just about anyone gain access to a CAD or BIM software license?

I remember reading a story on ENR.com, a while back, highlighting the dangers of unqualified staff working on projects. A warehouse in Philadelphia suffered a major structural collapse, resulting in damages of $3.5M USD. A building-collapse expert concluded, 'The team committed several engineering errors of "amazing proportions" that caused the Philadelphia warehouse to fail under the weight of snow'. He found that the warehouse had only one-third of the steel roof framing it needed.

Someone using analysis software grossly underestimated the quantity of materials, the error went unnoticed, the building was constructed, the snow fell... and they firm got hit with $3.5M in costs. Plus a number of people were hurt when the roof fell in. Ouch! Suddenly a few extra dollars on testing a person's proficiency using engineering software, plus a few hours training, seems like a good use of the company resources!

A CIO at a large US design firm once said to me; “The minute you give someone a BIM license on their desktop – you are effectively giving them full access to your end product. You want to be sure they know what they are doing!”. This firm doesn't allocate a license to someone's desktop unless they have first met the minimum threshold competence level for performance, as identified by the firm's BIM administrators (using KnowledgeSmart, in this instance).

A bus company doesn't give someone the keys to its shiniest double-decker without making absolutely sure they have the current skills required to drive the vehicle. Isn't it time that the AEC industry recognised the need for clear skills certification in core authoring, BIM, Civil and analysis software packages? And a continuous improvement program for design and engineering professionals to demonstrate their credentials to their employers on an ongoing basis?

Well, hopefully, that is precisely what is happening, as the UK Government mandates use of BIM Level 2 on all appropriate construction projects, by 2016. Despite (or perhaps because of) the ongoing economic hardship around the world, many firms are revisiting their technology assessment and training programmes, to make sure that they can measure the skills of key personnel. Firms who can demonstrate to clients that they have the requisite skills to deliver a BIM project safely, on time and budget, will have a definite competitive advantage, as the clock counts down to 2016.

R

AUGI Top DAUG 2011 - The Results

2011-12-08T09:07:00.000-08:00

The AUGI Top DAUG competition was introduced at Autodesk University, way back in 2000. The contest originally consisted of two parts, based on AutoCAD 2D knowledge. This year, AUGI teamed up with KnowledgeSmart to expand the range of topics to include 7 tracks; 3ds Max, AutoCAD 2D, AutoCAD Civil 3D, Inventor, Revit Architecture, Revit MEP and Revit Structure.

Here is an overview of the contest from AUGI Hot News:
http://augi.typepad.com/files/augi-hotnews-daily---wednesday-nov-30-2011.pdf

There were some great prizes up for grabs, too. Track winners walked away with a $100 USD Amazon voucher (a combination of high scores and fastest times determined the winner of each track) and the overall winner also won a HP notebook PC and a free pass to AU 2012!

We spent several hours on the Monday, setting up 18 networked PC's in the exhibition hall, at the AUGI stand. All 126 copies of the Autodesk software needed checking, then all the KS sample data sets had to be uploaded to each machine and checked again. Big thanks to Daniel Heselwood, from Evolve, for answering our call for help. (We actually called Daniel to ask for his help with a missing xref - and 4 hours later he was still helping us finish up!).

Here's a snapshot of the stand, when we were all set up:

The competition ran over 3 days (2 x 3 hour slots on the first 2 days, with a final hour session on the last day). Contestants had to answer a range of questions, using the 2012 version of each software title. Each session was limited to just 14 minutes, so people had to work fast! The new format proved popular, with eager users queuing up to grab a spare seat. Special thanks to AUGI team members, Bob Diaz and Michael Patrick, for briefing users and making sure everything ran smoothly.

Here are the Top DAUG's in action:

By the end of the contest, we posted 305 results, across 7 tracks. On a conference venue web connection, with the added ingredients of free beer and wine, we had a completion rate of 99%, which was way ahead of our anticipated 95% target. (Our support team promptly rescued the missing two scores, so all completed tests were captured and logged on the AUGI dashboard).

Throughout the competition, we posted a rolling list of the top 10 contestants for each track, on the AUGI big screen.

The Results

Congratulations to the following contestants, who won their respective Tracks:

3ds Max Track Winner: James Clarke

AutoCAD 2D Track Winner: Brent McAnney

AutoCAD Civil 3D Track Winner: Brian Hailey

Inventor Track Winner: Joe Bartels

Revit Architecture Track Winner: Aaron Maller

Revit MEP Track Winner: David Raynor

Revit Structure Track Winner: Rebecca Frangipane

And a special mention to the overall winner of AUGI Top DAUG 2011:

Brian Hailey

Analysis

So, let's take a detailed look at the results of this year's Top DAUG competition.

Overall

No. of Tests Completed: 305

Overall Average: 53% in 12 mins 56 secs

Track 1 - 3ds Max

Track Winner: James Clarke

Winning Score: 64% in 13 mins 55 secs

Top 10 Contestants:

James Clarke

Jesse Sandifer

Daniel Heselwood

Paul Mazzoni

Fernando Oliveira

Jens Tange

Matti Oopik

Yosun Chang

Douglas Bowers

Charlie Forsythe

No. Completed: 13

Group Average: 18% in 12 mins 15 secs

Top 10 Average: 20% in 12 mins 2 secs

Track 2 - AutoCAD 2D

Track Winner: Brent McAnney

Winning Score: 100% in 11 mins 25 secs

Top 10 Contestants:

Brent McAnney

Alex Lepeska

Scott Wilcox

Heather Shrieves

Timothy Vaughan

Ben Rand

Decio Ferreira

Jim LaPier

Richard Lawrence

Youssri Salman

No. Completed: 110

Group Average: 42% in 13 mins 43 secs

Top 10 Average: 82% in 12 mins 42 secs

Track 3 - AutoCAD Civil 3D

Track Winner: Brian Hailey

Winning Score: 100% in 6 mins 10 secs

Top 10 Contestants:

Brian Hailey

Richard Lawrence

Travis Winter

Bryan Thomasy

Kirk Noonan

Jeff Nichols

Bruce Klug

Brent McAnney

Charles D'Errico

Bill Neuhauser

No. Completed: 26

Group Average: 61% in 12 mins 33 secs

Top 10 Average: 90% in 11 mins 24 secs

Track 4 - Inventor

Track Winner: Joe Bartels

Winning Score: 80% in 8 mins 20 secs

Top 10 Contestants:

Joe Bartels

Gerrard Hickson

Andrew Warren

Bill Graham

Alex Karan

Marius Minnen

Anders Tokerud

Chris Brown

Scott Wayand

Curtiss Cooke

No. Completed: 27

Group Average: 59% in 13 mins 6 secs

Top 10 Average: 75% in 12 mins 9 secs

Track 5 - Revit Architecture

Track Winner: Aaron Maller

Winning Score: 100% in 6 mins 55 secs

Top 10 Contestants:

Aaron Maller

Brian Mackey

Anthony Tiefenbach

Steve Faust

Sean Darnell

Eric Bernier

Adam Ward

Douglas Bowers

Andrew Fisher

David Ivey

No. Completed: 84

Group Average: 79% in 11 mins 35 secs

Top 10 Average: 99% in 10 mins 7 secs

Track 6 - Revit MEP

Track Winner: David Raynor

Winning Score: 80% in 13 mins 25 secs

Top 10 Contestants:

David Raynor

John Karben

Fernando Oliveira

Jason Vaia

Philip Charlson

David Rushforth

Clifford Baker

Douglas Bowers

Maxime Sanschagrin

Paul Beseman

No. Completed: 23

Group Average: 54% in 13 mins 50 secs

Top 10 Average: 67% in 13 mins 49 secs

Track 7 - Revit Structure

Track Winner: Rebecca Frangipane

Winning Score: 98% in 13 mins 45 secs

Top 10 Contestants:

Rebecca Frangipane

Andrew Lawrence

Dezi Ratley

Brian Mackey

Eric Bernier

Matthew Hill

Tina Bos

Michael Patrick

Jason Rhodes

Fernando Oliveira

No. Completed: 22

Group Average: 56% in 13 mins 32 secs

Top 10 Average: 77% in 13 mins 5 secs

Further Analysis & Observations

Popular Tracks

The most popular tracks, in order of completed tests, were as follows:

AutoCAD 2D - 110 results

Revit Architecture - 84 results

Inventor - 27 results

AutoCAD Civil 3D - 26 results

Revit MEP - 23 results

Revit Structure - 22 results

3ds Max - 13 results

Training Needs Analysis

So what does all this mean, in terms of performance and training?

For 3ds Max, the top 10 training items were, in priority order:
Creation Parameters, Cross Sections, Map Channels, Map Coordinates, Object Data Flow, Object Modifiers, Object Properties, Pivots, Rotate and Space Warps.

For AutoCAD 2D, the top 10 training items were, in priority order:

Attaching References, Modifying References, Dimensioning Objects, Measuring, System Variables, Object Snaps, Arcs, Drafting Tools, Lines and Tangent.

For AutoCAD Civil 3D, the top 10 training items were, in priority order:

Changing a Slope, Pipe Networks, Sections, Corridor Surfaces, Corridors, Breaklines, Surfaces, Survey, Alignments and Parcels.

For Inventor, the top 10 training items were, in priority order:

Parameters, Tolerances, Features, Revolve, Sketch, Hole, Project Settings, Styles, Components and Parts.

For Revit Architecture, the top 10 training items were, in priority order:

Clipboard, Data Integrity, Modeling, Family Editor, Families, File Management, Worksharing, Room Tags, Walls and Annotation.

For Revit MEP, the top 10 training items were, in priority order:

System Inspector, Apparent Load Values, Creating Power Systems, Editing Properties, Modifying Light Fixtures, Panel Circuits, Panel Schedules, Photometric Web Files, Selecting Equipment and Electrical Settings.

For Revit Structure, the top 10 training items were, in priority order:

Families, Scope Boxes, Types, Visibility, Roofs, Floors, Slabs, Structural Deck, Annotation and Span Direction Symbol.

Revit Data

It's interesting to compare the Revit results, across the 3 disciplines, which arguably reflects where Revit currently sits, in terms of wider industry adoption. Revit Architecture proved to be the most popular track and the overall experience of the users taking part in Top DAUG, suggests that this software is the most mature, in terms of individual expertise. Revit MEP and Revit Structure were very close, in terms of overall performance, but considerably behind Revit Architecture, on a like-for-like productivity comparison.

RAC overall: 79% in 11 mins 35 secs

RMEP overall: 54% in 13 mins 50 secs

RST overall: 56% in 13 mins 32 secs

It's also worth noting that the average Revit Architecture score, for the top 10 users, was an impressive 99% in 10 mins 7 secs. In fact, 31 users out of 84 posted scores of 90% or higher, which is outstanding.

As an additional comparison, we posted 28 results for Revit Architecture at RTC USA, earlier in the year. The overall average for Revit Architecture, at the RTC 'Top Cat' contest, was 79% in 9 mins 7 secs. (NB the questions were different at RTC, but the level of difficulty was on par with Top DAUG).

Range of scores

Interestingly, across the 7 tracks, we saw scores ranging from 0% to 100%. Here is a summary of both ends of the performance scale:

7 x 100% scores (1 x AutoCAD 2D, 1 x AutoCAD Civil 3D, 5 x Revit Architecture).

13 x 0% scores (1 x 3ds Max, 11 x AutoCAD 2D, 1 x Revit Architecture).

Honourable mentions

Along with our track winners, the following contestants deserve a special mention, for their performance in the competition:

Brian Mackey - winner of the RTC USA Top Cat contest in the summer, Brian placed second in the Top DAUG RAC track (scoring 100%), and also placed in the top 4 for RST. An exceptional effort!

Brent McAnney - won the AutoCAD 2D track and placed in the top 8 for AutoCAD Civil 3D.

Eric Bernier - placed inside the top 6 for both the RAC and RST tracks.

Douglas Bowers - placed inside the top 8 for both the RAC and RMEP tracks.

Fernando Oliveira - placed inside the top 10 for the 3ds Max, RMEP and RST tracks.

Richard Lawrence - placed inside the top 10 for both the AutoCAD 2D and AutoCAD Civil 3D tracks.

Heather Shrieves - placed inside the top 5 for AutoCAD 2D and top 12 for Inventor.

Jason Vaia - placed inside the top 5 for RMEP and top 18 for RAC.

Boys vs Girls

Finally, let's take a look at the demographic breakdown of the competition. Out of 305 contestants, 271 were male and 34 female. The average overall performance for each group breaks down like this:

Female: 61% in 12 mins 37 secs

Male: 53% in 12 mins 57 secs

So, there we have it! A thoroughly enjoyable 3 days at AU2011. 305 completed tests, across 7 popular Autodesk software applications. It's fair to say that the overall standard was extremely high, with some truly outstanding individual performances from our track winners and top 10 contestants.

Congratulations to all our winners. Thanks again to the AUGI team for all their support, in helping us put together the new format for Top DAUG. Particular thanks to AUGI President, David Harrington, past-President, Mark Kiker and past-Board member, Bob Diaz. Lastly, a big thank you to everyone who took part in this year's contest. See you all at the Mandalay Bay in 2012?

KS devs - a brief look ahead

2011-11-25T07:38:00.000-08:00

As we head to the close of the year, I thought it might be useful to drop a quick update on the blog, about the work currently under development.

The next release is split into two parts. Part one goes live next month and introduces some fairly big changes to the core KS system. These updates involve introducing a hierarchical tree structure into the user accounts, sharing tests/modules between child accounts, moving users/results data between accounts, and including a new 'draft' phase when creating new test content.

Part two focuses on improving the 'Invites' area of the dashboard, with a number of great ideas from the KS user group being added to the system. We'll also introduce personal dashboard pages, so individuals can track their own progress, over time.

A key theme for KS in 2012 is to continue the concept of 'links to learning'. We are in discussion with a number of training providers, discussing how to make the journey from assessment to learning even smoother.

Exporting user and results data is another common theme, which will be addressed further in 2012. Tighter integration of skills assessment data, into wider HR systems and company Intranets, has been a popular discussion topic this year.

We'll be adding some new question types to the dashboard in a future release, including survey tools and the ability to gather and present user feedback and opinion, as well as test results. The KS 'community' area will also continue to grow in the months ahead, offering KS firms more opportunities to exchange test questions and material.

More benchmark stats and reporting options will be appearing in the next release, with more detailed geographical and industry breakdowns for AEC businesses to compare performance. We'll also be adding some new content management tools, which will be particularly helpful to KS firms with large test libraries.

Speaking of which, the KS 'off the shelf' library continues to roll on, with new modules appearing each month. Next up, we'll be adding more Civil 3D questions, more Revit Structure and Revit MEP questions, a Navisworks Manage module, an Ecotect module, plus some new material based on Oracle's Primavera PCM application.

So that's a quick overview of what's coming over the next 3, 6 and 12 months. As always, we welcome your feedback, constructive comments and new ideas. It looks like 2012 is shaping up to be a busy one.

May we (continue to) live in interesting times!

R

AU 2011

2011-11-25T07:11:00.000-08:00

OK, it's that time of year again. KS is exhibiting at this year's Autodesk University. We’re also co-hosting the AUGI Top DAUG skills contest this year: http://www.augi.com/autodesk/AU/augi-top-daug.

In spite of the ongoing financial woes around the world, this year's show promises to be one of the busiest yet. No doubt there will be much talk of BIM, VDC, IPD, Cloud computing and, if rumours prove to be true, announcements about an Autodesk move into the PLM space. We'll see!

For now, if you're heading out to Las Vegas next week, do please drop by the KS booth, or come and see us at the AUGI Top DAUG area.

We look forward to meeting you.

R

BIMshow Live

2011-10-04T11:49:00.000-07:00

Amongst a host of upcoming Conferences, Roundtables and Webinars, all on the ubiquitous topic of BIM, one in particular stands out from the crowd.

In November, the inaugural BIMshow Live takes place at the Business Design Centre, in London.

HOK's prolific blogger and tweeter, David Light, provides a handy summary on his popular BIM & Revit blog.

Here's the link: http://autodesk-revit.blogspot.com/2011/09/bimshow-live.html.

R

KS Community - BIM Fundamentals

2011-10-04T11:40:00.000-07:00

There is a wide range of quality assessment content, currently appearing in KS dashboards, written by AEC firms from all over the world. Topics vary from in-house CAD & BIM processes, to 'how to use the company Intranet', to BIM workflows and standards, to Project Management skills, and more.

An interesting trend is that firms are actively engaging KS to facilitate a 'community' pool of content, whereby firms, using the law of reciprocity, are volunteering to share their own, self-authored, non-proprietary test content, with other KS firms, in exchange for the option to access material from the wider KS community.

It is an environment of cooperation and collaboration, which is refreshing in its simplicity and effectiveness. And we're just getting started! Already, we are seeing a stream of fresh, new, current material coming online, every month. Oh, and the best modules are being added to existing KS libraries FREE of charge.

Under the umbrella of the free community content, something very interesting happened just last week. A handful of firms contributed their in-house questions on 'BIM Fundamentals', to the community pool.

Now, whilst this module is a work in progress, I thought it would be interesting to send some invites out to the BIM community, to take a look at what has been authored so far. To say that this has caused quite a lot of conflicting opinion, would be quite the understatement! I knew that attempting to create a general BIM test was going to be tricky, as ‘good’ BIM practice is still subjective, across the regions.

The fact is, this set, is really meant to be ‘raw material’, i.e. not to be used by any firm un-edited. For example, there is a slight Revit bias in one or two of the technical questions, but that is principally because all these questions were written by AEC firms, many of whom are primarily Autodesk users. (One could also argue that this is reflective of industry, I suppose). A Bentley or Graphisoft firm, will almost certainly have an opposing view and that's as it should be.

From my perspective, I think this set could be used by a firm (appropriately edited) to stimulate a more intelligent discussion in-house about BIM, BIM process, BIM strategy, what BIM is (and is not), and so on. The coaching notes are quite helpful to fill in the blanks in people’s general level of understanding.

We’ve had some interesting suggestions on additional topics to cover. These include; IFC, IPD, IDM, information exchange protocols, clash detection and coordination procedures, BIM implementation plans, level of development standards, managing model merge from other consultants/sub-contractors, BIM workflows (who starts first, who has priority), 4D, 5D, COBie, field BIM, model validation, BuildingSMART, sustainability and the McLeamy curve.

The simple fact is that, globally, the industry is still at a comparatively early stage of BIM adoption and the questions in the set so far – all donated by firms in the field, using BIM every day on projects – reflect where they are currently up to, with their BIM thinking.

I think we’ll see some new BIM questions rolling in, over the coming weeks, as the discussion continues. We’ve had such a wide range of comments already (in just a few days) so it is very clear that opinion remains divided, even amongst BIM ‘experts’! :)

On a related note, I was reading a thought-provoking article by Martyn Day, at AEC Magazine, entitled, 'The Trouble with BIM'. (See link: http://aecmag.com/index.php?option=com_content&task=view&id=450).

As the KS community pool gathers momentum, I think we'll see a host of new test material coming online. My view is that it’s easier for firms to start from an existing set of material, than to re-write the whole thing from scratch. That's why having full editorial control over the content is vital.

As far as a universally accepted BIM standard goes, I think we're some way off realising this goal. I'll leave the final word to one of the beta testers, who displayed an admirable sense of self-deprecation, and who clearly grasped the intent with which the beta invites were sent out; 'Does this mean I'm too DIM for BIM?' :)

R

BIM and Integrated Design - a new book by Randy Deutsch

2011-09-12T11:49:00.000-07:00

Friend of KnowledgeSmart, Randy Deutsch, has written a fantastic new book, called, 'BIM and Integrated Design: Strategies for Architectural Practice'.

Unlike many of the recent books and manuals on BIM, this book is devoted to the subject of how BIM affects individuals and organizations working within the construction industry.

As one of the most popular speakers and presenters on the AEC circuit, Randy is well placed to comment on the journey that many firms are currently making towards BIM adoption. In his book, Randy discusses the implementation of building information modeling software as a cultural process with a focus on the technology’s impact and transformative effect—both potentially disruptive and liberating—on the social, psychological, and practical aspects of the workplace.

BIM and Integrated Design answers the questions that BIM poses to the firm that adopts it. Through thorough research and a series of case study interviews with industry leaders, this book helps us learn:

• Effective learning strategies for fully understanding BIM software and its use
• Key points about integrated design to help you promote the process to owners and your team
• How BIM changes not only the technology, process, and delivery but also the leadership playing field
• How to become a more effective leader no matter where you find yourself in the organization or on the project team
• How the introduction of BIM into the workforce has significant education, recruitment, and training implications

Covering all of the human issues brought about by the advent of BIM into the architecture workplace, profession, and industry, BIM and Integrated Design shows how to overcome real and perceived barriers to its use.

Click here for a link to Wiley's website, for more information and to order a copy of Randy's book.

R

Links to Learning

2011-09-11T12:34:00.000-07:00

An important theme for KnowledgeSmart this year, is 'Links to Learning'. That is, making the connection between highlighted skills gaps and corresponding training content, as simple a journey as possible for KS customers.

The latest KS release takes an important step along the path to achieving closer links to relevant, targeted, training material.

Here is a summary of how AEC firms can now display focused, modular training content, as part of their skills assessment results analysis.

We have added a new stage to the question create/edit process, called 'Learning material'.

From here, KS admins can link modular training material to their KS question library. The key is to map the training content to the training tags (meta data) for each question. The training content can be a variety of formats. For example in-house training notes, PDF help-sheets, Intranet pages, etc. Or, alternatively, training material from a 3rd party, such as video tutorials or web based training. KS admins can use the text box to add notes or web links and the file upload tool to assign files or video links. (NB to ensure that any web links open in a new browser, please use the 'Insert/Edit URL link' icon, enter the relevant web address(es) and select 'New window' in the dropdown).

Next, there is a new option in Settings > Test Feedback, called 'Feedback, Coaching & Learning on Report'.

Checking this option, will prompt a new section to display in the user test report, at the end of their test session. From here, users can access the wider training content, relating to questions where they dropped marks. So the idea is that, KS Coaching notes provide feedback on each specific task. Whilst Learning notes provide broader feedback on the topic being covered by each question.

We are in discussion with a number of 3rd party training providers, to map their learning material, to the topics being covered by KS skills assessments, across a range of software titles. In the Results > Training Requirements page, we have listed a number of training providers, who are developing modular content, which can map to the KS training tags. (KS is not a reseller for any of these companies, but we are working together to provide closer links to their respective courseware solutions).

This development is another important step along the road to achieving a continuously improving journey, between assessment and learning.

R

Managing your Users

2011-09-11T01:21:00.000-07:00

We recently updated the Users section of the KS dashboard. Here is a guide to the new features.

When you log into your KS dashboard, you'll see some new sections in your menu sub-navigation:

It is worth pointing out that changes made to the user data in your dashboard, do not affect your results data. This is simply an area where you can manage your user information.

Adding Users

There are a number of ways to add user records to your dashboard. First, when you send assessment invites, the user record is automatically added to your user list (unless it is already present, in which case it will not be duplicated).

Next, you can go to the 'New User' page and add individual records from here.

Third, we have improved the way that user records can be uploaded via csv file. You can also export your user data as well. Go to 'Import/Export User Data'. In the Help notes box, you will find a link to a formatted sample csv template.

Save the template and copy across your user list from MS Excel, using the same 10 columns that you see in the template. They are: Email, First name, Last name, Status, Datafields 1-5 and Additional Information.

Browse for your csv file and use the 'Import User Data' tool to upload your new user list.

Please not that if you don't use the correct format on your csv file, then the KS system will not accept the imported file.

You can also export your user list to csv, using the 'Export User Data' tool.

Additional Datafields

You can capture 5 additional fields of information for each user, by selecting the magnifying glass icon, next to the user record.

When you click the magnifying glass tool, you will see a dropdown display, where you can edit a number of fields, including changing the status of a user (i.e. from Interviewee to Employee, or Employee to Ex-Employee).

You can use the information captured in the 5 datafields to filter your assessment results, in the Results page of your dasboard.

You can add custom labels to the 5 datafields, by going to Settings > User Datafields. Input your new labels and hit 'Save'. When you go back to the Users page and view individual records (by selecting the magnifying glass tool), the new labels will display against each user.

Filtering User Records

We have added searching and grouping functionality to the Users page, to make it easier for KS admins to filter their user data and create sub groups of user records. This is especially useful, when inviting different teams to take a skills assessment.

Select the 'Show search' and 'Show groups' tools, on the orange bar located at the top of the user page.

This opens up two new fields, where you can search against a number of user parameters, including, name, username, email, user status and the 5 additional datafields.

When you have run your search, select the user records by checking the box next to the user name (the top box is a 'select all' option). Then enter your group name in the 'Create New Group' field and select 'Create'.

You can view your new groups by selecting the dropdown field, in the Groups table. You can also add/remove new users to/from existing groups.

Inviting Groups

To invite a sub-group of users to take a KS skills assessment, go to the Groups table, select the group in the dropdown, check the box(es) next to the names of your invitees and hit the 'Invite selected users' button.

This will pre-populate your user list on the assessment Invites page. Just edit your invite text and email header and hit 'Send Email'.

So managing KS user data is now even easier. Remember, changes made to your user records have no direct impact on your live results data.

Evolve - Training Help Sheets for MicroStation and AutoCAD

2011-09-10T15:41:00.000-07:00

Leading CAD & BIM Consultancy, Evolve, led by Principal Nigel Davies, have developed an excellent training resource for AEC firms using MicroStation and AutoCAD software.

Designed to align with the keywords and meta tags associated with the assessment questions in the KnowledgeSmart library, AEC firms can now deliver high quality, targeted training material to their teams, matching precisely with highlighted skills gaps, for a fraction of the price of traditional AutoCAD or MicroStation training courses.

This is another important step in our aim to provide Links to Learning, across a range of popular AEC technical software applications.

For more details, go to http://www.evolve-consultancy.com and request a free sample.

R

White Frog - Modular Courseware for Revit

2011-09-10T15:19:00.000-07:00

As part of our 'Links to Learning' development, we have been working closely with the team at White Frog Publishing, to write new assessment content for Revit, across the 3 core disciplines (Architecture, Structure & MEP).

The Principals of WF have exceptional industry pedigree. Peter Routledge was MD of Tekla UK and has a huge amount of experience in the AEC sector. Paul Woddy is one of the UK’s leading independent Revit authors and trainers, having been involved with the product since the very early days, before it was acquired by Autodesk.

The WF training materials cover 13 core modules, plus a further 6 discipline-specific topics. Similarly, the KS assessments present 95 new questions, across these 19 modular areas. The first new module, Revit Architecture 2012, goes live next week. The questions are written to comply with the AEC (UK) BIM Standard.

The idea, very simply, is to dovetail the KS results and highlighted skills gaps, into the appropriate modular courseware developed by WF. Unlike other Revit training resources, or video tutorials, the WF content is intended for instructor-led training and designed with the trainer in mind, whether a full-time professional trainer or a part-time internal trainer, it includes all the tools to deliver high quality consistent training for AEC firms. In addition to delegate notes, the trainer gets access to trainer notes, slideshows and datasets to support the lectures, as well as more datasets, drawings and AVI videos to support the practical exercises.

For more details, go to http://www.whitefrog.co and register for a web demo.

By the close of the year, our intention is to create assessments for RAC 2012 (metric & Imperial), RST 2012 (metric & Imperial) and RMEP 2012 (metric & Imperial). More information and live product demonstrations of the combined assessment & training solution from KS & WF will be on display at Autodesk University.

R

Content Studio - a new way to manage Revit Families

2011-09-10T14:56:00.000-07:00

Managing Revit Families is one of those thorny subjects that perplexes even the most BIM-savvy AEC firms.

Chris Senior, Principal of Revit Factory, has created a brilliant new tool for helping AEC firms to manage their Revit content. I have had the pleasure of working with Chris for a number of years. He is one of the most experienced independent Revit consultants and trainers in the UK, and a regular contributing author to the KS assessment library.

Content|Studio is a digital library solution used to collect, manage and share Revit Families in a quality controlled environment. Now, AEC firms can file and index all their Revit files in one consistent electronic filing structure.

Chris explains, 'The advantages in efficiency will repay the investment made by a firm in months rather than years. Content|Studio offers much more than a conventional filing system; information is stored in an intelligent quality controlled library making it easy to capture, manage, share and use Revit files'.

For more details, go to: http://cs4revit.co.uk and register for a free demo.

R

CADsmart software - end of an era

2011-09-02T04:28:00.000-07:00

Some nine years ago, we started developing software plug-ins for AutoCAD and MicroStation, which delivered a practical skills assessment experience, linking to the API of the Autodesk & Bentley applications.

This was a successful formula for many years. But, as is often the case with technology, time moves on and, particularly with the impact of BIM on the AEC sector, we realised that we needed a more flexible approach, in order to embrace the future direction of our industry, with respect to skills assessment.

We also experienced many headaches trying to keep up with API changes across a broader range of tools, such as Revit, Civil 3D, Bentley Architecture, Rhino, the list goes on.

Hence the KnowledgeSmart web based testing format was created a couple of years ago and continues to thrive, in spite of the challenging economic backdrop.

So, it is with mixed feelings that we announce the official 'retirement' of the 'CADsmart' suite of tools, at the end of September. Farewell old friend.

R

AUGI Top DAUG

2011-09-02T04:05:00.000-07:00

Breaking news..

KnowledgeSmart and AUGI will be working together to provide a broader range of assessment topics for this year’s AUGI Top DAUG skills contest, at Autodesk University, in November.

In recent years, the contest has focused exclusively on AutoCAD 2D skills. However, for the 2011 Las Vegas conference, we will be expanding the range of topics to include, Revit Architecture, Revit Structure, Revit MEP, AutoCAD Civil 3D, Inventor, 3ds Max (and, yes, AutoCAD 2D will still be represented!).

Based on our recent experiences, delivering a successful formula for the RTC Revit Top Cat contests in Australia and the US, we are confident that the new Top DAUG format will be a hit with AU regulars.

Contestants will be presented with a mix of knowledge and task based questions, using the 2011 release of Autodesk’s most popular AEC software applications. Because the range of titles is so diverse, there will be no grand final this year. Just a straight shoot-out, for the title of Top DAUG for each vertical!

As for rumours of a heavyweight Revit title fight, between the RTC Top Cat and the AUGI Revit Top Daug, at the time of writing, we can neither confirm nor deny this story... J

More to follow on this topic in the months ahead.

CAD & BIM Manager's Survey 2011

2011-08-19T00:47:00.000-07:00

The UK Architectural, Engineering & Construction CAD & BIM Manager's Survey 2011 is now open for your input!

Organised by respected independent BIM consultants, Evolve, the survey is one of the most important of its kind in the UK construction industry, comparing the roles performed by CAD, BIM and Technology Managers in the sector. Topics covered include, roles and responsibilities, salaries, the ratio of CAD & BIM support staff to users, opinions on support providers, quality of training, standards and much more.

Evolve Principal, Nigel Davies, comments, 'Responses in previous years have been phenomenal, so we encourage people to take part this year to help paint a true picture of CAD and BIM within the UK as we continue to slowly progress out of a deep recession, and BIM begins to become more and more important.'

Nigel adds, 'The questions are mostly multiple-choice so the survey should take around 20-30 minutes to complete fully. Please feel free to leave any questions unanswered if you wish, there are no required fields.'

Here's the link: https://www.surveymonkey.com/s/UKCADBIMManagerSurvey2011

Results will be published in the Autumn.

R

KS Library update

2011-08-01T02:54:00.000-07:00

We've had a few mails in recent weeks, asking about updates to the KS library, so I thought I'd drop a few lines on the blog about what we're working on..

Updates to existing data sets are a common theme this summer. We work with a network of independent specialist authors, to create the KS OTS (off the shelf) titles. Juggling schedules with talented authors is challenging - as their skills are always in demand! However, we are making good progress, with a rolling programme of updates and new releases, taking us into the autumn period.

So, in approximate date order:

AutoCAD 2D fundamentals 2012 - Aug 11

AutoCAD 2D 2012 - Xpress - Aug 11

AutoCAD 2D for occasional users 2012 - Aug 11

AutoCAD Civil 3D 2011 (part 3) - Aug 11
RAC 2012 fundamentals (metric), new questions - Aug 11
RST 2011 fundamentals (metric), including new coaching notes - Aug 11
RST 2011 - Xpress (metric) - Aug 11
RST 2012 fundamentals (metric) - Aug 11
RST 2012 - Xpress (metric) - Aug 11

Google SketchUp fundamentals - Sept 11

RAC 2012 fundamentals (metric), including new coaching notes - Sept 11

RAC 2012 - Xpress (metric) - Sept 11

RAC 2012 fundamentals (Imperial) - Sept 11

RAC 2012 - Xpress (Imperial) - Sept 11
RST 2012 fundamentals (Imperial) - Sept 11
RST 2012 - Xpress (Imperial) - Sept 11

Inventor 2012 module 1 - Oct 11
Revit for Interiors 2012 - Oct 11
Revit Standards module 1 (community resource) - Oct 11
RST 2012 fundamentals (metric), new questions - Oct 11

AutoCAD Civil 3D 2011 (part 4) - Nov 11

Ecotect 2012 - Nov 11

Navisworks Manage 2012 - Nov 11
Revit for occasional users 2012 - Nov 11
RMEP 2012 fundamentals (metric), new questions - Nov 11

We'll also be adding 2012 data sets for the following titles (dates TBC):

Revit Content Creation
Revit MEP fundamentals (Mechanical) (Imperial)
Revit MEP fundamentals (Plumbing) (Imperial)
Revit MEP fundamentals (Electrical) (Imperial)
AutoCAD Civil 3D (parts 1-4)
3ds Max (modules 1-4)

So that's about it for the next few months. These dates might be subject to change, but this gives you a pretty good idea of where we're aiming to be by the close of the year. There are a few more ideas in discussion, including topics like Codebook, Bentley Navigator, MS Excel, MicroStation Select Series 1-3, Adobe Illustrator and USACE Attachment F. We'll keep you posted..

R

Revit Standards - update & comments

2011-07-22T04:34:00.000-07:00

OK, so we’ve had quite a few comments on this topic over the past week. Most in favour, but some who are having a hard time believing that a ‘generic’ pool of material can have relevance to the way they work. I accept both viewpoints, but ultimately my feeling is that it’s worth a try.

There are a number of firms who have agreed to pitch in and contribute a couple of questions, including, HASSELL, Opus, NBBJ, Capita, KEO, Cox, Woodhead, BDP, M+NLB, Jasmax, Zaha Hadid, HOK, Stantec, ARUP, Geyer, Levitt Bernstein, Pascall + Watson and FRCH.

If a handful of key people can find a spare couple of hours, over the next 5 or 6 weeks, we should have a sufficient volume of questions to comprise Revit Standards module one – Project start-up.

BTW, if anyone else wants to help, the proposition is pretty straightforward. This is a FREE community resource. (KS does not need to monetise this in any way. If it works out, the association with a useful, helpful, relevant Revit community initiative will be worth the effort). But there is a cost of entry, to join the club, if you like. To use banking parlance, in order to make a 'withdrawal', a firm must first make a 'deposit'. The cost of entry, to access the library of tools, is one new question, for each module in the Revit Standards series.

So we currently have 20+ firms writing a couple of questions apiece (either task based, or knowledge based), on any aspect of their choosing, on ‘good practice’ for project start-up, using Revit.

For each question, the required info is as follows:
- Question copy (i.e. name, summary, question wording, answer(s), training keywords) using a sample s/s template (which we can happily email across).
- Coaching notes, which either walk a user through the correct steps to successfully complete a task, or provide a line or two of background info, about the subject being covered in the question (and why it is important).
- Sample Revit file (if applicable, i.e. for task based questions). If possible, we have requested that any sample data sets are created using Revit 2012.

We've suggested a 'deadline' for the first cut of material, of end August. (I fully appreciate that everyone contributing to this experiment is super-busy, and doing this in their personal time, so we very much appreciate the willingness to participate).

We’ll then compile the questions into the KS system, present them in module format, and release the live content into the KS library. I would anticipate that firms will then make the appropriate edits locally, to make it fit for purpose in their own environment.

It has been interesting to read the range of comments which have filtered through on this subject. Here are a selection of (anonymised!) replies, both for and against.

Glass Half Full

"I think it would be a fantastic idea to have an assessment of someone’s understanding of either industry best practice or corporate standards. Essentially…’take this course to learn how we do things’…’now take a test to demonstrate you understand.’"

"This all sounds achievable - I'll programme in some time to complete. Am excited to be a part of this. There is definitely common ground and (global) best practice for all of those core areas - I'd love to test our teams here on Project startup process....we have one in place but no one bothers to use it, which in turn leads to a higher support component..."

"I am a strong believer in model managers and have previously had a check list for model managers to go through on project start-up and throughout the project. I could see this module of KS acting as a tool to help analyse the best person to perform this role."

"Quite spookily we’re currently discussing (the need for) a Revit Project Setup KnowledgeSmart test to ensure Revit project team members understand the ‘absolute must do’s’ before being allowed to start work on a live Revit project."

"Yes, yes yes yes yes!! Count us in. I agree that it’s a great idea and is definitely worth a try."

"Love the idea – I’ll give you 2 items."

"You asked “is there sufficient common ground, to enable the creation of a 'generic' set of Revit standards questions” and I think that the crowdsourcing approach answers that question perfectly. I suppose you can assume that if your most KS friendly firms can’t develop a standard set of questions, it probably can’t be done. But I think it can be."

"Coincidentally, this is something that (firm name) have been discussing recently – a test or checklist of some sort for project standards (setup, etc…). There are generic fundamentals prior to the more firm-specific things that all projects have to account for. We would argue that the more industry standards we can set now, the better off the industry will be in the future. CAD standards developed completely out of control and I would hope that we all learned a good lesson from that. . I love this concept!"

"Anything with a standard is tough. I think crowd sourcing assessment material is a great idea."

"I think this is a great idea, one of the main things being that once again this will be a collaborative effort, and bring the BIM community together to hopefully create a common best practice series of ideas that aren't company specific. I love the idea of open source approach."

"I really like the “crowdsourcing” idea for a Revit standards assessment. Our (leadership) group just finished up trying to create a “BIM assessment” – something that would tell us if people “get” how BIM is different from CAD. I think we’d all agree that it would’ve been great to have a pool of questions to start from and tweak, rather than start from scratch!"

"Your email is well timed, as we have just been discussing the very issue of standards, their implementation and assessment. I think the idea is a good one and with all new ideas, even if it isn’t perfect the first time, it very soon will become perfect as everyone chips in. I feel this will be invaluable and we are happy to participate."

"Anyone having spent time in Revit or in the Revit community knows that the attitude is a bit more open for sharing content and procedures than has previously been expected from other software packages. My observation is that while every firm has a twist on a standard, there are indeed commonalities for those best practices. The concept of sharing these should not be viewed with trepidation but embraced as a learning opportunity. I think testing on best practices is a good idea to ensure the data has been communicated well to the users. At the very least it will make them think about those best practices again to remind them it is a concern and is important."

"I think this is a very workable idea."

On The Fence

"Personally I really like the idea however I am a little dubious as to whether or not it will work. It’s hard enough getting a consensus about anything BIM related with just a handful of people, never mind opening it up to all. I agree that it does seem pointless everyone going through the same process but I kind of think this is unfortunately engrained into the architectural world so deeply that I can’t see it changing. The jury is out as to if the open standards would work.. my gut feeling is not.. I would love to be proved wrong!"

"Pulling together content, be it families, libraries, best practise in the way you have suggested is a great idea (assuming people are happy to participate)."

"I think there is something to be said for some well managed crowd sourcing. You have a massive task ahead trying to achieve this, so good luck!"

"This could have some legs, you just need to get the right people to submit questions... I will find some time to submit a couple to see where this goes."

"We use Revit standards internally based on the AEC (UK), Autodesk and other published standards and best practice guidelines and our BIM process methodology draws on the work of Penn State, Georgia Tech, the VA and buildingSMART. BIM standards are emerging, but there is still nothing definitive, maybe the US NBIMS with fill that gap. Like CAD standards of old, I believe that Revit standards will tend to be developed by companies internally, with reference to external standards. We have some very specific requirements which fit with our process and back office systems, so I couldn't imagine relying on external standards alone."

Glass Half Empty

"What makes a firm unique? Why you would hire one Architect over another is the unique way they design. Each firm comes to solving the problem of design in a different manner. To me the CAD/BIM world is just a set of tools we apply to the way we design. Those tools must be flexible enough to be applied to very different methods of design from conception to working drawings. The more we try to find standards the more we will restrict our different ways of designing."

"The idea is sound, but I suspect you won't find anyone who will actually put in the time. Crowdsourcing sounds great in theory, but the reality is that most people are too busy (or too selfish!) to share."

"Sorry, we don't have time to get involved in this. But keep us posted, as it sounds like an interesting idea, in theory, at least."

"A universal standard is the paramount end goal, yes. But do I think it's valuable for KS to get involved? Not really."

So there you have it. An interesting mix of reactions, but on the whole more positive than neutral or negative comments. Of course the 'proof of the pudding', is in whether people can actually free up the time to write a question or two. Any crowdsourcing initiative lives or dies by the momentum generated by a few key people.

I think back to last year's KA Connect, where Chris Parsons told the story of 'shirtless dancing guy', which provides an interesting view on the significance of momentum and the all-important, 'tipping point'. http://sivers.org/ff

So, here we are, in the field. Anyone care to dance? :)

R

Revit Standards

2011-07-15T06:20:00.000-07:00

There has been much discussion in recent months, surrounding the topic of common methods of 'good' working practices, in particular for firms using Revit. Earlier this week, I attended the Leeds RUG and enjoyed listening to visiting speaker, Neil Munro, from Opus NZ, talking about Revit Standards & Protocols.

Just a few weeks ago, whilst attending RTC AU, I sat in on an interesting presentation from the committee responsible for the Australia & New Zealand Revit Standards document. Here is a link to the ANZRS site, which is definitely worth a look: http://www.anzrs.org/blog.

And of course we have our own home-grown initiative, the AEC (UK) BIM Standard for Revit, which has been making impressive progress over the past couple of years, with a strong committee-led approach: http://aecuk.wordpress.com.

There is a useful discussion thread on standards to be found over on Revit Forum.org.

And this month, the Twittersphere has been ablaze with postings about the new Open Revit Standard, led by industry stalwarts, Sean Burke and David Fano, among others. For those Twitterers among you, search for the hash-tag #OpenRevStds, to follow the ongoing dialogue. And check out the new wiki-style site: http://openrevitstandards.com. This promises to be a fascinating initiative.

So I got to thinking.. as Revit is already a hugely popular topic for skills assessment material, how useful/feasible would it be, to create a series of KS modules which cover 'good' practices for Revit standards?

I'm thinking of modules covering the following core areas: Creating a Revit template file, Installation & setup of Revit, Project start-up process, Work sharing, Creating & managing Revit Families, Collaborative working, Draughting standards, Model management. Anything else?

And whilst I appreciate that such modules would naturally vary from firm to firm, is there sufficient common ground, to enable the creation of a 'generic' set of Revit standards questions, which could then be edited by individual firms, to wrap around their own in-house standards & protocols? Would it be too much to suggest that anyone starting a new Revit project, must first demonstrate to the team Principal or Job Captain, that they have taken (and aced) the project start-up module? That they do, in fact, understand key 'do's and don'ts' before being allowed to start work on a live Revit project.

Further, what if, instead of one individual author doing all the heavy lifting on this one, we employ the concept of 'crowdsourcing' the content, one module at a time? For example, if we start with Revit project start-up process and deliver a module of 20 questions on this subject. This makes more sense than 200+ firms all reinventing their own version of essentially the same wheel! In my experience, it is far easier to start with an existing set of data, than to write everything from scratch, each time.

So, to start the ball rolling - and to see if the concept has potential - what if we invite 10 KS-friendly firms, to write two questions each, on general Revit standards for project start-up?

My thinking is that this could be a 'free' resource, for any firms using Revit, who want to assess their users' understanding of in-house standards. The 'cost' of entry to the club, however, could be to submit two (or more) questions to the overall Revit Standards 'pool'.

We have a number of Revit specialists whom we can call on, to sanity check the content and generally ensure that it is fit for purpose, prior to release into the Revit Standards library.

We'll be speaking to the team behind the Open Revit Standard, in more detail over the coming weeks. It will be interesting to see if we can align any assessment content for Revit Standards, with the community-led initiatives currently taking place.

More to follow on this topic..

R

Revit Top Cat - Australia vs USA

2011-06-28T12:34:00.000-07:00

So, now that both RTC AU and RTC USA are finished for 2011, who walks away with the honours for the Top Revit Top Cat? The uber-Top Cat, if you like! :)

Let's examine the evidence..

In the heats, the overall benchmark for 76 assessments (AU - 36 / USA - 40), across all 3 disciplines of Revit combined, looks like this:

RTC ALL = 75% in 9 mins 22 secs.
RTC AU = 73% in 9 mins 27 secs.
RTC USA = 78% in 9 mins 23 secs.

So, first blood to the US of A!

How about drilling down to the 3 separate Revit flavours? Here are the stats for each one:

RTC AU
RAC = 80% in 8 mins 49 secs (24 results).
RST = 66% in 9 mins 27 secs (5 results).
RMEP = 56% in 11 mins 37 secs (7 results).

RTC USA
RAC = 79% in 9 mins 7 secs (28 results).
RST = 75% in 9 mins 45 secs (5 results).
RMEP = 76% in 10 mins 10 secs (7 results).

So, for RAC, AU users take the prize by a nose, but RST and RMEP comfortably goes to camp USA.

And so to the Grand Final. 4 heat winners from Australia, 3 from the USA. 10 questions on advanced concepts of Revit Project Process. These questions were pretty challenging! Here's how the finalists compared:

RTC Final ALL = 63% in 8 mins 5 secs.
RTC Final AU = 62% in 6 mins 47 secs.

RTC Final USA = 64% in 9 mins 48 secs.

So, extremely close, but the USA just about takes the prize. However, it is worth noting that the average time taken by the AU finalists was much faster. It's also interesting to note that all 7 finalists were men!

So what training issues did the data from the Grand Final highlight? Even though every single one of the finalists proved themselves to be Revit champions, there is still more to learn! Here are the top 12 training workshops, in priority order, from the Revit Project Process module:

Annotation, Key Schedules, Schedules, Tagging, Creating Views, Worksharing, Model Management, Clipboard, Families, Worksets, Local Files and View Types.

And last - the big one - who takes the title of overall Revit Top Cat?

Well, once again, the prize goes to the USA! 2 of the top 3 individual scores were posted by users from RTC USA. And the overall Revit Top Cat for 2011... Brian Mackey from CAD-1.

(Special thanks to Autodesk for providing a grand prize of an i-Pad 2, which Brian claimed as the overall winner. (Although the RTC grapevine suggests that Brian's better half, Dezi, has the inside track for who gets to actually use the i-Pad! :) )).

So there you have it, the gauntlet has been thrown down. The inaugural Revit Top Cat challenge has been won convincingly by our American friends. Can the Aussies prevail in 2012? Will the Europeans throw their parametric hat into the ring? Or perhaps a Top Cat Asia? (Rory has graciously volunteered to travel over for the Caribbean leg of the contest!). And perhaps a Grand Grand final, where the winners from each continent duke it out in a Winner-Takes-All Revit extravaganza?? The RTC Revit World Series? The possibilities are endless..

Revit Top Cat USA - The Results

2011-06-28T11:34:00.000-07:00

Last week, in the stunning location of Orange County, California, nearly 300 Revit aficionado's descended for 3 days of knowledge-sharing, teaching, learning, networking and a whole lot of fun. After several years in the planning, the inaugural Revit Technology Conference (USA) had finally arrived.

Just like at the parent conference (RTC AU) which took place in May, on Australia's Gold Coast, KnowledgeSmart was tasked with delivering the 'Revit Top Cat' contest. A practical skills assessment, pitching Revit top guns from across the industry, where only the bravest of BIM souls can prevail.

To recap on the format; Round One - a skills assessment comprising a mix of 14 knowledge and task based questions, using Revit 2011 across 3 streams (Architecture, Structure or MEP).

The Grand Final - where the 3 candidates with the best scores from each stream go head to head, tackling 10 advanced questions about Revit Project Process.

Congratulations to the following victors, from each heat (pictured left to right):

Charles Jolin - Revit MEP - 96%
Brian Mackey - Revit Architecture - 100%
Priit Luhakooder - Revit Structure - 100%

And so to the finale. (And to give the proceedings even more spice, two of the three contenders were representing Autodesk ATC's!). After 10 spine-tingling minutes, a new champion was crowned. And the winner of Revit Top Cat USA 2011 is.....

Brian Mackey from CAD-1. (Editor's note: er, shouldn't that be BIM-1, Brian? :) ).

So let's take a look in more detail at some of the results from the competition..

The overall average for the 40 results was 78% in 9 mins 23 seconds.

Drilling down a bit more, there were 28 results for RAC, 5 for RST and 7 for RMEP.

The average scores for each discipline were:

RAC = 79% in 9 mins 7 secs.
RST = 75% in 9 mins 45 secs.
RMEP = 76% in 10 mins 10 secs.

So how did the boys vs girls stats look, this time around? Well, unlike the RTC AU stats, this time the boys took the title, in the Revit battle of the sexes!

Boys = 80% in 9 mins 18 secs.
Girls = 66% in 9 mins 50 secs.

So what does all this mean, in terms of performance and training?

For Revit Architecture, the top 10 training items were, in priority order:
Conceptual Massing, Shortcuts, User Interface, Dimensions, Principles of BIM, Coordinates, Shared Locations, Phasing, Floor Elements and View Properties.

For Revit Structure, the top 10 training items were, in priority order:
Brace Symbols, Structural Settings, Repeating Details, Drawing Lists, Project Information, View Properties, Visibility/Graphics, Beam Systems, Copy/Monitor and Crop Regions.

For Revit MEP, the top 10 training items were, in priority order:
Pressure, Pipe Properties, Pipe Types, Fixture Units, Plumbing Fixtures, Routing Duct, Duct Fittings, Duct Sizing, Duct Systems and Properties Pallet.

Overall, an extremely enjoyable conference. Special thanks to Wes, Heidi and the RTC team for all their hard work, over the last year. Thanks to the RTC AU committee (Clay Hickling, Michelle "Disco Diva" Leonard, Rodd Perey and Simon "90% even when blotto" Whitbread) for making the trip. And thanks to the RTC USA committee (Jim "Gadgets" Balding, Bob "Saturday Night Fever" Bell, David "Blondie" Harrington, Nicholas Kramer, Steve "Keith Moon" Stafford and Phil Read) for all their efforts in hosting a knockout first conference. Looking forward to RTC USA 2012 (somewhere on the East coast!).

R

Revit Top Cat AU - The Results

2011-06-01T04:05:00.000-07:00

In the beautiful setting of Australia's Gold Coast, last week saw the gathering of almost 500 enthusiastic Revit professionals, for 3 days of learning, networking and general BIM shenanigans. The seventh annual Revit Technology Conference was, in short, a tremendous success.

As part of the proceedings, KnowledgeSmart organised a practical skills contest for Revit gurus - from all 3 disciplines - to demonstrate their Revit knowledge and credentials. And so 'Revit Top Cat' was born. (Recognition must go to RTC committee members Rodd Perey and Wesley Benn for their role in conceiving and naming the competition!).

The format was straightforward. Round one - a skills assessment comprising a mix of 14 knowledge and task based questions, using Revit 2011 across 3 streams (Architecture, Structure or MEP, as appropriate).

The candidates with the best score from each stream would qualify for the grand final on Saturday. In the event of a tie for best score, the user with the fastest time gets the nod.

After some technical challenges (it's tricky to run a web-based assessment when you have no web! :) ), we eventually got everything set up.

36 brave souls stepped up over the course of the Friday afternoon, going toe-to-toe with their fellow Reviteers. The results were tallied up and on Saturday morning the winners were announced, ready to take their place in the final.

Congratulations to the following victors, from each heat:

Andrew Bagnall - Revit MEP - 93%
Glenn Sinclair - Revit Architecture - 96%
David Harrington - Revit Structure - 100% *
Adam Sheather - Revit Structure - 100% *

(* David's time was a bit faster than Adam's, but as they both did so well, and we had an extra PC, Adam was also invited to take his place in the final.).

And so to the finale. A new assessment module, comprising 10 questions about Revit project process. Not for the faint hearted! After 10 tense minutes, a new champion was crowned.

And the winner of Revit Top Cat 2011 is.....

Andrew Bagnall from GHD. Many congratulations to Andrew and indeed the winners from all 3 streams!

So let's take a look in more detail at some of the results from the competition..

The overall average for the 36 results was 74% in 9 mins 17 seconds.

Drilling down a bit more, there were 24 results for RAC, 7 for RMEP and 5 for RST.

The average scores for each discipline were:

RAC = 80% in 8 mins 49 secs.
RST = 66% in 9 mins 27 secs.
RMEP = 56% in 11 mins 37 secs.

At the risk of courting controversy, we thought it might be interesting to see how the boys compared to the girls. You could argue that this is an unfair comparison, as there were more scores posted by the boys than the girls overall, however a counter argument might be made that this was also reflective of the wider demographics at the conference. Either way, we think it's a relevant metric, so here it is... it turns out that the girls comprehensively beat the boys - for both accuracy and speed - so no great surprises there! :)

Girls = 84% in 8 mins 40 secs.
Boys = 72% in 9 mins 33 secs.

So what does all this mean, in terms of performance and training? Well, not surprisingly, the overall level of performance, across the 3 disciplines, was very high. But you might expect that to be the case at a user conference.

So what were the main training issues that were highlighted, in a detailed analysis of the results..?

For Revit Architecture, the top 10 training items were, in priority order:
User interface, Conceptual Massing, Shortcuts, Principles of BIM, Dimensions, Phasing, Coordinates, Floor elements, Shared locations and View properties.

For Revit Structure, the top 10 training items were, in priority order:
Drawing lists, Project information, Brace symbols, Crop regions, Roofs, Structural settings, View properties, Visibility/graphics, Repeating details and Floors/slabs.

For Revit MEP, the top 10 training items were, in priority order:
Plumbing, Fixture units, Pressure, Pipe sizing, System inspector, Pipe properties, Pipe types, Duct sizing, Duct systems, and Creating supply systems.

So, overall a thoroughly enjoyable conference, with many new friends made along the way. A robust contest, across the three main disciplines of Revit. The structural engineers posted the only maximum scores in the heats. MEP engineers won the overall contest. And the girls comprehensively beat the boys, in what may now turn into a Revit battle of the sexes! And it all began in a small town on Australia's Gold Coast...

R

RTC - Revit Top Cat

2011-05-19T00:32:00.000-07:00

The seventh Revit Technology Conference takes place next week, on the Gold Coast, Australia. (I know, there could be worse places to spend 3 days learning about Revit & BIM, right?).

In the lead up to the show, KS has been working with the RTC Committee to arrange a fun contest, for users to demonstrate their Revit credentials, flex their modelling muscles - and prove beyond doubt who is the top player in town.

In fact (using a play on words from AUGI's well established 'Top-Daug' AutoCAD quiz format) we're calling this year's contest - Revit Top Cat (RTC, geddit? :) ).

Here's how it will all go down..

On Thursday and Friday, there will be 6 PC's set up with copies of RAC, RST and RMEP 2011. Users will have 10 minutes (12 mins for MEP) to answer 14 questions about their chosen Revit discipline.

The winners from each stream, on both days, will qualify for Saturday's Grand Final. (Here's where we really separate the Revit men/women from the boys/girls!). In short, best score combined with fastest time wins each heat.

On Saturday, we'll have our 6 finalists lined up to take the new Revit Project Process test. (NB This is the first of our new 'level 3' modules - and it's not for the faint hearted! In beta testing, we had more than one Revit veteran literally run screaming from the room whilst taking this test. Well, OK, not really, but it is a challenge!).

So, the Grand Final. 10 questions in 12 minutes - and the new Revit Top Cat will be crowned.

As well as some cash prizes for the finalists - the overall winner will have the kudos of being recognised as the most renowned Reviteer on the planet! (Or at least the Gold Coast!).

So, do you have what it takes to be the RTC AU - Revit Top Cat??

See you at the show..

R

Nightingale & Evolve case study

2011-05-10T06:31:00.000-07:00

When I speak to firms about measuring and improving CAD & BIM performance, there are a number of factors which come into play. The fact is that it takes a lot of hard work, management focus, commitment, the right tools, strong technical leadership and reliable consulting advice - to bring it all together.

One such example is the work being undertaken by the team at Evolve (www.evolve-consultancy.com) - a leading UK BIM consultancy - and healthcare specialists Nightingale Associates, under the technical leadership of Chris Seymour-Smith.

Take a look at the full story here: http://www.evolve-consultancy.com/article.php?incat_id=161&category_id=38.

Over a two year period, the firm improved overall accuracy by 21%, with nearly half of their Architects posting above average or excellent scores. Further, over 40% of their users raised their game individually, by 25% or greater. That's a remarkable achievement!

R

KS sponsors RTC AU

2011-05-05T01:58:00.000-07:00

KnowledgeSmart has signed as a sponsor of the seventh Revit Technology Conference, which takes place later this month, on Australia's Gold Coast.

This will be our second trip down under, to support this event. It's a great conference, covering all things Revit and BIM, run independently of, but with full endorsement from, Autodesk.

The conference has been growing steadily in size and stature, over the past six years, to the point that this year also sees the inaugural RTC USA later in the summer.

Many of the industry's top Revit trainers and authors make the trip, to present classes and workshops, including, Paul Aubin, Robert Manna, David Harrington, Chris Price, Simon Whitbread and Phil Read.

KnowledgeSmart will be showing the latest web based skills assessment tools, for Revit and a range of other CAD, BIM and Engineering software titles.

Full details can be found here: http://www.revitconference.com.au.

R

KS Admin Help Notes

2011-04-28T03:18:00.000-07:00

Here is a short guide for KS administrators, which shares some tips from other KS customers, on how to prepare for a skills assessment rollout.

Communicate your plans

It is important to explain to users precisely why you are asking them to undertake a skills assessment. You might do this by including the assessment in a wider discussion about training needs and more efficient use of the overall CAD & BIM infrastructure within your company. This could include training needs, recruitment, induction and annual appraisals processes.

Improving technical software skills is a continuously improving process – the KnowledgeSmart tools simply highlight a likely starting point. As with most things, good communication goes a long way. An effective strategy is to hold a team meeting and explain your overall plans for improving CAD & BIM performance within the firm. This can include a Q & A session, where people have an opportunity to clarify any aspects of the assessment process – and understand how it fits into the bigger learning picture.

It is important that people don’t feel the results of their skills evaluation may be used in a negative context, particularly if they don’t perform well first time around. Ensure they understand this is a designed to be a learning process – and everyone has to start somewhere. If you have a CAD or BIM software upgrade scheduled, this can also feed into the process. By taking a snapshot of each individual’s current skills at a basic level, you can feed this data into your training plans moving forwards.

Evidence shows that better overall understanding of basic commands can also benefit the implementation of in-house Standards. Once you have explained to your teams the business case for measuring and improving CAD & BIM productivity, you need to set a timeframe for rolling out your assessments.

Prepare a timetable

You need to decide over what period of time you want to complete your assessments. This will depend on 4 variables; 1) the number of people to be assessed 2) the number of PC’s you can make available (with a licensed copy of CAD or BIM software and a network connection) 3) the number of offices in your organisation and 4) general day-to-day work pressures.

You need to decide what your preferred assessment environment will be. Some companies prefer their teams to use their own work stations, whilst others favour a more formal setting. In truth, there is no right or wrong answer to this question; you want people to feel comfortable and remove any potential for apprehension or pre-assessment nerves.

Experience shows that a formal assessment environment makes for a smoother rollout, where people simply take an hour or so at a specified time to sit their assessment. This way, they aren’t distracted by project work, email, ringing phones or interruptions from colleagues.

Use the invite tools in the KS dashboard to manage your assessment programme. For multi-office firms, it is best to start with one office and, when this office is done, gradually ramp up the level of activity until the assessment programme is complete. Principal admins will need the support of regional project technology co-ordinators if this is to be a success.

Prepare your team

We recommend you brief your teams prior to their assessments. You might like to share with them a simple Q & A document or present your plans at a team meeting. Here's an example of what other KS customers have done: http://the-knowledgesmart-blog.blogspot.com/2011/04/sample-ks-candidate-q.html.

Here's a brief user guide to taking a KS assessment: http://the-knowledgesmart-blog.blogspot.com/2011/04/taking-ks-assessment.html.

It is best to circulate the information at least a few days before people are due to take their assessments.

Send your invites

Use the invite tools in the KS dashboard to send your test invites. You can send invites individually, or add a list of users via csv import and invite them as a group.

Here's a brief blog article about sending invites: http://the-knowledgesmart-blog.blogspot.com/2011/04/inviting-users-to-take-ks-test.html.

Next, you may customise your welcome message in the invite mail and on the test landing page. You might wish to present users with specific instructions on where to locate their sample test files (if you have chosen to store them on your network, for example).

Here's a brief blog about editing your invite text: http://the-knowledgesmart-blog.blogspot.com/2011/03/customising-your-test-invite-text.html.

Complete your assessments

You can monitor your team's progress, as they complete their assessments. Admins receive an email with a link to test results, as and when they are completed.

You can use the invite history page of your dashboard to track invites and completed assessments. You can also re-send invites from here.

After the assessments

At the end of their assessment, users may receive an instant result and report with a breakdown of their score, training keywords and coaching notes. In addition, they can be sent an email containing a link to their summary report.

Account admins can access all assessment results, feedback, training keywords and coaching notes – for each candidate – in the KS dashboard.

Via the dashboard, it is possible to search results, view performance charts and export data into Excel for in-house reports. It is important to provide the opportunity for individuals to access a structured training programme, after their skills have been evaluated. The results will highlight areas where future training workshops can be focused for maximum improvement.

Here's an 8-step guide to better training needs analysis: http://the-knowledgesmart-blog.blogspot.com/2011/04/using-ks-for-training-needs-analysis.html.

R

Sample KS Candidate Q & A

2011-04-27T15:08:00.000-07:00

It is important for AEC firms to explain to users precisely why they are being asked to take a skills assessment. The key message is that the KS tools are an aid to identifying skills gaps and highlighting training needs.

Here are some sample notes that KS admins can copy - and adapt for their own use - to help give their teams a bit more background to the KS assessment process.

What is KnowledgeSmart?

KnowledgeSmart is a leading developer of skills assessment tools for CAD, BIM and engineering software applications – designed to help AEC firms measure and improve productivity. Since 2003, KnowledgeSmart has facilitated in excess of 20,000 individual assessments – across a range of architectural and engineering practices globally – helping to create a benchmark for basic CAD & BIM proficiency.

The KnowledgeSmart tools perform a comprehensive assessment of a user’s ability focusing on fundamental skills groups associated with design and engineering software applications, including AutoCAD, Revit, 3ds Max, Civil 3D, Rhino, SketchUp, InDesign, Photoshop, MicroStation and Bentley Architecture. The process aims to assess current performance in order to identify areas where individuals can benefit from additional training.

What is the point of KnowledgeSmart - and why should I do it?

This process is not about a big-brother style test to check up on unsuspecting staff! Nor is it a matter of pointing out who is the more knowledgeable software user. Rather, the point of the exercise is to highlight how much each individual currently knows – in order to identify those areas where focused training can improve basic knowledge.

Everyone uses CAD, BIM and engineering software in a different way. Some people use it full time in their role; others may only use the tools from time to time. The objective is not to teach a veteran CAD user how to use a polyline or write text. But the fact remains, a more detailed knowledge of basic commands and operations can alleviate apprehension in using technical software applications and assist in speeding up routine tasks.

Is it the same for full-time and casual users?

There are a number of different test modules to choose from, covering a range of skill levels and created by a team of specialist authors. Firms can also write their own test material in-house. For basic CAD skills, as an example, practices can choose from a 2D fundamentals module or an ‘occasional users' module.

The fact is, there are many differing levels of ability and many different requirements placed upon the way CAD & BIM software is used within each practice.

Summary reports take into consideration whether an individual is a core user, or someone who only uses the software occasionally as part of a wider role. Follow-up comparisons and training recommendations should be reflected accordingly.

What topics does the assessment cover?

KnowledgeSmart assessments typically break down technical software applications into a series of modular topics.

For example, the 2D fundamentals assessments (for AutoCAD & MicroStation) cover a range of basic CAD commands, including; basic object/element creation, layers/levels, blocks/cells, annotation, referencing & printing, UCS/ACS and preferences.

Assessments typically comprise 20-25 questions and present scores as a percentage, based on the accuracy of the answers presented. There are 2 question types and 5 question styles. Question types are either task-based (i.e. you need to perform a task using the software in order to work out the answer) or knowledge-based (i.e. you are being tested on your understanding of certain software functions, without having to actually use the tools). Question styles include; free text, multiple choice, pick list, order list and true/false.

What if I do not score very well?

This exercise is not about who is the best. You will identify areas where you can improve your own performance with the help of more targeted training workshops.

Following the assessment everyone receives an overall result and summary report with feedback on dropped marks, training keywords and coaching material.

When and where will the assessments take place?

Your technology administrator will have details of your assessment schedule. If you are unavailable to attend on the date or time scheduled please contact your administrator to rearrange. You will need a PC or laptop, with a web connection and a copy of the software you are being tested on.

How long will the assessment take?

There is no limit to how long the exercise will take. The average time is approximately an hour, but this is not a strict rule.

Try to work at your usual speed. Don’t rush, as many simple mistakes are made by people not reading the instructions carefully. Conversely, don’t take too long to complete each task, as your score plus time taken will be looked at in the overall results analysis. Your score will not be affected by the time you take.

How long before I know my results?

Depending on the level of feedback assigned by your administrator, you will typically receive your score straight away when you finish the assessment. You will also be emailed with a link to your results report and feedback on any marks which have been dropped.

Additional Info

The amount of time you take will not affect the accuracy of your score.

Before you start, you may be presented with a zip folder containing files which you’ll need to refer to during your test. Save the zip folder locally on your machine – and remember where you put it – as you will need to navigate back to this folder during your assessment.

Please read the written instructions carefully. This is also an assessment of your ability to follow instructions. Often, marks are dropped through candidates simply not doing what they have been asked to do.

You can answer the questions in any order, using the KnowledgeSmart 'question navigator', or simply read through the test instructions before attempting to answer each question. You can also edit your answers at any time during your assessment.

The time taken to complete an assessment varies depending on the individual, but an average is about an hour. You don’t have to complete all the questions in one sitting; you can exit and return later if you prefer. When you resume, your previous answers will have been saved.

Here’s a link to a KS blog article which tells you more about what to expect during a KS assessment: http://the-knowledgesmart-blog.blogspot.com/2011/04/taking-ks-assessment.html.

R

Revit - 3 levels of Proficiency

2011-04-27T05:22:00.000-07:00

Based on a number of similar discussions in recent months, we're introducing a subtle change to the labeling of some of the KS assessment modules.

I attended a web meeting this week with the Head of Model Development for a major UK construction client. Like many AEC firms, they are wrestling with the challenge of making BIM work for their business, whilst finding the cost savings, efficiency gains, and so on, that all big businesses have to make these days.

Looking back to the KS user group in London, a couple of months ago, a similar topic arose as part of the general discussion. That is, how do we identify someone who has basic skills in, say, Revit and distinguish them from an advanced user?

Further, do AEC firms actually need all of their users to be 'specialists' in Revit? The simple answer is no, in my opinion. In many instances, users just need to receive a file, open it up and perform some straightforward analysis of the model (without breaking anything!). For example, construction managers, PM's and the like. Then, for the general modeling team, AEC firms require a broader level of understanding for their wider user base. The people creating the detailed production information on a project. And last, firms need to have a handful of Revit 'specialists'. People with a deeper understanding of the way in which Revit is deployed on a project, an appreciation of the collaborative process and the impact of using the technology across a broader environment. And maybe, if content creation is a key part of their business, a handful of Revit Family specialists, who understand the importance of creating quality content, that can be re-used on subsequent projects, not simply re-modelled from scratch every time.

So it makes sense to establish 3 recognised levels of competence;

Level 1 – basic skills
Level 2 – intermediate skills
Level 3 – advanced skills

With respect to Revit, this will work as follows;

L1 – Basics of Revit (i.e. a test aimed at Construction Managers, PM’s, etc.)
L2 – Revit fundamentals
L3 – More advanced concepts of Revit, i.e. Revit project process, collaborative working, Family creation, etc.

AEC firms can use the assessments, as a means of demonstrating to clients (and prospective clients) that they have the skills required, to truly deliver on a BIM project. For example, all CM’s can be trained and encouraged to take the Revit L1 assessment (scoring X% or higher in Y mins or less). All Revit modelers can add the Revit L2 assessment to their CV (scoring X% or higher in Y mins or less). All Revit ‘specialists’ need to ace the L3 assessment(s). Anyone working on content creation, must tackle the families assessment. You get the picture.

Firms might also consider writing their own modules, based on their BIM Standards. All users can demonstrate an understanding of the correct standards, by passing this module.

Now, we've made extremely good progress in building our Revit assessment library. We already have the following modules in place:

- RAC 2010 fundamentals (Metric & Imperial) – L2
- RAC 2011 fundamentals (M & I) – L2
- RAC Xpress (M & I) – L2
- RST 2010 fundamentals (M & I) – L2
- RST 2011 fundamentals (I) – L2
- RST Xpress (M & I) – L2
- RMEP 2011 fundamentals (Mechanical) (I) – L2
- RMEP 2011 fundamentals (Plumbing) (I) – L2

We are currently working on the following new modules:

- Revit Project Process (goes live early May) – L3
- Revit Families (goes live late May) – L3
- RMEP 2011 fundamentals (Electrical) (I) (c.June) – L2
- RAC 2012 fundamentals – 13 modules (M) (c.June) – L2
- RMEP fundamentals 2012 (M, E & P) (I) (c.July) – L2
- RST 2012 fundamentals – 13 modules (M) (c.August) – L2
- RMEP 2012 fundamentals – 13 modules (M) (c.September) – L2
- Revit for occasional users (i.e. general content for PM’s, CM’s. Not discipline specific). (M&I) (c.September) – L1

In the Fall, we'll be adding these modules:

- RAC 2012 fundamentals – 6 additional modules (M) – L2
- RST 2012 fundamentals – 6 additional modules (M) – L2
- RMEP 2012 fundamentals – 6 additional modules (M) – L2
- Imperial data sets for all Revit 2012 content – L2

- RAC 2012 advanced (M & I) (i.e. harder test than fundamentals (L2) for core users) – L3
- RST 2012 advanced (M & I) (i.e. harder test for core users) – L3
- RMEP 2012 advanced (M & I) (i.e. harder test for core users) – L3

I think this will help AEC firms, Construction clients and contractors alike, to get a much better handle on who knows what, with respect to deploying Revit on a BIM project. It will also help firms to demonstrate their BIM credentials and give users a logical (and more appropriate) path for skills development.

R

Taking a KS Assessment

2011-04-20T08:38:00.000-07:00

It's extremely important for AEC firms to explain to their users precisely why they are being asked to take a technical software skills assessment. The best firms always handle this well. As with all things in life, a little bit of good communication goes a long way.

This is not all about who is the smartest, fastest, or best Revit user. Or the most experienced MicroStation wizard. Although these are useful things to know. The real value in the KS tools is trying to help users, at an individual level, identify what they know and also what they need to brush up on next, in order to improve their skills, with a particular piece of software.

Over the past 8 years, we have facilitated over 20,000 individual test sessions, across hundreds of AEC firms. Experience has shown that, in general, people don't like the idea of taking a test - at least that's usually how they feel at first. However, experience has also shown that, for most people, when they complete their assessment, they realise that it wasn't half as bad as they thought it would be! Further, they often learn new skills as they go through the process of answering the test questions. And lastly, they find the test feedback and coaching notes extremely helpful, in giving them an unbiased appraisal of their skills using a particular software application - relative to their peers in industry. It's important that users have the opportunity to review their test scores and feedback with their team leader or BIM manager. And it's also important that firms explain to their users that all test scores are confidential. Best practice would recommend that only senior administrators should have access to firm-wide results data.

Provided firms explain to their teams that the main point of rolling out an assessment program, is to gather valuable training needs analysis data, rather than a big-brother style exercise in peeking over shoulders, then most users feel OK about the idea. And when firms follow through with more focused, modular training workshops, which target recognised skills gaps, then it's a win-win for users and firms alike.

So let's take a look at the user journey, when asked to complete a KnowledgeSmart skills assessment.

We'll assume the firm has already done the pre-assessment prep - and explained to their teams what is happening and why they are being asked to take an assessment.

The next step is for a user to receive their test invite. (Here is a separate article on how a KS admin sends out test invites: http://the-knowledgesmart-blog.blogspot.com/2011/04/inviting-users-to-take-ks-test.html).

Admins can set up test sessions straight from a browser, but most invites are sent by email. Either way, the user ends up in the same place. Here is a typical KS test invite:

The system assigns a username and password to each user. They access their assessment by clicking the unique URL at the bottom of their invite mail. This takes them to the assessment login page, where they will be prompted to enter their username and password. (If a user loses or forgets their password, they can hit the 'Forgot details?' link to receive a new one).

When they log in, they arrive at the main assessment landing page. Here they will see a summary of the test they are scheduled to take and any special instructions from their administrator.

There are 5 types of questions, which typically comprise a KS assessment. Let's take a brief look at each one..

Free Text
This type of question requires a specific answer, which needs to match the answer stored in the KS database. Look out for precise instructions on how to format your answer, for example, number of decimal places, case sensitive answers, accurate spelling, and so on.

Multiple Choice
Here, you will be offered a selection of possible answers - and you need to select the best one. Watch out for 'distractor' entries, which are designed to make you think before selecting your answer.

Pick List
This question type offers you a range of options, which could represent a correct answer. Choose the best answers from the list provided, using the check boxes, to the left of the answers. Partial marks are often awarded for entering some of the correct answers on pick list questions.

Order List
This question type requires you to place a number of items in the correct sequence. Drag and drop the answer options, until they are in the right order.

True or False
This question type is pretty straightforward! Read the statement or question and decide if you think it is true or false. Enter your answer using the radio buttons.

OK, so when the user arrives at the landing page, before they begin, they need to download the sample data files that they will be using during their assessment. There are a couple of ways they might have to do this. The usual way is to select the orange, 'Download files' button.

This prompts the user to save the files locally on their work station. They will need to navigate back to this folder during their assessment, so most users choose their desktop, or My Documents, etc. The data sets do not need to be saved during a test session, so they can simply be deleted after the assessment has been completed.

Sometimes firms choose to zip up all sample test files and host them locally, on their own servers. If this is the case, then the orange button will not display. Instead, the admin will provide special instructions on the assessment landing page, explaining which file path to follow, in order to find the sample data sets.

Either method is fine - and the user will achieve the same result! They simply need to know which folder to navigate to, in order to access the files for their randomly generated set of test questions.

When they have their sample test files and they have read any other instructions from their administrator, they can hit the green 'Start' button and their assessment will begin.

Each assessment is timed, although the time taken to finish does not impact the overall score. The test user interface (UI) is easy to navigate. All test sessions start on question 1. There is also a small timer, which allows users to keep track of how long they are taking during their session.

Each question has a name and brief summary, which explains to users the main features of the software being addressed.

If a question is task-based, i.e. the user has to use the software in order to answer the question, then the corresponding sample data set can be found in the zip folder they downloaded at the start. The orange instruction box explains where to find the files for each question.

Users can answer the test questions in any order. The 'Question Navigator' allows users to move around the questions and generally familiarise themselves with the exercises. The simple colour code helps users to keep track of their progress. Green squares indicate questions for which an answer has been submitted. Orange represents the current question. Red squares are questions that have yet to be attempted.

If a user makes a mistake, they can re-visit the green squares at any time, to edit their answers. A test session finishes when all the questions have been answered. Users have one last opportunity to review and edit their answers, before hitting the green 'Finish' button, which triggers their assessment to be graded.

(Sometimes, an admin will set a time limit for an assessment, in which case the session will automatically close after that time has elapsed. Answers submitted will be marked and any questions which have not been attempted will be scored as zero).

When the session is finished, the user may receive a test report, with feedback on any dropped marks. The level of feedback received can vary, depending on the settings prescribed by the account admin.

A helpful summary of the test questions, feedback on incorrect answers, coaching notes and training tags may be viewed by the user, by selecting the 'Show question detail' link at the top of their summary report.

The training tags for questions where score = < 100%, form a personal curriculum of suggested training workshop topics.

R

Organising your KS Library

2011-04-08T01:35:00.000-07:00

After a while, you will probably find that your KS test library starts to grow. This is particularly relevant for our 'Enterprise' customers, who have access to a wide range of 'off the shelf' KS test topics. Further, many firms are writing their own test questions, modules and tests, which adds even more content to your library. So how do you query your questions, manage your modules and tame your tangled test titles? In short, how do you organise your KS content library?

All tests in the KS 'off the shelf' library are locked down, i.e. you can't edit them or modify them in any way - whilst they remain in your OTS library page. (This is the page in your dashboard called, 'KnowledgeSmart tests').

Managing your Tests

When you want to edit a KS test, the first step is to create a copy of the original. To do this, go to the 'KnowledgeSmart tests' page of your dashboard, select the test you want to copy and click the 'Import to your account' tool.

This creates an exact copy of the test, which appears in the 'Your Tests' page of your dashboard. You'll notice that your copy has a different ID number from the original. Every test in the KS system has a unique ID number, so you can distinguish between various copies and easily manage your library material.

Your tests display in A-Z order on this page. When an OTS test is 'imported' into your account, this unlocks the test and you can now modify and customize many features of the test.

The quickest way to sort your tests is to use the ordering tools at the top of the list; these quickly arrange your tests in ascending (or descending) order by ID number, or A-Z (or Z-A) order by test name.

To find out more about creating and editing KS tests, take a look at this separate blog article:
http://the-knowledgesmart-blog.blogspot.com/2011/03/creating-and-editing-ks-tests.html.

Managing your Modules

When you create a copy of a KS OTS test, you also create a separate library of the individual component modules, which comprise the overall test. These will appear in the 'Your Modules' page of your dashboard.

When you import several tests into your account, you will find that your KS modules library pretty quickly grows to several pages of content. You have a number of content management tools available, to help you filter, organize and manage your modules.

Unlike your tests, which list in A-Z order, your modules will be listed in the order that they were imported into your account. The quickest way to sort your modules is to use the ordering tools at the top of the list; these quickly arrange your modules in ascending (or descending) order by ID number, or A-Z (or Z-A) order by module name.

If you want to filter your KS module library, you can use the searching and grouping tools. You'll find these tucked away in the orange bar, at the top of the page.

Hit the 'Show search' link to expand the searching tools box.

You can search on module name, module ID number and also by category tag. When a module is created, we assign a number of keywords, which describe the main subject areas being covered by the module. You can use these keywords to search and filter your modules.

When you have run a search and located the module(s) that you need, you can use the grouping tools, to create sub-sets of your module library.

Highlight the module(s), using the check boxes to the right of the module names. Create a new group, i.e. Revit MEP, in the groups table.

Then select 'Create' and your modules will be copied across to the new group.

In this way, you can create a number of sub-groups for your overall module library. When you want to filter your library, simply use the 'View group' dropdown to navigate to the required module(s).

You can also use the grouping tools to delete groups, edit group names, add new modules to existing groups and remove modules from existing groups.

To find out more about creating and editing KS modules, take a look at this separate blog article:
http://the-knowledgesmart-blog.blogspot.com/2011/03/creating-and-editing-ks-modules.html.

Managing your Questions

When you create a copy of a KS OTS test, you also create a separate library of the individual questions, which in turn comprise the modules, which combine to make the overall test. These will appear in the 'Your Questions' page of your dashboard.

When you import several tests into your account, you will find that your KS question library pretty quickly grows to several pages of content. You have a number of content management tools available, to help you filter, organize and manage your questions.

Like your modules library, your questions will be listed in the order that they were imported into your account. The quickest way to sort your questions is to use the ordering tools at the top of the list; these quickly arrange your questions in ascending (or descending) order by ID number, or A-Z (or Z-A) order by question name.

If you want to further filter your KS question library, you can also use the searching and grouping tools. You'll find these at the top of the page.

You can search on question name, question ID number and also by category tag. When a question is created, we assign a number of keywords, which describe the main subject areas being covered by the question. You can use these keywords to search and filter your questions.

When you have run a search and located the question(s) that you need, you can use the grouping tools, to create sub-sets of your question library.

Highlight the question(s), using the check boxes to the right of the question names. Create a new group, i.e. AutoCAD 2D, in the groups table. You can use the 'select all' check box at the top to highlight all questions on the page. Each page displays 20 questions, so might need to navigate several pages, in order to capture all the questions you need for your new group. You can select check boxes across multiple pages.

Then select 'Create' and your questions will be copied across to the new group.

In this way, you can create a number of sub-groups for your overall question library. When you want to filter your library, simply use the 'View group' dropdown to navigate to the required question(s).

You can also use the grouping tools to delete groups, edit group names, add new questions to existing groups and remove questions from existing groups.

As an additional aid to help you manage your question library, when you import a KS test into your account, the system automatically creates a sub-group of the related questions for each test. You'll find these sub-groups by using the 'View group' dropdown.

To find out more about creating and editing KS questions, take a look at this separate blog article:
http://the-knowledgesmart-blog.blogspot.com/2011/03/creating-and-editing-ks-questions.html.

One final point; any benchmark data published by KS will only relate to the original OTS tests. When you start making changes to your test library, we lose the ability to make a reliable comparison, from a benchmark point of view.

R

Inviting Users to take a KS Test

2011-04-07T15:31:00.000-07:00

Administering the KS tools is a pretty straightforward process. The system is web-based, so there is no software to install, maintain and upgrade. In general, it's a low-impact system, from an IT administration point of view.

Let's take a look at the process for inviting users to take a test. There are three different options, depending on your requirements. Let's take a look at each one..

Option one - inviting a small number of users to a test

Step one
Select the test you want your user(s) to take, from your library. Each test has a small 'envelope' icon, next to the test name.

Click on the envelope icon, which takes you to the test invite page. Your selected test will automatically appear in the test list dropdown.

Alternatively, you can select the INVITES link, from the top navigation bar of your dashboard. (You'll end up in the same place). However, bear in mind that if you do use the top navigation route (rather than the envelope icon next to a specific test title) then your required test will not automatically get selected in your dropdown. (It presents your test library in A-Z order, starting with 3ds Max). If you don't select the test you want from the dropdown, before you send out your invites, then your users will receive an invite to a 3ds Max test! :)

Step two
Enter the email, first name and last name of your test taker(s) in the fields provided. You can also capture user status, i.e. employee, interviewee, student, etc. Click Add to confirm your candidate's details.

Step three
Decide on your test invite text. You have a couple of different options for customising your invites. Here is a separate blog entry on this topic: http://the-knowledgesmart-blog.blogspot.com/2011/03/customising-your-test-invite-text.html.

Step four (optional)
You can set an expiry date on your test invites. After this date has passed, your users will no longer be able to access this test session. If they haven't completed their test before the expiry date, you will have to issue a new test invite.

Step five
OK, so you have completed the following steps; selected the correct test title from your library; entered all your user data; edited your invite text; maybe set an expiry date. The final step is to hit the Send Email button. This step prompts the following routines to take place, in the background; it generates your users' profile(s) on the KS system, adds them to your user list (in the 'Users' page of your dashboard), creates a username/password for each user and sends out their individual test invite emails.

Option two - inviting a large number of users to a test

You have the option to batch invite larger groups of users to take a test. Rather than entering their user details one at a time, as highlighted in the process above, you can invite them all at once, from the 'Users' section of your dashboard. Here is a separate blog entry about building your user list and batch inviting users to the same test: http://the-knowledgesmart-blog.blogspot.com/2011/03/building-your-list-of-users.html.

Option three - setting up a test session from a browser

Sometimes, you will need to set up a test session, without sending an email invite to the user. It is possible to set up a KS test session, straight from a web browser. Here is a separate blog entry on this topic: http://the-knowledgesmart-blog.blogspot.com/2011/03/setting-up-test-sessions-from-browser.html.

R

Logging in to the KS system

2011-04-07T14:32:00.002-07:00

OK, this might seem like a pretty basic task, but we thought it was worth documenting for new users of the KS system. How do admins access and log in to the KS system?

Here's how..

Step one
Go to the KS website homepage and click on the CUSTOMER LOGIN link, top right.

Step two
This takes you to the KS admin dashboard landing page. Here is the direct link: https://online.knowledgesmart.net.

Step three
To access your KS admin dashboard, click on the ADMIN DASHBOARD LOGIN link, just under the KnowledgeSmart logo.

Enter your administrator username and password, in the grey box and hit the Log In button.

And that's it! The first page you will see is the KS test list, which lists all of the 'off the shelf' tests on your account.

Here are a couple of additional things to be mindful of. First, when you are assigned your initial password by the KS system, it is generated by a software program. Consequently, it is a random series of characters and numbers. When you first log in to your dashboard, we suggest you change your system generated password, to one of your own choosing. Here's how..

Hit the Change password link, top right of your dashboard homepage. Copy & paste your system password in the first box. Enter your own password in box 2 and repeat in box 3. Then hit the Change password button - and you're done!

If you forget or lose your KS password, you have the option to re-set the password at any time. Go to the login page, hit the ADMIN DASHBOARD LOGIN link and click the Forgot details? link at the footer of the grey box.

This takes you to the password re-set page. Type in your username and hit Submit. You will be sent a new password to your registered email address.

Lastly, a word about the difference between the 'CADsmart' database and the 'KnowledgeSmart' system. You'll notice that there is a separate link called 'CADsmart Login', at the footer of the KS dashboard landing page. This link takes you to the login page for legacy CADsmart users. It is an entirely separate system and should not be confused with the login routine for the KnowledgeSmart tools. The CADsmart system has a separate account number and password combination. If you attempt to login to the CADsmart system using your KS username and password, you won't get very far. So don't try! :)

Conversely, you can't access a CADsmart dashboard with your KS credentials!

8 Steps to More Effective Training Needs Analysis

2011-04-01T08:47:00.000-07:00

OK, so you've captured a range of test scores in your firm. What next? The KnowledgeSmart dashboard offers you a number of options for reporting on your firm's results and identifying skills gaps across your teams.

Here are some steps you can take to interpret your test data..

Step one
Use the searching tools in the 'Results' page of your dashboard, to filter your raw test data. You have a range of search parameters available. For example, min score, max score, elapsed time, user name, test name & ID, date ranges, training meta tags, plus 5 extra datafields for categories of your own choosing.

You can use the 'List all training tags' tool, in the 'Your Tests' page of your dashboard, to export a master list of training workshop topics, for each test in your library. You can then search on these keywords, to identify users who require training on each category.

Step two
When you have run your search, create a new sub-group of your data, using the grouping tools. For example, all users who scored 60% or less on basic AutoCAD; all users who took longer than 90 mins to complete a basic MicroStation test; all users who require training on worksets in Revit Architecture; all users in your Kansas City office who scored 80%+ on Adobe Photoshop, and so on.

Step three
Use the charting tools to create management reports for each group and highlight priority training issues. The Training Requirements chart gives you an easy at-a-glance way of identifying the most urgent training topics, for a given group of users.

The Question Performance chart helps you to monitor question scores for each test in your library and further recognise any potential problem areas. Use the export to PDF tools to create hard copies of your charts.

Step four
Use the Group Comparison charts to analyze and compare results across a range of search results. For example, the same group of users, pre-and-post training (to measure performance improvement); multi-office results comparison; interview scores vs employee scores, and so on.

The Global Comparison chart offers you an external benchmark, for our most popular test titles.

Step five
You also have the option to export your results data to csv. When you export results, you will see on the right hand side of the sheet, a field called, 'Training Tags'. This is a personal summary of training topics for each user, representing keywords relating to questions where, for whatever reason, they didn't score 100% on the question.

Step six
It's a good idea to schedule a brief one-to-one sit down with your users, to discuss their test results and training recommendations summary. Disregard tags relating to questions where they made a silly mistake, or misread the instructions, etc. What you are left with is a list of training workshop topics that you - and they - both agree represent genuine skills gaps for a given software topic.

Step seven
Follow up with modular training workshops, which specifically target the highlighted skills gaps. If possible, try to group people together according to experience and ability. For example, avoid training novice users and expert users in the same session. You can use the KS meta tags to link to corresponding modular learning content from third parties (i.e. Evolve Consultancy, White Frog, 4D, Bentley Learn, ATC's, etc.) and your own in-house training collateral.

Step eight
Identify your most experienced users for the most significant software titles used by your firm. They will typically be the people appearing in Quartile 1 or Quartile 2 on your Performance Spread charts, or the people featuring in the top third of your Group Scores charts.

Challenge your leaders to help create bite-sized, modular training material, which you can then use to share knowledge and best practice across the firm. This is an effective way to turn 'tacit' knowledge (i.e. the information locked away in people's heads), into 'explicit' knowledge (i.e. documented, searchable and accessible to your organisation). Tag your learning material, so it can be searched and easily found via your firm's intranet, LMS, or similar.

Using the KS tools in this way, you can identify both problem areas and examples of excellence and best practice, across your firm. And target the main productivity issues with focused, relevant training and learning resources.

KS Test Library Update

2011-03-31T14:14:00.000-07:00

It's fair to say that the KS test library is a work in progress. It has also been mentioned (on more than one occasion!) that creating up to date test material for the plethora of design, engineering, modelling, civil, structural, building services and analysis tools used by AEC firms - is a job for life! :)

However, we like a challenge, so ever onwards do we march with our library roadmap!

Progress to date has been very good, with almost 30 test titles currently available to AEC firms, across a range of leading vendors.

Here's an up to date list, as of the end of March:

3ds Max – modules 1-4
Adobe InDesign for occasional users
Adobe Photoshop for occasional users
AEC Layers Standards – Autodesk
AutoCAD 2D fundamentals
AutoCAD 2D – Xpress
AutoCAD 2D for occasional users
AutoCAD Civil 3D (part 1)
AutoCAD Civil 3D (part 2)
Design Review fundamentals
Revit Architecture fundamentals
Revit Architecture – Xpress
Revit MEP fundamentals (Mechanical)
Revit MEP fundamentals (Plumbing)
Revit Structure fundamentals
Revit Structure – Xpress
AEC Levels Standards – Bentley
Bentley Architecture fundamentals
Bentley GEOPAK (Drainage)
Bentley InRoads (Roadway Design)
Bentley View fundamentals
MicroStation 2D fundamentals
MicroStation 2D – Xpress
MicroStation 2D for occasional users
MicroStation 3D fundamentals
Rhino fundamentals

We also have a pretty hectic 'work in progress' list, with a variety of new modules at various stages of completion.

Here's what's on the drawing board at the moment:

Revit Families
Revit Project Process
USACE ‘Attachment F’
Ecotect
Navisworks Manage
Revit Architecture 2012 (additional questions)
Revit MEP fundamentals (Electrical)
AutoCAD Civil 3D (part 3)
Revit MEP 2012 (additional questions)
Google SketchUp
AutoCAD Architecture
Revit Structure 2012 (additional questions)
Bentley ProjectWise
Bentley Navigator
MicroStation V8i (Select Series 1-3)
IES Virtual Environment
Tekla Structures
Bricscad
Codebook

So by the time we get through that little lot, we'll have close to 50 titles in the KS library. Well, that's our goal for the end of 2011, anyway!

In addition, we're seeing quite a bit of activity in customer dashboards, with some high quality custom test material being authored in many AEC firms.

So, that's a quick summary of where things currently stand with our test library. As always, we welcome your comments, feedback and requests for new material. If it isn't on the list, please let us know and we'll make sure it gets added to the workflow.

R

Customising your Test Invite Text

2011-03-30T09:21:00.000-07:00

KS admins have two main options for adding custom text to their test invites.

Option one
The default invite email text appears in the 'Invites' page of the KS dashboard.

From this page, a KS admin can overwrite the default text, in the 'Email subject line' field and in the 'Your message' field, on a per invite basis. You have a maximum of 800 characters for your test invite text.

Option two
Alternatively, KS admins can write their own custom invite text, which overwrites the KS default text. To do this, go to Settings > Test Invite.

Here, a KS admin can enter their custom invite text and email subject header. You have a maximum of 800 characters for your test invite text. Click 'Save' to register the changes. This custom text will now replace the KS default message on all test invites sent from the account.

The custom invite text appears on the main test invite page.

To revert to the default KS invite text, at any time, a KS admin simply has to delete the custom text from the Settings > Test Invite page of the dashboard.

R

Building your List of Users

2011-03-30T09:02:00.000-07:00

KS admins can upload a list of users into their dashboard and capture additional data against each user record, which can later be used for searching and grouping test results. Here's how the process works:

Step one
Log in to your admin dashboard and select 'Users' from your navigation tools.

Step two
Click on the 'Upload User Data' link and take a look in the 'Help' notes box on the right hand side. Tucked away in paragraph 2 is a link to a sample csv file, which you can use to build a list of your users.

Step three
Copy your user info from Excel, into your csv file. You need the following fields; Email, First name, Last name. You can also capture user status and 5 additional data fields for information of your own choosing. For example, employee ID, project team name, office location, job title, industry discipline, etc. Think of information you might like to use, when searching and filtering your results data later on.

Step four
Browse for your completed csv file and use the 'Upload' tool to add your user list to your KS dashboard.

Step five
Click on the magnifying glass icon, next to the user name (far right of the row), to add to or edit the 5 user datafields.

That's it - you've built your KS user list! If you subsequently upload a new csv, with additional user records, then the new list will merge with the existing user list.

Searching your user datafields
You can use the information captured in these 5 datafields, to filter your results data in the 'Results' area of your dashboard. Select the 'Show search' link in the orange bar, at the top of the Results page. You'll see the five datafields on the right of the search box.

Enter the information in the relevant datafield to filter your data, then select the 'Show groups' link in the orange bar, to create new sub-groups, based on the content of your user datafields.

Batch inviting
Once you have built your user list, you can invite groups of people at the same time, to take a test (provided they are all taking the same test!). Check the boxes next to the names of your invitees and hit the 'Invite selected users' button at the bottom of the page.

This takes you to the test invite page and pre-populates your user fields. It is important to remember to select the test you want from the library dropdown at the top. The default list displays A-Z, with 3ds Max at the top. (If you forget to change the test title, before sending your invites, you may have some confused users asking why you're asking then to take a 3ds Max test! Unless, of course, that's the one you wanted! :) ).

When you're happy with your invite text, just hit the 'Send Email' button to send your batch invites.

Test Feedback Options

2011-03-30T07:55:00.000-07:00

KS admins can set different levels of feedback for users at the end of a test session. It is also possible to select different levels of feedback for interview candidates and in-house users. Sometimes firms don't want interview candidates to receive a test report immediately after their test session. So now you have the choice of when to provide user feedback - and just how much information to share.

It's a straightforward process to choose your feedback settings:

Log in to your KS dashboard and click Settings > Test Feedback.

There are 4 choices:

1) No report
2) Basic report
3) Feedback on report and
4) Feedback & coaching on report.

The default setting is as follows:

Interview candidates - basic report

In-house users - full report with feedback and coaching notes

When you've made your selection, hit 'Save' to register your changes. This setting applies to all tests taken on your account. If you want to adjust the amount of feedback for different groups, just amend the setting in your dashboard, before you send out the test invites.

R

Branding your Test UI and Reports

2011-03-30T07:40:00.000-07:00

We recently added a light touch of corporate branding for KS customers.

It is now possible to replace the KS logo on all user-facing test UI's and reports, with your own company logo. We recommend that you do this, as it makes the whole experience for users feel more like an in-house training needs analysis exercise, rather than a scary third party skills test!

It's a really simple process..

Step one
Choose a copy of your company logo. Keep the file size sensible, as it is designed to fit in a comparatively small space, at the top of your admin dashboard, test UI and test reports. About this size should do just fine.

Step two
Log in to your admin dashboard and select Settings > Branding from your navigation tools.

Step three
Use the browse and upload tools to select your logo and upload a copy to the KS dashboard.

Step 4
Check the boxes to display your logo in 3 places; your KS dashboard, your test UI and your test reports.

Click the 'Preview' buttons, to view how your logo will display in the test UI and test report. You'll probably note that we've removed the KS orange colouring from the test reports, replacing it with a more neutral light grey - to avoid any unfortunate clashes with your own colours!

Sample Test UI

Sample Test Report

We're gradually dialling back on the KS branding, in favour of your own corporate colours. In future, you'll see a smaller, 'Powered by KnowledgeSmart' logo, appearing at the footer of most of our collateral.

Setting up Test Sessions from a Browser

2011-03-30T06:27:00.000-07:00

It's not always possible for users to access a test invite via email. For example interview candidates, or in-house candidates who are using a training room, where access to their email account is not available.

For this reason, we have introduced a new feature, which allows KS administrators to set up a new test session straight from a browser.

Here's how it works:

Step one
Admin selects the link called, 'Admin Test Setup Login', on the dashboard landing page.

Step two
Admin enters username and password to access their account.

Step three
Admin selects account from the dropdown (for firms with multiple accounts) and selects the required test title from their library dropdown. This list includes KnowledgeSmart OTS (off the shelf) tests and self-authored tests, displaying in A-Z order. Test ID numbers are also included, so admins can easily identify which test they want from their library.

Step four
This is where the candidate takes over. They enter their name and email address, select their status (employee or interview candidate) and proceed to the test landing page.

Step five
This is the test landing page. This is the same page as user's see when they click on the test URL in their email invites. From here, the process for taking a test is the same, irrespective of how they were invited. When they have read their test instructions and downloaded their sample data files, they hit the green 'Start' button to begin their test session.

Creating and editing KS tests

2011-03-24T16:21:00.000-07:00

AEC firms can use the KS system to create their own skills assessments from scratch.

Let's take a brief look at the test creation journey. We have built an intuitive, 4-step process, for writing and editing custom tests.

To edit an existing KS test, first hover your mouse over the 'Edit test' button, in the 'Your Tests' page of your admin dashboard.

Note that a small pop-up menu appears to the left of the 'Edit test' icon. This menu allows you to skip straight to the relevant step of the edit process, depending on the change you want to make.

Step one is where you enter the test name and a brief description of the key features being covered by the test.

Step two is where you assign modules from your library, to be linked together in a 'daisy-chain', creating a longer test. Use the green '+' icon, to add modules to your test. Conversely, you can remove modules by selecting the delete icon next to the module name, in the 'Your modules' panel on the right of your dashboard.

Step three is where you choose how many questions you want to be assigned in each module. The 'Available questions' column displays the total number of questions in the library 'pool'. The 'Assigned questions' column displays how many questions will be randomly presented to users during a live test. Click on the 'Edit test module' icon, to select the required number of questions in each stage, then click the green tick to save your changes.

Step four is where you can edit your test settings. Here you can set your preferences for presenting the sample test files to users, set a time limit on test sessions, provide additional instructions to your users, hide or show the timer in the test UI and assign meta tags, or keywords, to each test. Category tags are used to help you search and group your tests, back in your main library.

You can view a summary page for each test, by clicking on the magnifying glass icon, next to the test name, on the 'Your Tests' page.

KnowledgeSmart offers AEC firms a flexible solution for assessing the technical software skills of their teams. They can choose from a wide range of 'off the shelf' test titles, for design and engineering software. And they can write their own test material from scratch.

R

Creating and editing KS modules

2011-03-24T15:52:00.000-07:00

When building a KS test, we usually group questions by topic into modules. Modules can be deployed individually as a short test, or, in turn, they can be linked together (in what we refer to as a 'daisy-chain') to create a longer test.

Let's take a brief look at the module creation journey. We have built an intuitive, 3-step process, for writing and editing test modules.

To edit an existing KS module, first hover your mouse over the 'Edit module' button, in the 'Your Modules' page of your admin dashboard.

Note that a small pop-up menu appears to the left of the 'Edit module' icon. This menu allows you to skip straight to the relevant step of the edit process, depending on the change you want to make.

Step one is where you enter the module name and a brief description of the key features being covered by the module.

Step two is where you assign questions from your library, to be grouped together according to topic. Use the green '+' icon, to add questions to your module. Conversely, you can remove questions by selecting the delete icon next to the question name, in the 'Your questions' panel on the right of your dashboard.

Step three is where you assign your meta tags, or keywords, to each module. Category tags are used to help you search and group your modules, back in your main library.

When you have finished creating or editing your modules, you can view a complete summary of all the key elements for each module, by clicking on the module name, back in 'Your Modules'. This displays a green popup summary box - what we refer to as the module 'footprint'.

The KS system is very flexible. Administrators can move questions around between modules and move modules around between tests. You can use the KS content, 'off the shelf' (completely unchanged), you can edit the KS content and you can create test material from scratch. Then use a blend of all this material, to build your own skills assessments for your firm.

Next, we'll look at the process for creating and editing KS tests.

R

Creating and editing KS questions

2011-03-24T15:15:00.000-07:00

The KnowledgeSmart online skills assessment system offers a high degree of flexibility to administrators, in terms of the way they can customize their test library.

We work closely with specialist authors around the world, to create quality assessments for a range of design and engineering software titles. These assessments can reliably be used 'out of the box', or, as we prefer to say, 'off the shelf'. (We don't actually have a box!).

But many firms prefer to edit the OTS content, to better match their own particular standards, workflows or project environment. They also have tools to create their own test library material from scratch.

Let's take a brief look at the question creation journey. We have built an intuitive, 5-step process, for writing and editing test questions.

To edit an existing KS question, first hover your mouse over the 'Edit question' button, in the 'Your Questions' page of your admin dashboard.

Note that a small pop-up menu appears to the left of the 'Edit question' icon. This menu allows you to skip straight to the relevant step of the edit process, depending on the change you want to make.

Step one is where you enter the question name, a brief summary of the particular features being covered by the question and the question wording.

Step two provides you with the option of adding a file (or multiple files) to your task-based questions. There are a number of file types which can be assigned to questions, for example, .dwg, .dgn, .rvt, PDF's, image files, movie files, and so on. If you move questions around your library, from one module to another, the files move with the question.

Step three is where you input the answer to a question. The fields vary, depending on the type of question, for example, free text, multiple choice, pick list, order list, or true/false question styles.

Step four is where you assign your meta tags, or keywords, to each question. Category tags are used to help you search and group your questions, back in your library. Training tags are used to help determine skills gaps amongst your users. It is important to think carefully about which tags are assigned to your questions, as these will play an important part in your results analysis later on.

Step five is where you can view and edit the coaching notes for each question. Coaching notes provide detailed feedback for users, on how to answer each question accurately and efficiently. Coaching notes can be provided in a number of formats, for example; text only, text with accompanying screen shots, PDF’s with words & images, short movies, etc.

When you have finished creating or editing your questions, you can view a complete summary of all the key elements for each question, by clicking on the question name, back in 'Your Questions'. This displays a green popup summary box - what we refer to as the question 'footprint'.

Here are some practical hints & tips for KS authors:

- A full question set is defined as follows; usually 18-25 questions, format to be determined by author, typically covering a range of core skills in a specified software application.

- A question set comprises live skills (task based) and theory (knowledge based) content. (Author to determine appropriate mix of questions).

- A task based question requires the candidate taking the test to have access to and use the software in order to elicit the answer.

- A knowledge based question can be answered without direct use of the software.

- You have 5 types of questions to choose from; Free text, Multiple choice, Order list, Pick list and True or False. Try to get a balance of different styles in your question set.

- When writing multiple-choice or pick-list questions, provide at least 5 answer options. Try to avoid general answers, such as ‘All of the above’, or ‘None of the above’.

- Don’t write too many ‘True or False’ type questions – it’s too easy to guess the answers!

- Use screen shots and image files, or short videos, to enhance the effectiveness of some questions (i.e. PNG, JPEG, Camtasia, etc.).

- Think about the meta tags to be assigned to your qestions. You will be making practical use of these tags when you analyze your test results and create groups and charts with your results data.

- Coaching notes should accompany each question. They are the author’s way of telling a user how they should answer a particular question accurately and efficiently.

Next, we'll look at the process for creating and editing your test modules.

R

AEC IT Leaders Roundtable

2011-03-21T15:56:00.000-07:00

KnowledgeSmart will be showcasing the latest release of our skills assessment tools for CAD, BIM, Engineering & Analysis software tools, at this year's AEC IT Leaders Roundtable. The meeting takes place in San Jose, California, on 14 & 15 April.

KS CEO, Rory Vance, will be talking about the benefits of capturing performance metrics for a range of design and engineering software titles - and the shortcomings of firms who rely too heavily on self-assessment, as a means of identifying training needs for their teams.

For more information, go to: http://sites.google.com/site/aecitleadersorg.

New charting options

2011-03-10T14:14:00.000-08:00

This month sees the launch of the latest release of the KS tools, with a host of new features to be found in the admin dashboard.

One of the key updates relates to the range of charting and management reporting options now available.

Here's a brief summary..

Search & Group Results
The first task for an admin, is to carve up the raw results data into more manageable blocks. There are a range of searching and grouping options available, including; score ranges, date ranges, elapsed time, test topic, user names, training keywords, plus 5 additional custom datafields.

Performance Scatter
This chart displays group results in performance quartiles. The upper left quadrant (Q1) contains results, where the users completed their assessment accurately and fast. The upper right quadrant (Q2) shows results which have high accuracy, but slower completion times. Bottom left (Q3) represents lower scores, but in a fast time. Lastly, the bottom right quartile (Q4) shows test scores which are inaccurate and slow. Hint: you don't want your core users anywhere near quartile 4!

Training Requirements

This chart highlights training topics for a given group of test results. The logic analyzes all of the results for a group, references the training tags assigned to questions presented during a test and lists those tags in priority order.
Red indicates the tasks which have been answered incorrectly by most people in a given group. Orange is the next highest priority, followed by Yellow. Green training tags are the topics which have been answered correctly by most of the group, so represent the least urgent issues.
For example, 10 people are presented with one or more questions, which include the tag, ‘Lines’. If 7 of the 10 people answer one or more ‘Lines’ questions incorrectly, then the keyword, ‘Lines’, will be flagged at 70% and appear in Red on the chart.

Question Performance

This chart looks at how each individual question in a library has performed, in any given test. The logic analyzes all of the results for each test and presents an aggregate percentage score for each question, divided by the total number of results for that test on the account.

For example, 10 people answer a question called ‘Lines’. If 7 of the 10 people answer the question 100% correctly, 1 person scores 50% and the other 2 score 0%, then the question will score an aggregate of 75% and appear in Yellow on the chart.

Group Scores
This chart displays user performance for any given group, in descending order. The X-axis shows the % score attained and the Y-axis displays user names.

Group Comparisons
This chart allows firms to compare performance, from one group to another, across (up to) 9 sets of data at a time. Group vs group comparisons can be used to compare a range of results data. For example; pre and post-training performance, different project teams, firm offices from different geographic locations, data representing different job titles or industry disciplines, in-house data vs interview candidates, and so on.

Global Comparisons

This chart allows AEC firms to compare in-house performance against a wider set of anonymous industry aggregate results data, for selected tests. Currently, we are publishing results for; AutoCAD 2D, MicroStation 2D and Revit Architecture. When some of our other test titles accumulate a sufficient body of results data, we will publish additional benchmark figures.

This release is scheduled for go-live mid-March. The new range of charting and reporting options should make identifying skills gaps and training needs much easier for CAD & BIM administrators and training and learning development professionals.

We'll be revisiting charting options again later in the year, so if you have any additional requests for chart styles and reports, please let us know and we'll add them to the Summer/Fall roadmap.

Question Types

2011-03-02T05:31:00.002-08:00

When creating skills assessments, it is important to include a range of question styles, so the user experience is varied and engaging.

Here is a list of different question types, which commonly feature in most skills assessments:

Multiple-choice questions usually have up to eight answer choices with only one correct answer.

Pick-list or Check box questions usually have up to ten answer choices. Two or more of these answer choices can be correct.

True or False questions have two answers, True and False, one of which is correct.

Fill in the blanks questions have statements with blanks in them. The test taker must type text in each blank as the answer.

Short answer or Free text questions require the test taker to type in an answer to the question.

Order list answers must be selected in a specific order or sequence.

Point and click or Hot spot questions contain a picture or some other form of media. The test taker must select a point on the picture or image.

Numeric question types are similar to Short answer questions but require numeric instead of text answers.

Likert questions allow you to design a question based on a scale of answers, i.e. 1-5 or 1-10. (NB this type of question is better suited to surveys than straight assessments).

Matching questions have a content area and a list of names or statements (List A) which must be correctly matched against another list of names or statements (List B). List B usually includes a number of 'distractor' answers.

Essays are useful for fact-based or critical thinking test items. Answers can be a word, phrase, sentence, paragraph or lengthy essay. Essay questions are scored manually.

For capturing skills gaps across a range of design and engineering software applications, KnowledgeSmart typically uses the following types:

Free text
Multiple-choice
Pick-list
Order list
True or False

Following on from a series of customer review meetings, we plan to include some new question types in the KS system this year:

Fill in the blank
Hot spot
Matching

We'll also look into whether firms would like added flexibility on the free text questions. Up until now, we have gone with the approach that an answer is either right or wrong. However, it is possible to tweak the marking routines, so that a test taker can receive credit for entering one of multiple variations/spellings/etc. of the answer.

One of the main advantages of our system is the automated marking routine. For this reason, we avoid using essay-style questions in our assessments. We combine Free text and Numeric question types in the KS system.

R

Bloom's Taxonomy

2011-03-02T04:15:00.000-08:00

In creating practical skills assessments for AEC firms, we try to offer a blend of question styles and types. Some questions are task based (i.e. the user needs to physically perform a task, usually relating to a particular technical software application, in order to figure out the right answer). Some questions are knowledge, or theory, based (i.e. the user should be able to answer the question without referring to any other resources, or opening up a software program).

Before we examine question types, however, it might be helpful to take a look at how people learn - an accepted definition of the cognitive process.

"Taxonomy" is a system of categorizing and organizing. 'Bloom's Taxonomy' is a system created to improve testing precision by categorizing cognitive functioning into different levels. Appropriate questions could then be developed to assess the desired level. Psychologist Benjamin Bloom developed this system at the University of Chicago in the late 1940s. His goal was to improve the quality of discussion of educational goals among teachers, administrators, and researchers.

Bloom's levels are, in ascending order:

Knowledge
Comprehension
Application
Analysis
Synthesis
Evaluation

In this case, the taxonomy is hierarchical; each level is subsumed by the higher levels. In other words, a student functioning at the "application" level has also mastered the material at the "knowledge" and "comprehension" levels.

Let's take a brief look at each stage - and what it means in terms of a learner's level of understanding..

Knowledge
Recall content in the exact form that it was presented. Memorization of definitions, formulas, or procedures are examples of knowledge-level functioning.
Types of action: List, define, label, identify, name.

Comprehension
Restate material in their own words, or can recognize previously unseen examples of a concept.
Types of action: Describe, associate, categorize, summarize.

Application
Apply rules to a problem, without being given the rule or formula for solving the problem.
Types of action: Apply, calculate, illustrate, solve.

Analysis
Break complex concepts or situations down into their component parts, and analyze how the parts are related to one another.
Types of action: Analyze, compare, separate, order, explain.

Synthesis
Rearrange component parts to form a new whole.
Types of action: Combine, modify, rearrange, "what-if".

Evaluation
Evaluate or make judgments on the worth of a concept, object, etc. for a purpose.
Types of action: Assess, decide, grade, recommend, explain, judge.

In my next blog, we'll take a look at the different types of question, which typically make up a skills assessment.

R

KnowledgeSmart user group meeting

2011-02-10T08:48:00.000-08:00

We're organising our next user group meeting in London next month.

Here are the details:

Date: Tuesday 15 March

Time: 9am – 1pm (working lunch included)

Venue: Stirling Room, RIBA, 66 Portland Place, London

Cost: Free of charge

Places are filling up rapidly - and we always end up with a waiting list for these sessions - so if you'd like to come along, you'll need to let us know in the next few days.

We have room for about 30 people in total and, as of this afternoon, we have 10 places remaining.

The agenda on the day will be pretty flexible, with plenty of scope for open discussion. From our end, the main talking points will be:

KS system tools update – what’s new?

KS wish list – tell us what else you’d like to see in the next 3-6 months..

Training links – marrying the KS training outputs with appropriate learning material

The impact of BIM on technical software in AEC firms

A closer look at testing Revit and Civil 3D skills

Writing your own assessment material – do’s and don’t’s

Plus whatever else comes up in open debate!

Hope to see you there!

Assessing Revit skills

2011-02-03T09:08:00.000-08:00

We have a rolling program of new test content creation; our aim is to double the size of the KS library this year, depending on how things go, economically speaking. So far, so good in 2011.

It will come as no great surprise to learn that Revit tops the list of customer requests, so we have a few ideas for new content coming up in the next 3-6 months.

That's not to say the existing Revit test library material isn't looking good already! We have some excellent RAC questions, authored by leading Revit protagonists Paul Aubin, Paul Woddy and Chris Senior. We have a comprehensive RST library, written in partnership with Thomas Weir. And our RMEP modules are coming along nicely, too, with the help of US based author, Tony Tedder.

It's just that people want more! We're happy to comply, so here's what we have in mind..

Revit Families - a module which looks specifically at creation of Revit families. We'll probably split this into 2 sections; one for general families and one for discipline specific content.

Revit process - a module covering the practical aspects of using Revit on a project and the impact of working in a team environment.

RAC 2012 - more basic test content for Revit Architecture 2012, linking to corresponding training courseware from authors White Frog.

RMEP 2012 - more basic test content for Revit MEP 2012, linking to corresponding training courseware from authors White Frog.

RST 2012 - more basic test content for Revit Structure 2012, linking to corresponding training courseware from authors White Frog.

Timescales are not set in stone, but work is underway on the families and process modules. We'll start the new 2012 modules in March, once the code for the new releases from Autodesk is more or less frozen! We're aiming to have much of the new library material in place by the Summer.

Last, we have some intermediate level material planned for all 3 disciplines, during the Summer/Fall period.

Assessing AutoCAD Civil 3D skills

2011-02-03T01:54:00.000-08:00

Another new addition to the KS test library went live this week - AutoCAD Civil 3D 2011, module 1.

Now I'm not a believer that you can reliably create a 'general' Civil 3D test, so we're taking a slightly different approach with this one.

Because the KS tools offer AEC firms a tremendous amount of flexibility, in the way they can customise test content, we are giving firms the final say on what modules their version of the C3D test will contain. The daisy-chain feature, which allows admins to link a series of individual modules together to form a longer test, means that firms can mix & match the C3D content in a wide variety of ways.

So how are we approaching this task? Well, first, we're working with leading C3D author, Eric Chappell, to create 4 overall test modules, covering a range of C3D tools and commands. Each module will contain c.25 test questions, with the usual mix of task-based and knowledge-based exercises.

These are the main topics, which will be covered across the 4 modules:

Surfaces
Alignments
Profiles
Points
Sections & quantity take offs
Assemblies & Subassemblies
Corridors & intersections
Parcels
Grading
Pipes & pipe networks
Plan production
Data exchange
Rendering
Survey
Hydraflow

Module 1 is now available, covering Profiles, Alignments and Surfaces. Work has already begun on module 2, which will be finished in March. Module 3 will follow in April/May. The last module will be written in the summer; we'll review progress to date at that point and take feedback on any remaining features which haven't been addressed in modules 1-3.

Enterprise customers will see the new C3D content appear in their accounts this week. As always, we welcome your feedback on the material and also any topics you'd like to see covered, which are not mentioned in the summary list above.

SECI Model for Organizational Learning

2010-12-14T13:37:00.000-08:00

How does an organization learn?

In 1995, Professors Nonaka & Takeuchi (at Hitotsubashi University, Tokyo) developed a four stage spiral model of organizational learning.

SECI:

Socialization
Externalization
Combination
Internalization

Tacit knowledge is personal, context specific, subjective knowledge, whereas explicit knowledge is codified, systematic, formal, and easy to communicate. The tacit knowledge of key personnel within an organization can be made explicit, codified in manuals, and incorporated into new systems and processes. This process is called "externalization".

The reverse process (from explicit to implicit) is called "internalization" because it involves employees internalizing an organization's formal rules, procedures, standards and other forms of explicit knowledge.

“Socialization" denotes the sharing of tacit knowledge and the term "combination" denotes the dissemination of codified knowledge.

According to Nonaka & Takeuchi, knowledge creation and organizational learning take a path of socialization, externalization, combination, internalization, socialization, externalization, combination . . . and so on, in an infinite spiral.

So what exactly is an 'expert'?

2010-12-14T13:17:00.000-08:00

The term 'expert' is defined as 'somebody with a great deal of knowledge about, or skill, training, or experience in, a particular field or activity'.

In CAD & BIM circles, the label AutoCAD 'expert', Revit 'expert', MicroStation 'guru', and so on, are used pretty liberally. In some cases, the monicker is perfectly justified. In others, however...

It's often interesting to capture a user's perception of how good they think they are at a particular software app, prior to them taking a skills assessment. We like to ask the following question:

Please rate yourself on your ability to use the software, for which you are about to take a test, on a scale of 1 to 5:

1 = Very basic knowledge; not enough to work confidently on a project

2 = Basic knowledge; can get by working on a project but could do better

3 = Good knowledge; can produce a good standard of work on projects

4 = Advanced knowledge; can teach the basics to others

5 = Expert knowledge; can perform and teach others at an advanced level

Interestingly, we are mirroring a recognised format for gauging skills measurement. In the fields of education and operations research, the Dreyfus model of skill acquisition is a model of how students acquire skills through formal instruction. Brothers Stuart and Hubert Dreyfus proposed the model in 1980 in an influential report on their research at the University of California, Berkeley, Operations Research Center for the United States Air Force Office of Scientific Research.

The model proposes that a student passes through five distinct stages; novice, advanced beginner, competent, proficient and expert.

1. Novice

• "rigid adherence to taught rules or plans"

• no exercise of "discretionary judgment"

2. Advanced beginner

• limited "situational perception"

• all aspects of work treated separately with equal importance

3. Competent

• "coping with crowdedness" (multiple activities, accumulation of information)

• some perception of actions in relation to goals

• deliberate planning

• formulates routines

4. Proficient

• holistic view of situation

• prioritizes importance of aspects

• "perceives deviations from the normal pattern"

• employs maxims for guidance, with meanings that adapt to the situation at hand

5. Expert

• transcends reliance on rules, guidelines, and maxims

• "intuitive grasp of situations based on deep, tacit understanding"

• has "vision of what is possible"

• uses "analytical approaches" in new situations or in case of problems

In the worlds of music and dance, becoming an expert requires an investment of approximately 10,000 hours in practice and execution!

Consider how this breaks down:

Hours per week Hours per year Years to achieve 'expert'

4 200 50

8 400 25

12 600 16.7

16 800 12.5

20 1,000 10

24 1,200 8.3

So the next time someone professes to be an expert, ask yourself, is that what they really mean?

KnowledgeSmart wish list

2010-12-14T13:02:00.001-08:00

We've been holding a series of web review meetings with customers in recent weeks, discussing how their assessment programs are coming along, and what tools they would like to see next in the KS admin dashboard.

Here are the most requested items, the majority of which will feature in the next point release (scheduled for Feb 2011);

More charting; would like an easier way to tell (at a glance) what the main problem areas/training topics are
Would like to see industry comparisons for most popular test scores
Want to display more results per page
Would like ability to invite interview candidates more easily
Would like to control level of test report feedback for interview candidates
Ability to delete test scores
View results for linked accounts
Notify admin if test expiry date passes (but test not taken)
Change status of users (i.e. ex-employee, interview candidate to employee, etc.)

We'll also be adding basic branding options for firms, new chart styles in the dashboard and more content management options for test modules.

In addition, personal dashboards for users, basic survey tools, a community area for sharing test content and a new coaching mode are all coming up in the next few months.

Our rolling program of new test content authoring is ongoing. This week sees the release of our first Adobe InDesign module. More RMEP content will be out shortly, plus the first in a series of Civil 3D modules goes live in January. Intermediate RAC content, a Revit families module and basic Photoshop content will follow in early 2011.

If you have additional suggestions for new features or tools, we'd welcome your feedback.

Unskilled and unaware; why self-assessment is fundamentally flawed (part two)

2010-12-14T08:23:00.000-08:00

This is part two of my case against the use of self-assessment, as a reliable means of measuring staff performance.

The Latin maxim “ignoramus et ignorabimus”, meaning "we do not know and will not know", stood for a position on the limits of scientific knowledge, in the thought of the nineteenth century.

In September 1930, mathematician David Hilbert pronounced his disagreement in a celebrated address to the Society of German Scientists and Physicians, in Königsberg;

“We must not believe those, who today, with philosophical bearing and deliberative tone, prophesy the fall of culture and accept the ignorabimus. For us there is no ignorabimus, and in my opinion none whatever in natural science. In opposition to the foolish ignorabimus our slogan shall be: We must know — we will know!”

In the late twentieth century, Ex-United States Secretary of Defense, Donald Rumsfeld, whilst defending his country's position on the Iraq war, made the following (now infamous) statement; “There are ‘known knowns’. These are things we know that we know. There are ‘known unknowns’. That is to say, there are things that we now know we don’t know. But there are also ‘unknown unknowns’. These are things we do not know we don’t know”.

There are four recognised stages of competence.

1) Unconscious Incompetence

The individual neither understands nor knows how to do something, nor recognizes the deficit, nor has a desire to address it. The person must become conscious of their incompetence before development of the new skill or learning can begin.

2) Conscious Incompetence

Though the individual does not understand or know how to do something, he or she does recognize the deficit, without yet addressing it.

3) Conscious Competence

The individual understands or knows how to do something. Demonstrating the skill or knowledge requires a great deal of consciousness or concentration.

4) Unconscious Competence

The individual has had so much practice with a skill that it becomes “second nature” and can be performed easily. He or she may or may not be able to teach it to others, depending upon how and when it was learned.

"The Invisible Gorilla" experiment is one of the most famous psychological demo's in modern history. Subjects are shown a video, about a minute long, of two teams, one in white shirts, the other in black shirts, moving around and passing basketballs to one another. They are asked to count the number of aerial and bounce passes made by the team wearing white, a seemingly simple task. Halfway through the video, a woman wearing a full-body gorilla suit walks slowly to the middle of the screen, pounds her chest, and then walks out of the frame. If you are just watching the video, it’s the most obvious thing in the world. But when asked to count the passes, about half the people miss it. It is as though the gorilla is completely invisible.

(http://www.theinvisiblegorilla.com/gorilla_experiment.html).

In his popular KM blog, Nick Milton (http://www.knoco.co.uk/Nick-Milton.htm) writes in detail about the impact of this experiment and picks up on a number of key trends discussed in the book of the same name, authored by Christopher Chabris and Daniel Simons (the guys behind the original experiment).

The subtitle of the book is "ways our intuition deceives us", and the authors talk about a number of human traits (they call them illusions) which we need to be aware of in Knowledge Management, as each of them can affect the reliability and effectiveness of Knowledge Transfer.

To paraphrase Milton, the illusions which have most impact on Knowledge Management are;

• The illusion of memory

• The illusion of confidence

• The illusion of knowledge

Our memory of events fades over time, to the point that even firm documentary evidence to the contrary doesn't change what we remember. The implication is that if you will need to re-use tacit knowledge in the future, then you can't rely on people to remember it. Even after a month, the memory will be unreliable. Details will have been added, details will have been forgotten, the facts will have been rewritten to be closer to "what feels right".

Tacit knowledge is fine for sharing knowledge on what's happening now, but for sharing knowledge with people in the future then it needs to be written down quickly while memory is still reliable.

Without a written or photographic record, the tacit memory fades quickly, often retaining enough knowledge to be dangerous, but not enough to be successful. And as the authors say, the illusion of memory can be so strong that the written on photographic record can come as a shock, and can feel wrong, even if it’s right.

Any approach that relies solely on tacit knowledge held in the human memory can therefore be very risky, thanks to the illusion of memory.

The illusion of confidence represents the way that people value knowledge from a confident person. This would be fine if confidence and knowledge go hand in hand, but in fact there is almost an inverse relationship. A lack of knowledge is, instead, allied to overconfidence. Lack of knowledge leads to confidence, which leads to you being seen as knowledgeable.

Each chess player is given a points rating based on their competition results, which is in fact a very effective and reliable measure of their ability. Yet 75% of chess players believe they are underrated, despite the evidence to the contrary. They are overconfident in their own ability.

In studies of groups of people coming together to solve a maths problem, you would expect the group to defer to the person with the greatest maths knowledge, wouldn't you? In fact, the group deferred to the most confident person, regardless of their knowledge. In trials, in 94% of the cases, the final answer given by the group is the first answer suggested, by the most confident person present, regardless if whether it is right or wrong.

In a Harvard study of confidence vs knowledge in a trivia test, they certainly saw overconfidence in individuals - people were confident of their answer 70% of the time, while being correct only 54% of the time! When people were put together in pairs, the counterintuitive outcome was that the pairs were no more successful than the individuals, but they were a lot more confident! When two low-confidence people were put together, their overall confidence increased by 11%, even though their success rate was no higher than before.

The Illusion of Knowledge is behind the way we overestimate how much we know. The authors refer to how people think they know how long a project will take, and how much it will cost, despite the fact that projects almost always overrun in both cost and time. "We all experience this sort of illusory knowledge, even for the simplest projects" they write. "We underestimate how long they will take or how much they will cost, because what seems simple and straightforward in our mind typically turns out to be more complex when our plans encounter reality. The problem is that we never take this limitation into account. Over and over, the illusion of knowledge convinces us that we have a deep understanding of what a project will entail, when all we really have is a rough and optimistic guess based on shallow familiarity"

"To avoid this illusion of knowledge, start by admitting that your personal views of how expensive and time-consuming your own seemingly unique project will be are probably wrong. If instead, you seek out similar projects that others have completed, you can use the actual time and cost of these projects to understand how long yours will take. Taking such an outside view of what we normally keep in our own minds dramatically changes how we see our plans"

If we are unaware of these 3 illusions, we can feel confident in our knowledge, based on our memories of the past, without realising that the confidence is false, the knowledge is poor, and the memories are unreliable and partially fictitious. Awareness of these illusions allows us also to challenge the individual who confidently declares "I know how to do this. I remember how we did it 5 years ago", because we recognise the shaky nature of confidence, knowledge and memory.

A natural human tendency is that we tend to think that we know more than we do and that

we are better than we are. We suffer from what psychologists call the “Lake Wobegon effect”.

Based on Garrison Keillor’s fictional town where “all the women are strong, all the men are good-looking and all the children are above average.” According to the author’s own survey, 63% of Americans consider themselves more intelligent than the average American.

In contrast, 70% of Canadians said they considered themselves smarter than the average Canadian. In a survey of engineers 42% thought their work ranked in the top 5% among their peers. A survey of college professors revealed that 94% thought they do ‘‘above average’’ work – a figure that defies mathematical plausibility! A survey of sales people found that the average self-assessment score (for sales demos) was 76%. The average % of demos that achieved

objectives (for the same group) was 57%. The list goes on..

So, in summary, any strategy for capturing user skills data, which relies solely on an individual's ability to self-rate themselves on a given subject, is simply doomed to fail. I leave the last word to David Dunning; “In essence, our incompetence masks our ability to recognize our incompetence”.

Unskilled and unaware; why self-assessment is fundamentally flawed (part one)

2010-12-14T07:40:00.000-08:00

In October, I travelled to Boston, to present a paper to the AIA HR Large Firm Roundtable. I decided to make the case against the use of self-assessment, as a reliable means of capturing management metrics for staff performance.

Over the years, many AEC firms have confidently stated to me, 'We don't need independent skills testing, we already know how good our users are'. When one enquires further, what they actually mean, is that they sent out a user survey, asking staff to rate themselves (usually out of 5) on a range of different skills topics, including AutoCAD, Revit, InDesign, et al. What they end up with is a spreadsheet (why is it always a spreadsheet??) with a bunch of names down one side, a list of skills categories across the top - and a sheet filled with 3's and 4's. Why 3's and 4's, I hear you ask? Simply because people don't want to risk the personal penalties that might go along with admitting they're a 1 or a 2. And conversely, they don't want to stick their head above the parapet by admitting to a 5 (even if they are a 5) because this can cause all sorts of new issues (more work, more people pestering them for answers to the same questions, you get the picture). So it's 3's and 4's all the way. Congratulations XYZ Architects, you have your completed spreadsheet, so you are now totally self-aware, as an organization. (Not). And the real rub here is that, more often than not, people have no clue how good they are, relative to the rest of the team, or wider industry!

So I decided to put this argument to bed, once and for all. Here's my evidence..

Let me begin with a story. Earlier this year, NY Times Online posted a series of articles by filmmaker Errrol Morris. He tells the tale of Bank robbery suspect McArthur Wheeler, who was recognized by informants who tipped detectives to his whereabouts after his picture was telecast one Wednesday night, during the Pittsburgh Crime Stoppers segment of the 11 o’clock news. At 12:10 am, less than an hour after the broadcast, he was arrested. Wheeler had walked into two Pittsburgh banks and attempted to rob them in broad daylight. What made the case peculiar is that he made no visible attempt at disguise. The surveillance tapes were key to his arrest. There he is with a gun, standing in front of a teller demanding money. Yet, when arrested, Wheeler was completely disbelieving. “But I wore the juice,” he said. Apparently, he was under the deeply misguided impression that rubbing one’s face with lemon juice rendered it invisible to video cameras.

Pittsburgh police detectives who had been involved in Wheeler’s arrest explained that Wheeler had not gone into “this thing” blindly but had performed a variety of tests prior to the robbery. Although Wheeler reported the lemon juice was burning his face and his eyes, and he was having trouble (seeing) and had to squint, he had tested the theory, and it seemed to work. He had snapped a Polaroid picture of himself and wasn’t anywhere to be found in the image. There are three possibilities:

(a) the film was bad;
(b) Wheeler hadn’t adjusted the camera correctly; or
(c) Wheeler had pointed the camera away from his face at the critical moment when he snapped the photo

Pittsburgh Police concluded that, 'If Wheeler was too stupid to be a bank robber, perhaps he was also too stupid to know that he was too stupid to be a bank robber — that is, his stupidity protected him from an awareness of his own stupidity.'

Now, this sorry tale might have been just another footnote in history, were it not for the fact that it came to the attention of David Dunning, a Cornell professor of social psychology. After reading this story in 1996, Dunning wondered whether it was possible to measure one’s self-assessed level of competence against something a little more objective – say, actual competence.

Over the next 3 years, Dunning (assisted by colleague Justin Kruger) undertook a major academic study and, in 1999, published the paper, “Unskilled and Unaware of It: How Difficulties of Recognizing One’s Own Incompetence Lead to Inflated Self-assessments”.

Dunning’s epiphany was; “When people are incompetent in the strategies they adopt to achieve success and satisfaction, they suffer a dual burden; not only do they reach erroneous conclusions and make unfortunate choices, but their incompetence robs them of the ability to realize it. Instead, like Mr. Wheeler, they are left with the erroneous impression they are doing just fine. In essence, our incompetence masks our ability to recognize our incompetence”.

Dunning & Kruger also quote the “above-average effect”, or the tendency of the average person to believe he or she is above average, a result that defies the logic of statistics. Participants scoring in the bottom quartile on tests grossly overestimated their performance and ability. Although test scores put them in the 12th percentile they estimated themselves to be in the 62nd.

Conversely, Because top performers find the tests they confront to be easy, they mistakenly assume that their peers find the tests to be equally easy. As such, their own performances seem unexceptional. In studies, the top 25% tended to think that their skills lay in the 70th–75th percentile, although their performances fell roughly in the 87th percentile.

Dunning and Kruger proposed that, for a given skill, incompetent people will:

 tend to overestimate their own level of skill;
 fail to recognize genuine skill in others;
 fail to recognize the extremity of their inadequacy;
 recognize and acknowledge their own previous lack of skill, if they can be trained to substantially improve.

As a follow up study, “Why the unskilled are unaware: Further explorations of (absent) self-insight among the incompetent”, was published in 2006. (David Dunning, Justin Kruger, Joyce Ehrlingera, Kerri Johnson, Matthew Banner).

In part two, I'll take a look at the 4 stages of competence - and how the combined illusions of memory, confidence and knowledge can impact on a firms' knowledge management strategy.

UK BIM Roundtable - meeting notes

2010-12-10T09:09:00.000-08:00

Thanks again to those of you who attended and contributed to the UK BIM Roundtable meeting, last month.

The meeting notes have been posted online, here: http://www.bimroundtable.com/.

Please feel free to provide any comments or feedback, as appropriate.

We anticipate running a follow up session in the Spring, focusing on one or two of the key discussion points from meeting one. More details to follow in the NY..

Testing Adobe InDesign skills

2010-12-10T08:57:00.000-08:00

Another day, another milestone! We’ve had quite a few requests this year, mostly from our Architectural customers, to create some assessments for Adobe’s Creative Suite of tools.

To that end, the first module – Adobe InDesign for occasional users (CS5) – goes live this month.

In the same style as our assessment content for Autodesk and Bentley Systems' technical software apps, our system provides Adobe users with a live online skills test experience, presenting a mix of theory and task-based questions. (Theory questions can be answered without opening InDesign; task-based questions require a copy of the software). A test takes on average 30-60 mins, depending on the experience of the user. At the end, candidates can view a detailed test report, complete with feedback, coaching notes and suggestions for further training workshops.

Having spent the past 7 years or so, involved solely in the Autodesk and Bentley space, it's quite a treat to be adding a third major vendor to our portfolio of assessment offerings!

Over the coming months, we have the following additional assessments planned:

Adobe Photoshop for occasional users (CS5) – early 2011
Adobe InDesign fundamentals (CS5) – Spring 2011
Adobe Illustrator for occasional users (CS5) – Spring 2011
Adobe Photoshop fundamentals (CS5) – Spring/Summer 2011
Adobe Acrobat 9 fundamentals – Summer 2011

If you'd like to trial the new InDesign test, drop us a line or use the contact form on the main KS website. Enterprise customers will see the new modules appear in their dashboards, as soon as we go live.

Testing software skills of new hires (in the US)

2010-11-25T05:23:00.000-08:00

I've blogged on this subject in the past, but it just keeps coming up in conversation with HR professionals, Stateside; is it OK to test the basic CAD or BIM skills of prospective employees in the USA? A common conundrum surrounding skills assessments is whether you are legally permitted to screen the technical software ability of prospective employees and contract staff, before they join your team.

The short answer is yes, but only under the right circumstances. Let’s examine some of the evidence.

The most important issue underlying the use of pre-employment assessments is validity. The question we need to ask is this; ‘Is the test valid for this intended purpose; does it support the decisions that are going to be made?’

So what is 'validity'? Validity measures how appropriate a test is for a specific purpose. Simply put, a test may be considered valid for one use and invalid for another. Why do pre-employment tests need to be validated? In 1978, the Equal Employment Opportunities Commission (EEOC) created guidelines to ensure that the knowledge gained from testing is applied with impartiality to protect minority applicants from discriminatory employment procedures. What is the best method of validation? The EEOC guidelines do not state that one method is better than another; the method used must fit the needs of the business or organization.

There are three methods of validation set forth by the EEOC:

1) Criterion Validity: If data demonstrates that a test is significantly correlated with a vital measure of job performance, the test is said to demonstrate criterion validity. For example, if all the users that scored highly on a selected test to measure CAD skills completed their projects accurately and on time, the test would demonstrate criterion validity.

2) Construct Validity: The term construct is a technical term for personality traits like intelligence and creativity. Construct validity is demonstrated if a test measures traits that have been found to influence successful performance of a job. A test that measures the interpersonal communication skills of a potential customer service representative would demonstrate construct validity.

3) Content Validity: This is demonstrated if the questions that make up an assessment are representative of content that is required to perform a particular activity or task. A test made up of algebra questions given to an applicant for a maths teacher's position would demonstrate content validity.

Many employers use employment tests and other selection procedures in making employment decisions. Examples of these tools, many of which can be administered online, include the following:

- Cognitive tests assess reasoning, memory, perceptual speed and accuracy, and skills in arithmetic and reading comprehension, as well as knowledge of a particular function or job;

- Physical ability tests measure the physical ability to perform a particular task or the strength of specific muscle groups, as well as strength and stamina in general;

- Sample job tasks (e.g. performance tests, simulations, work samples, and realistic job previews) assess performance and aptitude on particular tasks. NB CAD & BIM skills assessments fall into this category;

- Medical inquiries and physical examinations, including psychological tests, assess physical or mental health;

- Personality tests and integrity tests assess the degree to which a person has certain traits or dispositions (e.g. dependability, cooperativeness, safety) or aim to predict the likelihood that a person will engage in certain conduct (e.g. theft, absenteeism);

- Criminal background checks provide information on arrest and conviction history;

- Credit checks provide information on credit and financial history;

- Performance appraisals reflect a supervisor’s assessment of an individual’s performance; and

- English proficiency tests determine English fluency.

An important item to remember when interpreting assessment scores is to put the results in context, and to compare them to an external performance benchmark. For example, a 58% score does not reflect ‘failure’. If this score is presented by a CAD user against an office average of 60%, it is likely the candidate will comfortably fit in to the team.

Test questions should also have a recognizable skill level; basic, intermediate or advanced. In this way, the benchmark data can be reliably used to compare performance and make sound hiring decisions.

In conclusion, for a CAD or BIM skills test to be valid, it must contain content that reflects a representative sample of the target skill. A reliable skills test should comprise sample job tasks (e.g. performance tests, simulations, work samples, and realistic job previews) and assess performance and aptitude on particular tasks.

Skills evaluations should not discriminate according to; race, colour, national origin, sex, religion, age (40 or older), or disability.

Lastly, employers should ensure that employment tests and other selection procedures are properly validated for the positions and purposes for which they are used. The test or selection procedure must be job-related and its results appropriate for the employer's purpose.

Business Benefits of Benchmarking

2010-11-25T02:06:00.000-08:00

Benchmarking is the process of comparing the cost, time or quality of what one organization does against what another organization does. The result is often a business case for making changes in order to make improvements. Also referred to as "best practice benchmarking" or "process benchmarking", it is a process used in management where organizations evaluate various aspects of their processes in relation to best practice, usually within their own sector. This then allows organizations to develop plans on how to make improvements or adopt best practice, with the aim of increasing performance.

The most prominent methodology is the 12 stage approach by Robert Camp (who literally wrote the book on benchmarking in 1989). It consists of;

1. Select subject 2. Define the process 3. Identify potential partners 4. Identify data sources 5. Collect data and select partners 6. Determine the gap 7. Establish process differences 8. Target future performance 9. Communicate 10. Adjust goal 11. Implement 12. Review / recalibrate.

Benchmarking can take various guises:

Process benchmarking - a firm focuses its observation and investigation of business processes with a goal of identifying and observing the best practices from one or more benchmark firms. Activity analysis will be required where the objective is to benchmark cost and efficiency.

Financial benchmarking - a company performs a financial analysis and compares the results in an effort to assess overall competitiveness.

Performance benchmarking - allows a firm to assess their competitive position by comparing products and services with those of target firms.

Product benchmarking - the process of designing new products or upgrades to current ones. This process can sometimes involve reverse engineering competitors’ products to find strengths and weaknesses.

Strategic benchmarking - involves observing how others compete. This type is usually not industry specific, meaning it is best to look at other industries.

Functional benchmarking - a company will focus its benchmarking on a single function in order to improve the operation of that particular function, i.e. Human Resources, Finance and ICT.

Internal benchmarking - involves benchmarking businesses or operations from within the same organisation (e.g. business units in different countries).

External benchmarking - analysing outside organisations that are known to be best in class provides opportunities of learning from those who are at the ‘leading edge’.

International benchmarking - best practitioners are identified and analysed elsewhere in the world; globalisation and advances in information technology are increasing opportunities for international projects.

Benchmarking involves four key steps:

1) Understand in detail existing business processes

2) Analyse the business processes of others

3) Compare own business performance with that of others

4) Implement the steps necessary to close the performance gap

Benchmarking should not be considered a one-off exercise. To be effective, it must become an ongoing, integral part of an ongoing improvement process with the goal of keeping abreast of ever-improving best practice.

Why Bother?

There are many benefits of benchmarking; the following list summarises the main ones:

• provides realistic and achievable targets

• prevents companies from being industry led

• challenges operational complacency

• encourages continuous improvement

• allows employees to visualise improvement which can be a strong motivator for change

• creates a sense of urgency for improvement

• confirms the belief that there is a need for change

• helps to identify weak areas and indicates what needs to be done to improve.

So how does all this apply to us - what are the benefits of benchmarking CAD & BIM performance? In summary:

• Gain visibility of core CAD & BIM skills

• Identify individuals’ strengths & weaknesses

• Implement better CAD & BIM training plans for staff

• Improve recruitment processes

• Share performance data across an organisation

• Promote collaborative working between teams & offices

• Measure the performance of outsourcing or off-shoring partners

• Provide ‘best practice’ for CAD & BIM development

• Offer clearer staff inductions and more meaningful staff appraisals

• Enjoy better skills resourcing for projects

• Develop a continuous improvement process for CAD & BIM

• Save time and money on construction projects and offer best value for clients

In the end, it all comes down to better visibility of your teams and their real ability to use often complex technology and tools for maximum effect. If you can't measure it, you can't manage it.

UK Construction - The Impact of Recession (so far)

2010-11-15T12:07:00.000-08:00

During the summer, I was invited to present to a group of AEC professionals about the impact of the recession on the UK Construction Industry. Whilst still a wee bit premature to be looking back with the benefit of hindsight - let's face it, we're not out of the woods just yet! - it is possible to gather some useful and interesting data on what has been happening during the past 3 years of economic downturn.

First, the basics; a recession is a significant decline in economic activity spread across the economy, lasting more than a few months, normally visible in production, employment, real income, and other indicators. A recession begins when the economy reaches a peak of activity and ends when the economy reaches its trough.

The first of 6 successive quarterly falls in UK GDP started in Q2 2008. The commonly accepted indicator that a recession has begun used to be, “two consecutive quarters of negative growth in GDP”. However, it's not quite so simple!

According to the National Bureau of Economic Research – UK (the official arbiter of recessions) the current recession began as early as December 2007.

5 key measurements NBER UK points to, with respect to a recession:

- GDP (a measure of a country's economic activity, namely of all the services and goods produced in a year)
- Retail/wholesale sales figures
- Overall employment
- Personal income
- Industrial production

Meanwhile, the Business Cycle Dating Committee of the National Bureau of Economic Research (NBER) – USA, maintains a chronology of the beginning and ending dates (months and quarters) of U.S. recessions. The committee determined that a peak in economic activity occurred in the U.S. economy in December 2007.

So whilst many firms have reported that the UK economy really went into a nose-dive in Q4 2008, it was actually nearly a year earlier that the problems began!

Some interesting data was quoted in a Building Magazine editorial, back in June 2010;

Experian predicts a slight upturn for 2011 and 2012 – depending on recovery in the private sector and no more than £5bn cuts in the public sector. (RV - hmmm not sure that they could foresee the current round of cuts when that was written!)

FT review of Office of National Statistics data found the following unemployment figures:

720% increase for architects
655% increase for QS’s
Similar figures for Planners, Construction Managers & Engineers (500%+)

Earlier in the year, the Chartered Institute of Building (CIOB) published the results of their third annual survey, where they polled opinion from 1182 UK construction firms.

Results show that the industry is still suffering a skills shortage despite the recession and downturn in construction demand.
77% of respondents believe there is a skills shortage in construction and 78% of those feel that the loss of skills will hinder the industry’s recovery when the economy improves.
54% state that their company has had to make redundancies and 14% expect further redundancies to occur.

Similar comments echoed by Construction Skills, who observed; 'Forecasts demonstrate the severe effect that the recession is having on the UK construction industry, with rising job losses risking huge skills deficits in the long term.

The Construction Skills Network (CSN) shows the prognosis for the UK construction industry in the short term (2009 to 2010) is poor. The industry is expected to contract in construction output terms by at least 12% in 2009, followed by a more marginal decline of around 2-4% in 2010.

Consistent recovery is not forecasted until 2011 and even then, it is likely to be a slow and steady return to moderate levels of growth as confidence returns to the market.'

Construction News published the following figures;

UK Construction employment began to fall in 2008 with a decline of 1% over the year, however, CSN forecasts a much larger drop of 15% across 2009 to 2010, with the largest losses (around 13%) expected in 2009. The total number of construction job losses from 2009 to 2010 could be up to 450,000 if output contracts by the suggested higher rate of 13% in 2009. The slow recovery forecast from 2011 to 2014 is set to create around 125,000 jobs.
Coupled with the large number of workers who are set to leave the industry in the next 10 years through retirement, the need to increase the flow of apprentices and graduates into the industry and retain as many existing skilled workers as possible is vital if the UK is to sustain a skilled construction workforce.

Data from the CIOB survey, noted the following points about Apprentices & Graduates;

Only 37% are sure their companies are still employing apprentices.
11% stated that their companies usually employ apprentices, but cannot afford to in the current economic climate.
12% of respondents are aware of their companies recruiting more graduates.
Only 1% are recruiting the same number of graduates as before the recession.

So what are the lessons we can take from all of this? What mistakes have firms made and what are the opportunities for growth when things pick up?

Mistakes:

Panic firing/firing the wrong people, i.e. firing the best software users or even the CAD/BIM Manager!
Hoping things will get better sooner; waiting too long to make key strategic decisions
Freeze on technology investment
Freeze on all hiring
Freeze on all training & learning budgets
Lack of cash flow management

Opportunities:

Removal of ineffective staff (who perhaps shouldn’t have been hired in the first place?)
Better users remaining
More time to spend on IT strategy & ‘housekeeping’
Better hiring practices in future
Better training & learning plans in future
Better cash flow procedures
Better, more realistic business plans
More focused roadmaps for technology, marketing and business strategy
Better relationships with remaining clients
Leaner working practices
More enthusiastic adoption of BIM

A final word from an article which appeared in the FT in the summer; “Those businesses that survive a recession of this magnitude, are the ones who are best placed to ride the wave back up again!”.

UK BIM Roundtable

2010-11-15T04:12:00.000-08:00

Recent months have seen a plethora of forums and conferences about this year's numero uno hot potato, namely, BIM. I've attended a number of these sessions and, whilst the networking is always useful, the content has varied tremendously. So many meetings still focus on the big picture of BIM, almost selling the idea that BIM is the future and we all need to get on board. I think, for most firms, they've already made that leap. They kind of get it already, that we can all do better in terms of delivering more efficient projects and harnessing the latest technology to make it happen.

The question now is, how exactly do we, as individual firms and collectively an industry, sally forth and turn all the rhetoric into BIM-reality? If you speak to the marketing teams of many UK design & engineering firms, the story is that BIM is altogether sorted, that they are 100% BIM proficient and can deliver projects in any technology, at any time, with maximum efficiency, no fuss and zero downtime! :) Talk to the operations teams, who are tasked with actually making this all happen, and the story isn't quite the same. And this is in no way a criticism of these hard-working people. Quite the opposite. But, ironically, when firms speak openly about this stuff, the reality is that they are all wrestling with the same challenges!

To this end, we've been working with a group of connected people this year, to put together a programme of round table meetings, whose sole aim is to provide detailed answers to some key questions about BIM - at a practical level. The nice thing about this initiative is that no-one is forcing an agenda onto the proceedings. Whilst we all have a vested interest in the UK Construction industry picking itself up off the floor, the rest is all about using whatever connections or influence we may have individually, and pooling our resources to bring together the right people, at the same time, to figure out some new answers to these questions.

This Thursday in London, we have the first UK BIM Roundtable. All being well, we anticipate that this could develop into a regular series of half-yearly, or even quarterly, forums, each addressing a different area of BIM.

The first meeting focuses on the dynamics of the relationship between Architects and Engineers and General Contractors. Discussion will be based around the following topics:

• What are the needs and requirements of consultants and what are the needs and requirements of contractors, when undertaking a BIM project?

• How can we negate conflicts occurring at the interface of a design-based BIM as it undergoes the transition to the construction phase?

• Getting BIM deliverables in sync throughout the project; exploring commonly accepted workflows and identifying the requirements of each discipline.

The meeting is by invitation and all 30 available places were filled very quickly. The following firms will have representation at the session;

Consultants: AECOM, WSP, Buro Happold, Halcrow, Hoare Lea, Capita Architecture, Zaha Hadid, Atkins, Levitt Bernstein, Foster + Partners, Davis Langdon, Waterman, IBI Nightingale, HOK, Tribal, Jacobs, Ramboll UK, Scott Wilson, BDP.

Contractors: Skanska, TPS Carillion, Carillion, Bovis Lend Lease, Mace, Laing O'Rourke, Costain, BAM, Vinci, Robert McAlpine, plus UKCG.

We also anticipate having people from TfL, Crossrail and BIS in attendance.

This is designed to be an interactive discussion forum. There will be an agenda and specific objectives for the session, steered by co-chairs representing both groups, but the rest is up to the delegates. We have no commercial sponsors, nor formal presentations from pre-arranged speakers. The minutes of the roundtable will be documented and issued in the form of a short report following the meeting. Thanks to Buro Happold for providing the venue for this meeting.

Sketchup is dead! (Long live Sketchup!)

2010-10-27T06:11:00.000-07:00

I've noticed an interesting trend on my travels this year, regarding what was, up until recently, the darling of the design office (at least from a drafting software pov) - yes, I mean our old friend Google Sketchup.

We have had a ton of requests for a Sketchup test this year. And I should make this clear, we have manfully resisted this for quite some time, feeling that a Sketchup test was somehow an exercise in overkill. Many times I have asked the question, 'Why on earth would you want a Sketchup test?!'.

Nevertheless, it still keeps coming up in feedback forums, so it has been added to our list of short term devs.

However...

...I don't know if something is in the water, but over the past few months, I have been in meetings with heads of design and technology (mostly in Architectural firms, to be fair) where Sketchup is no longer the poster-child of design software. Oh no, in fact, quite the opposite! 'We're trying to kill Sketchup', was the anguished cry from one leading design firm in New York. And back in Blighty, the same trend is emerging; 'Sketchup has become my biggest pain in the ****!', bemoans another technology leader.

But what has prompted such a collective venting of creative spleens, I hear you cry? Well, it seems that, whilst Sketchup is great for knocking out quick visualizations, too many people are trying to use it for much more than that.

In some ways, Sketchup is becoming a victim of its own simplicity. Because Architects are nowhere near as scared of learning Sketchup as they are of, say, Revit or MicroStation, they're now trying to use it for all sorts of inappropriate outputs. Let's make this clear - Sketchup is not Revit. You can't create a BIM model in Sketchup - so stop trying!

And it doesn't stop there. Not only is misguided use of the drafting tool causing the collective blood-pressure of BIM managers everywhere to reach Vesuvian levels, it is also now being cited in a couple of legal cases where, I kid you not, firms of Architects are being sued by clients for mishandled project information leading to mistakes during a build. Turns out that shadow diagrams quickly crunched out in Sketchup have been used as hard data during a build, which, in turn, has led to some pretty upset clients!

Suddenly, the ramifications of using technical software inappropriately on projects isn't funny at all. It can cost AEC firms thousands of dollars in lost income - not to mention the management time that goes into yet another round of unnecessary litigation. And those design firms who poo-poo the importance of technical software as a highly sought after skill? You have been warned. No longer can you get away with the familiar refrain, 'We only employ good designers. Software skills don't matter - we can always get them up to speed in Revit later'. (For the record, my blood freezes whenever I hear those 5 words - get... them... up... to... speed).

Oh, and don't blame Sketchup for this mess. It's not the one at fault here! The clue is in the name, stupid! ;)

R

Herbert Hoover's Engineer

2010-09-27T03:17:00.000-07:00

I saw an engaging post on a LinkedIn discussion forum this week, about why people became engineers and what their favourite quotes are about the profession.

Here's the one I liked best, a quote from engineer and past US President, Herbert Hoover:

"It is a great profession. There is the fascination of watching a figment of the imagination emerge through the aid of science to plan on paper. Then it moves to realization in stone or metal or energy. Then it brings jobs and homes to men. Then it elevates the standards of living and adds to the comfort of life. That is the engineer’s high privilege.

The great liability of the engineer compared to men of other professions is that his works are out in the open where all can see them. His acts, step by step, are in hard substance. He cannot, like the architect, cover his failures with trees and vines. He cannot like the politician screen his shortcomings by blaming his opponents and hope the people will forget. The engineer simply cannot deny he did it. If his works do not work he is damned.

On the other hand, unlike the doctor, his is not a life among the weak. Unlike the soldier, destruction is not his purpose. Unlike the lawyer, quarrels are not his daily bread. To the engineer falls the job of clothing the bare bones of science with life, comfort and hope. No doubt as years go by the people forget which engineer did it, even if they ever knew. Or some politician put his name on it. Or they credit it to some promoter who used other people’s money…

But the engineer himself looks back at the unending stream of goodness which flows from his success with satisfaction that few professions may know. And the verdict of his fellow professionals is all the accolade he wants."

540,000 hours

2010-08-19T02:43:00.001-07:00

I was chatting to one of our large AEC customers last week. They have about 1,000 CAD, BIM & Engineering software users around the group. During our chat, we discussed the fact that they are putting together a business case for management, aimed at securing additional budgets for investing in new training & learning tools. As part of this process, they worked out that they spend approx. 540,000 man (or woman) hours EACH YEAR, producing CAD drawings. And that's just in the UK, which is only part of the wider group!

I don't know what their hourly charge-out rates are, but let's say, for argument's sake, that the figure is £100/hr. That's a cool £54M per year on CAD production alone.

Now let's picture the impact of a 2% improvement in productivity across the year - that's over £1M in savings, or half a million quid for every 1% increase!

Suddenly, spending a few grand on testing and training tools starts to make a lot of business sense!

Let's look at some more numbers;

Take an average salary of £25k p.a. and a 225 day working year.

For every 1% improvement in productivity, a practice will save £250 or 2.25 days per person per year. Targeting a 5% annual improvement in performance, this is worth £1,250 or 11.25 days per person per year. (11.25 days = 90 working hours (average 8 hour work day)).

For a firm with 20 staff, this is worth the equivalent of a new member of staff! (20 x £1,250 = £25,000).

Now let's look at hiring; how much does it cost to employ inappropriate personnel for 3 months?

Assuming a £25k annual salary:

15% agency fee = £3,750

3 months’ salary = £6,250

Assuming £18/hr:

7.5 hrs x 5 days = £675 x 12 weeks = £8,100

These examples don’t include advertising costs for staff employed direct, or the cost of re-working for drawings which weren’t done right first time.

Daisy-chain

2010-08-05T10:52:00.001-07:00

This month, we are introducing a change to the way assessments are authored, based on the idea of linking separate modules in a 'daisy-chain', to create a longer test.

In essence, the KS system breaks down like this;

- A test is one or more modules linked together.
- A module is comprised of typically 4-6 questions, covering the same topic, i.e. Families in Revit or UCS in AutoCAD, etc.
- A question tests an individual's understanding of a particular software feature or command.

Questions and modules have training tag keywords (meta tags) associated with them, which makes training needs analysis, post-assessment, much more relevant.

Questions can be "task" based or "knowledge" based.

A task based question has files associated with it. The user has to do something to the file, in order to work out the answer, as follows;

- Task (interrogate) - the user has to open the file and find the answer, but they don't have to modify or change the file in any way.
- Task (modify) - the user has to open and modify the file, to work out the correct answer.
- Knowledge - this type of question does not require a file to be associated with it.

Test administrators can present a single module as a test, provided they create a new test from the module first.

A 'daisy-chain' happens when an administrator links a series of modules together, to create a longer test.

This concept allows us to create a wide range of test material, in bite-sized blocks, which map closely to corresponding training and learning material.

Daisy-chains go live next month.

R

MCMC 2010

2010-08-05T09:05:00.000-07:00

It's that time of year again! One of the year's most enjoyable events takes place on August 10-11; the Mid Continent MicroStation Community User Conference. The venue is once again the excellent Overland Park Convention Center, in Kansas City.

The organisers have pulled out the stops, with an agenda packed full of high-quality speakers, over the course of the two days.

See this link for full details:
http://www.tmcmcmc.org/meetings/2010/Onsite_Agenda_2010.pdf .

Tuesday night's entertainment includes dinner and a repeat trip to Power Play, for another high energy Whirlyball contest. (For the un-initiated, Whirlyball is a mix of lacrosse, basketball - and bumber cars - not for the faint hearted!).

Tickets are just $95 per day, including lunch and snacks throughout the day.

The theme for this conference is Versatility. Covering multiple tracks, including, MicroStation, Civils, Plotting, CAD Management, Collaboration, Building, Structural Analysis and US Army Corps projects.

What happened in Chicago...

2010-06-09T01:43:00.000-07:00

A brief postscript to my recent visit to Chicago, where I attended the inaugural KA-Connect Knowledge management conference. The team at Knowledge Architecture have been posting some of the presenter videos recorded over the two days, onto their website.

Here is mine: http://ka-connect.com/talks.php?vdx=48.

The funny thing about my presentation was that I had the very last slot on day 2. It's fair to say that of my Pecha Kucha session - 20 slides, 20 seconds per slide - about 75% of what I had to speak about, had already been covered at some point during the previous 35 talks! So much of what you see here is delivered on the fly. :)

There were some terrific presenters during the course of the 2 days. Here are some of my favourites:

Laurie Dreyer, head of HR at US design firm Anshen + Allen (34 mins long, but frankly unmissable viewing!): http://ka-connect.com/talks.php?vdx=40.

John Moebes, Director of Construction at Crate+Barrel, with a fascinating insight into what owners expect from their design, engineering and construction partners:
http://ka-connect.com/talks.php?vdx=2.

Michael Kilkelly, Associate at Gehry Partners, gives us a look behind the scenes of one of the world's leading design firms: http://ka-connect.com/talks.php?vdx=57.

KA-Connect 2011 takes place in San Francisco, in April 2011.

R

EYC CAD Managers Forum - summer 2010

2010-06-04T06:46:00.000-07:00

I'll be joining the EYC team at this summer's CAD Managers Forum in London, later this month.

See link for full details: http://www.eatyourcad.com/resource.php?resource_id=1676

The meeting takes place on June 30, at Buro Happold's Newman Street offices, from 10am to 4.30pm.

There are two main topics on the agenda; I'll be talking about the effects of the recession (isn't it over yet??) and what lessons we may have learned. The afternoon session looks at BIM contracts, technology and deliverables.

All in all, it promises to be an excellent forum. Tickets are just £50, to cover EYC's costs, so be sure to book your slot - and I look forward to seeing you in London.

R

Coming up in KS..

2010-06-04T06:06:00.000-07:00

Just back from a very enjoyable few days in Philadelphia, attending an excellent Bentley conference, Be Together. The keynotes were thought-provoking, the seminars plentiful, the after-hours activities stimulating and the live zone format worked a treat. Slightly earlier promotion of the event next time, coupled with a bit more time between sessions, and it's a winner.

Anyway, I digress. The point of this post is to shed some light on what's coming up on the KS dev side over the summer period. Our technical roadmap has always been, I think it's fair to say, a somewhat fluid affair. We regularly receive tremendously bright feedback from our user-base, which has a direct impact on our delivery schedules. (Especially the really good ideas, when I ask the team, 'Why didn't we think of that!?' ;) ).

So, here's what we're working on over the next 2-3 months, in rough priority order;

System architecture

- Options for admins to deploy test files locally.
- More searching fields in the results look-up area of the dashboard.
- Additional charting options and export of results feature.
- Individual dashboard login for users to track their own test history and results, training recommendations and coaching notes.
- Improved content management tools, for accounts with large question libraries.
Additional benchmark data and statistics for OTS (off-the-shelf) tests.

We're currently working on the following test modules, which will be going live between June and September;

Test Libraries

- MicroStation v8i 2D for occasional users
- AutoCAD 2D for occasional users
- MicroStation v8i 3D fundamentals
- Design Review
- Bentley View
- 3ds Max 2011 (4 modules; Fundamentals, Rendering, Modeling and Maps & Materials)
- Revit MEP (1st of 3 modules)
- Bentley GEOPAK (1st of 4 modules)
- Bentley InRoads (1st of 4 modules)
- Rhino
- Adobe Photoshop
- Google Sketchup
- Bentley ProjectWise
- Bentley Navigator (1st of 2 modules)
- Bentley Structural Modeler
- Civil 3D (1st of 4 modules)

Towards the end of the year, we'll be turning our attention to analysis tools, including; Navisworks Manage, Ecotect, GBS, Hevacomp and IES VE. Bentley RAM and Bentley Map are also coming up more in conversation, so we'll endeavour to squeeze those two in somewhere as well!

As with all technical wish-lists, it's a constant juggle of time and money. With respect to our content libraries, I fear we might have a job for life in attempting to deliver quality test material across such a broad product base! Our challenge is principally that of, 'do we go wide or deep?' i.e. do we attempt to add a wide number of test topics with relatively small question libraries, or do we offer less topics, with greater numbers of questions for each? In truth, there is no right or wrong answer, hence the perpetual juggling act. As in all things we do, the answer ultimately lies with our customers; 'What do they want us to do?', is the question which shapes our ultimate roadmap.

As anyone who develops software or learning material will attest, the job is never truly finished. There will always be a new product update, or a change in technology, or a change in industry direction, or a change in vendor strategy, or yet another acquisition, the list goes on.

Whilst we're pleased with the progress we've made in the arena of CAD, BIM & engineering software skills testing, over the past 7 years or so, I sometimes feel that we're just getting started! The industry has changed rapidly over the past couple of years, impacted in considerable fashion by the twin effects of BIM adoption and economic hardship. And so, inevitably, the technology roadmap for AEC firms has been impacted.

As always, if you have any suggestions, ideas, requests for test content or general feedback, we'd love to hear them. Our own roadmap is, after all, in your hands! :)

R