Benchmark Stats for Popular KS Test Titles

Benchmarking software capability at individual, team and office level is a key part of the KnowledgeSmart experience.

Our library of skills assessments for the AEC industry covers a variety of  popular software tools, used on projects on Construction projects worldwide.

Our results data comes from a wide variety of countries and territories, including: US, UK, Australia, Canada, New Zealand, South Africa, Singapore, Vietnam, India and the Middle East.

Here is the overall average results data for our most popular topics (% score and time to complete):

Test Title	% Score	Time
3ds Max fundamentals	54.5	57m
Adobe InDesign occasional	64	45m
Adobe Photoshop occasional	62	48m
AutoCAD 2D fundamentals	58.5	1h 13m
AutoCAD 2D occasional	73.1	36.5m
AutoCAD Civil 3D fundamentals	67.2	1h 3m
AutoCAD Civil 3D advanced	52.2	45.5m
AutoCAD Plant 3D fundamentals	63	53m
Bentley ABD fundamentals	51.5	1h 28m
Bentley ProjectWise fundamentals	59	1h 3m
BIM Management	68.5	46m
InfraWorks 360 fundamentals	58.5	1h 2m
Inventor fundamentals	64	1h 15m
MicroStation 2D fundamentals	59.3	1h 24m
MicroStation 3D fundamentals	52.5	58m
Navisworks fundamentals	58.4	43m
Revit Architecture fundamentals	68.5	1h 8m
Revit Architecture advanced	57.1	1h 24m
Revit Content Creation	57.8	1h 3m
Revit MEP fundamentals - Electrical	63.7	1h 6m
Revit MEP fundamentals - Mechanical	62.2	1h 7m
Revit MEP fundamentals - Plumbing	70.1	1h 11m
Revit MEP advanced	53.3	1h 2m
Revit Project Process	56.4	46.5m
Revit Structure fundamentals	67.1	1h 12m
Revit Structure advanced	57.2	1h 4m
Rhino	61.2	44.5m
SketchUp	53.4	50m

KnowledgeSmart independent skills testing enables businesses to compare performance and capability against industry standards and benchmark their organisation against peers, competitors and partners. Measuring and demonstrating technical software experience and BIM capability is a key differentiator for AEC firms when bidding for new projects.

R

KS Library Topics - Assessing Revit Architecture Skills

KnowledgeSmart skills assessment software helps firms of architects and engineers to measure software capability.

We have created an extensive range of assessments, covering the most popular design, engineering and BIM software titles, used by firms working in the global construction industry.

Assessments cover a range of popular vendors, including: Autodesk, Bentley Systems, Adobe, McNeel, Trimble and Graphisoft.

We work with a global network of world class authors. They are subject matter experts, published, thought leaders and working as consultants to industry, so they understand the practical application of the tools, as well as the menus and mouse-clicks. Popular KS authors include: CASE Inc, Chris Senior, Darryl McClelland, Eric Chappell, Envision CAD, Evolve, John Evans, Josh Modglin, Joel Harris, Paul Aubin, Paul Woddy, Revit Factory, Robert Manna, Scott Moyse, Scott Onstott, Tony Tedder, Thomas Weir and White Frog.

For our most popular assessments, we tend to create a variety of different modules, covering a range of different levels and abilities.

The current number one topic in our library is Autodesk Revit Architecture.  Here is a brief list of assessments available in our 'RAC Bundle'.  All of the following titles are available, as part of a KS Professional 1 license:

Revit Architecture for occasional users
Revit Architecture fundamentals
Revit Architecture – Xpress
Revit Architecture for Interiors
Revit Architecture advanced
Revit Content Creation
Revit Project Process
Revit Architecture – Extra qu's 1
Revit Architecture – Extra qu's 2
KS Community – Revit Architecture (White Frog)
KS Community – Revit Process & Workflow
KS Community – BIM Management (NB this one is software neutral, but included in the bundle)

So how does it all work?  In simple terms, it is a web based, practical test of knowledge and ability, so users will need access to a copy of the software, for which they are being assessed. They are presented with a series of task based and knowledge based questions, entering answers into the KS browser screen. At the end, a summary report with detailed feedback and coaching notes can be displayed to the user.  KS system administrators can access detailed reports, highlighting results, benchmark statistics and training recommendations.

Let's take a look at an actual example of our Revit Architecture fundamentals assessment, in this short video:

KS customers have full editorial control over all KS library material, so they are ultimately in the driving seat, as far as the material that is presented to their teams goes. And they have the option of writing custom question from scratch, using the KS authoring tools, if they want to capture knowledge on in-house work flows, standards and processes.

R

3 Levels of Revit Proficiency

How do you identify someone who has basic skills in Revit and distinguish them from an advanced user?

Further, do AEC firms actually need all of their users to be 'specialists' in Revit?  In some instances, people just need to receive a file, open it up and perform some straightforward analysis of the model (without causing any problems).  For example, Construction Managers, PM's, Engineers, Project Architects, and the like.  So a basic working knowledge of some key concepts is perfectly adequate for these users.

For the primary modeling team, AEC firms require a broader level of understanding across their user base, for example the people tasked with creating the detailed production information and modeling outputs on a project.  And firms need to have a handful of Revit ‘specialists'; people with a deeper understanding of the way in which Revit is deployed on a project, an appreciation of the collaborative process and the impact of using the technology across a broader environment.  And lastly, if content creation is a key part of the firm’s process, a handful of Revit Family specialists, who understand the importance of creating quality content, that can be re-used on subsequent projects, not simply re-modeled from scratch every time.

So it makes sense to establish 3 recognised levels of competence.  In a broad sense, we could label them:

• Level 1 – basic skills
• Level 2 – intermediate skills
• Level 3 – advanced skills

With respect to Revit, this could work as follows:

• Level 1 – Basics of Revit (i.e. an assessment aimed at occasional users; Construction Managers, PM’s, Engineers, Project Architects, etc.)
• Level 2 – Revit fundamentals (i.e. core skills for primary modelers) 
• Level 3 – More advanced concepts of Revit, (i.e. Revit project process & workflow, collaborative working, Family creation, etc., for BIM Co-ordinators and model managers)

AEC firms can use skills assessment, as a means of demonstrating to clients (or prospective clients) and partners, that they have the skills required, to truly deliver on a BIM project.  

KS Library 

In the KnowledgeSmart library, we have a range of modules available, which can assist firms in creating an appropriate assessment program for their organisation to capture helpful benchmark data and skills gaps information.

• Level 1 – Revit for occasional users (available Summer 12).

• Level 2 – Revit Architecture fundamentals, Revit for Interiors, Revit MEP fundamentals, Revit Structure fundamentals.

• Level 3 – Revit Project Process, Revit Process & Workflow, Revit Architecture advanced (available Summer 12), Revit Structure advanced (available Summer 12), Revit Content Creation.

Firms might also consider writing their own custom assessment modules, based on their BIM standards.  All users can demonstrate an understanding of the correct standards and protocols, by passing this module.

Self-Assessment

In the broader context of self-assessment, the “above-average effect” is the tendency of the average person to believe he or she is above average, a result that defies the logic of statistics.

In studies, participants scoring in the bottom quartile on tests grossly overestimate their performance and ability. Although test scores put them in the 12th percentile they estimate themselves to be in the 62nd.

Because top performers find the tests they confront to be easy, they mistakenly assume that their peers find the tests to be equally easy. As such, their own performances seem unexceptional.  In studies, the top 25% tend to think that their skills are in the 70th–75th percentile, although their performances fall roughly in the 87th percentile.

Business Benefits

There are five distinct business areas that can benefit from measuring team performance:

Project Technology Leadership:

• Measure returns on technology investment to eliminate risk of waste.

Business Development Team:

• Differentiate your services and benchmark your competitive advantage to win more work.

HR Leadership:

• Optimise the mix of people and skills via regular assessment to retain top talent.
• Filter out poorly skilled candidates at interview to avoid failed hires saving time and money.
• Encourage on the job training to reduce time away from the office and increase staff impact.

Learning and Development Team:

• Train staff based on individual needs to maximize impact and raise service quality.
• Measure returns on learning and development investments to eliminate risk of waste.
• Make your training budget go further by improving learning and development efficiency.

Project Delivery Team:

• Resource project teams with a balanced mix of skills to deliver high-quality services profitably.
• Create an environment of team learning and growth to improve client service levels.

Benchmarking

Benchmarking is the process of comparing the cost, time or quality of what one organization does against what another organization does. The result is often a business case for making changes in order to make improvements. Also referred to as "best practice benchmarking" or "process benchmarking", it is a concept used in management where organizations evaluate various aspects of their processes in relation to best practice, usually within their own sector. This then allows organizations to develop plans on how to make improvements or adopt best practice, with the aim of increasing performance. 

It is important to target your teams with clear goals for advancement, as part of your wider learning & development and talent management strategy.

Here are the current (2012) benchmark performance quartiles, for the 3 disciplines of Revit, for your firm to compare results against:

Revit Architecture 2012:  69% in 1 hour 4 mins 30 secs

Revit MEP 2012:  64% in 1 hour 9 mins 30 secs

Revit Structure 2012:  62% in 1 hour 6 mins 0 secs

New Hires

According to the report “U.K. Talent Acquisition Factbook 2011” by Bersin & Associates, the new hire failure rate for new recruits is 1 in 8. Put another way, for every 8 new recruits 1 will leave inside of 12-months because they were a bad fit for the company. The same report measured the average recruitment cost to be £5,300 GBP / $8,200 USD per person.

Learning & Development Data

According to data collected from software training professionals, many trainers spend approximately 40% of the training course time revising prior training topics to assess group skill levels. This ‘Redundant Training Time’ reduces classroom productivity, increases training costs, and keeps staff away from their project teams for longer.

Skills Gaps

Based on analysis of KS results data from firms located in the UK, US, Canada, Australia and New Zealand, here is a summary of the top 20 most commonly flagged training issues and skills gaps, for each of the 3 Revit disciplines.

Revit Architecture 

• ViewCube

• View Range

• Borrowers

• Family Editor

• Families

• Conceptual Massing

• View Properties

• Element Properties

• User Interface

• Detail Components

• Worksets

• Coordinates

• Rooms

• Walls

• Doors

• Floors

• Calculating Area

• Shortcuts

• Schedules

• CAD Files

Revit MEP

• Apparent Load Values

• Modifying Light Fixtures

• Photometric Web Files

• System Inspector

• Creating Power Systems

• Edit Properties

• Panel Circuits

• Panel Schedules

• Selecting Equipment

• Electrical Settings

• Creating Supply Systems

• Duct Systems

• Load Family

• Pipe Systems

• Spaces

• Family Elements

• Plumbing Fixtures

• Properties

• Visibility/Graphics

• Worksets

Revit Structure

• Beams

• Components

• Cut Length

• Element Properties

• Roofs

• Families

• Instance Properties

• Schedules

• View Parameters 

• Annotation

• Scope Boxes

• Visibility

• Structural Deck

• Span Direction Symbol

• Slabs

• Filters

• Slope Arrows

• Dimensions

• Repeating Details

• Structural Settings

For firms involved in the adoption, rollout and training of their teams in Revit, skills assessment and benchmarking is an essential ingredient, in the overall successful deployment of this technology.

Taking a more structured, scientific approach will help AEC firms, Construction Clients and Contractors alike, to get a much better handle on who knows what, with respect to deploying Revit on a BIM project.  It will also help firms to demonstrate their BIM credentials and give users a logical (and more appropriate) path for skills development.

R