'BIG 5' Benchmark Stats

If your firm increases productivity by 1% per person annually, you gain 2 extra working days per team member.

How do you measure your firm's productivity against your peers and the wider AEC industry?

KnowledgeSmart has been capturing test scores across a wide range of technical software products, used by Architects & Engineers around the world.

We have data-mined thousands of results for the 'BIG 5' Autodesk software titles, as follows:

Revit Architecture
Revit MEP
Revit Structure
AutoCAD
AutoCAD Civil 3D

The results form a fascinating insight into the way these tools are deployed and utilised in different territories around the world. This is the most detailed study of software proficiency ever carried out with live test data, across thousands of users and hundreds of AEC businesses.

A detailed PDF report can be downloaded for each topic from this microsite.

Now, for the first time, management teams can compare their in-house performance and technical software capability against the wider Global industry benchmark.

This data can be included in the hiring process, new team selection, annual appraisals and when bidding for new project work.

R

KS exhibiting at Autodesk University

A key theme for us this year is that of measuring technical software skills and performance, against a wider body of results data. We are featuring our analysis of the 'BIG 5' Autodesk skills assessments at this year's Autodesk University conference, which takes place from 15-17 November, in Las Vegas, USA.  Here is the conference link: http://au.autodesk.com.

The BIG 5 tests are:
Revit Architecture, Revit Structure, Revit MEP, AutoCAD and AutoCAD Civil 3D.

KnowledgeSmart will be exhibiting in the main hall, at booth number 2810.

We look forward to seeing you there!

R

How Good Are Engineers at Using Revit Structure?

With the global adoption of Building Information Modelling (BIM) well underway, it is a huge challenge for Engineering businesses to manage the change in their project delivery processes from more conventional drawing production environments, to that of virtual design, clash detection and 'intelligent' modelling.

Looking at the global AEC picture, it is clear that the commercial software application adopted by a wide majority of the world's Structural Engineering firms, as their modelling tool of choice, is Autodesk's Revit Structure platform.

In recent years, KnowledgeSmart's online skills assessment software platform has captured test scores from thousands of Engineers, representing hundreds of firms, spread across six continents and a couple of dozen countries.

We have analysed the results data generated from these live assessments, at a basic skills level, and noted the following trends. For this particular study, the majority of our data is generated in the USA, UK, Canada and Australia (in that order).

A simple key to interpret the data below:

Str Eng = Job Title

RST = Revit Structure

1 to 5 = 1 to 5 years' experience using the tool

6 to 10 = 6 to 10 years' experience using the tool

Regular = use the tool daily (or almost daily)

Occasional = use the tool weekly (or less than weekly)

Self taught = learned to use the tool primarily 'on the job'

Formal = learned to the tool primarily via formal classroom or online training

All = overall KS benchmark data

AUS = Australia

CAN = Canada

UK = UK

USA = USA

Here is a summary of the results presented by different types of Engineers.

Overall

RST Fundamentals score overall: 67.1% in 4320 s

Str Eng/RST/1 to 5/regular/self taught/All = 59.3% in 4854 s

Str Eng/RST/1 to 5/regular/formal/All = 58.2% in 4738 s

Str Eng/RST/6 to 10/regular/self taught/All = 68.3% in 4141 s

Str Eng/RST/6 to 10/regular/formal/All = 67.4% in 4372 s

Str Eng/RST/1 to 5/occasional/self taught/All = 55.2% in 5353 s

Str Eng/RST/1 to 5/occasional/formal/All = 54.5% in 5244 s

Str Eng/RST/6 to 10/occasional/self taught/All = 63.3% in 4967 s

Str Eng/RST/6 to 10/occasional/formal/All = 62.5% in 4777 s

Australia

RST Fundamentals score Australia: 68.7% in 4191 s

Str Eng/RST/1 to 5/regular/self taught/US = 59.4% in 4786 s

Str Eng/RST/1 to 5/regular/formal/US = 58.0% in 4553 s

Str Eng/RST/6 to 10/regular/self taught/US = 68.6% in 4003 s

Str Eng/RST/6 to 10/regular/formal/US = 67.2% in 3967 s

 

Str Eng/RST/1 to 5/occasional/self taught/US = 53.4% in 4822 s

Str Eng/RST/1 to 5/occasional/formal/US = 52.0% in 4687 s

Str Eng/RST/6 to 10/occasional/self taught/US = 59.6% in 4566 s

Str Eng/RST/6 to 10/occasional/formal/US = 58.7 in 4330 s

Canada

RST Fundamentals score Canada: 72.4% in 3886 s

Str Eng/RST/1 to 5/regular/self taught/US = 63.4% in 4378 s

Str Eng/RST/1 to 5/regular/formal/US = 61.6% in 4233 s

Str Eng/RST/6 to 10/regular/self taught/US = 72.6% in 3717 s

Str Eng/RST/6 to 10/regular/formal/US = 71.4% in 3578 s

 

Str Eng/RST/1 to 5/occasional/self taught/US = 58.1% in 4516 s

Str Eng/RST/1 to 5/occasional/formal/US = 56.7% in 4439 s

Str Eng/RST/6 to 10/occasional/self taught/US = 63.6% in 4248 s

Str Eng/RST/6 to 10/occasional/formal/US = 62.5% in 4036 s

UK

RST Fundamentals score UK: 71.1% in 4887 s

Str Eng/RST/1 to 5/regular/self taught/US = 62.2% in 5343 s

Str Eng/RST/1 to 5/regular/formal/US = 60.6% in 5154 s

Str Eng/RST/6 to 10/regular/self taught/US = 71.3% in 4673 s

Str Eng/RST/6 to 10/regular/formal/US = 70.1% in 4491 s

 

Str Eng/RST/1 to 5/occasional/self taught/US = 58.4% in 5616 s

Str Eng/RST/1 to 5/occasional/formal/US = 56.9% in 5552 s

Str Eng/RST/6 to 10/occasional/self taught/US = 62.6% in 5254 s

Str Eng/RST/6 to 10/occasional/formal/US = 61.7% in 4987 s

USA

RST Fundamentals score USA: 73.7% in 4218 s

Str Eng/RST/1 to 5/regular/self taught/US = 64.6% in 4843 s

Str Eng/RST/1 to 5/regular/formal/US = 62.9% in 4661 s

Str Eng/RST/6 to 10/regular/self taught/US = 73.8% in 4073 s

Str Eng/RST/6 to 10/regular/formal/US = 72.1% in 3891 s

 

Str Eng/RST/1 to 5/occasional/self taught/US = 59.3% in 4916 s

Str Eng/RST/1 to 5/occasional/formal/US = 57.9% in 4856 s

Str Eng/RST/6 to 10/occasional/self taught/US = 64.8% in 4668 s

Str Eng/RST/6 to 10/occasional/formal/US = 63.7% in 4448 s

Observations

All four territories in this study are mature markets, in terms of Revit adoption. 

Each country has an established Autodesk Training Centre market, offering high quality learning and development opportunities and consultancy. 

Each country has a pool of further education colleges, community colleges and universities, many of whom teach technical software skills as part of courses in the built environment. 

Each country is well supported by professional bodies and information-rich conference series', such as BIM Forum, RTC, AU, BIM Day Out and BIM Show Live.

Overall, regular users are more accurate and faster than part-time users.

Engineers with 6 to 10 years' of experience with these tools are the most efficient. They reach a plateau in performance after c. 7 years and make no further productivity gains.

Engineers who learn to apply these tools on projects are more accurate than those who have primarily formal training, but less practical experience. Formally trained users are faster overall than self-taught users.

R

Global Test Scores Posted by Structural Engineers

Structural Engineers from across the world have submitted a wide variety of KS test scores, in recent years. Let’s take a more detailed look at the people behind the headline stats.

We capture some interesting user demographic data at the start of each test session. This includes the following questions:

- Primary Industry/Discipline
- Primary Role
- Country & State
- Self rating
- How many years have you used this tool?
- Where did you learn to use this tool?
- How often do you use this tool? (Daily, weekly, occasional)
- How did you learn to use this tool? (On the job, Formal training, College)

The top 2 tests completed by Structural Engineers are, in order (by volume):

Revit Structure
AutoCAD

The results data comes from a wide range of countries and territories, primarily, US, Canada, UK, Australia, South Africa, Singapore, HK and Middle East regions.

Here are the overall results:

Revit Structure

Avg score/time for:

Regular users = 71.1% in 4201 secs

Part-time users = 67.2% in 3871 secs

1 to 5 years = 66.7% in 4110 secs

6 to 10 years = 69.9% in 3489 secs

11 to 15 years = 67.1% in 3892 secs

On the job learning = 69.1% in 4287 secs

Formal learning = 64.3% in 3956 secs

The average Revit Structure self-rating score for Structural Engineers is: 3.7.

AutoCAD

Avg score/time for:

Regular users = 56.7% in 4756 secs

Part-time users = 54.9% in 5064 secs

1 to 5 years = 51.3% in 4536 secs

6 to 10 years = 56.3% in 4201 secs

11 to 15 years = 56.7% in 5337 secs

On the job learning = 54.8% in 4466 secs

Formal learning = 52.1% in 4215 secs

The average AutoCAD self-rating score for Structural Engineers is: 3.3.

Trends:

Overall, regular users are more accurate and faster than part-time users.

Structural Engineers with 6 to 10 years' of experience with these tools are the most efficient. They reach a plateau in performance after c. 7 years and make no further productivity gains.

Structural Engineers who learn to apply these tools on projects are more accurate than those who have primarily formal training, but less practical experience. Formally trained users are faster overall than self-taught users.

Structural Engineers are generally confident in their ability to use technical software tools on Construction projects, although overall, there is still room for further improvement in their core skills.

Structural Engineers are more confident and more accurate using modeling software than drafting software.

R

Revit Structure Test Scores Around The World

Autodesk Revit is building information modeling software for architects, structural engineers, MEP engineers, designers and contractors. It allows users to design a building and structure and its components in 3D, annotate the model with 2D drafting elements, and access building information from the building model's database. Revit is 4D BIM capable with tools to plan and track various stages in the building's lifecycle, from concept to construction and later demolition.

Over the years, KnowledgeSmart has captured test scores for thousands of Revit Structure users, in multiple countries across the globe.

Here are the variances in our overall benchmark data, from 2012 to 2015:

2012
Average Score: 62.0%
Average Time: 1 hr 6 mins

2013
Average Score: 63.0%
Average Time: 1 hr 4 mins

2014
Average Score: 64.0%
Average Time: 1 hr 2 mins

2015
Average Score: 67.1%
Average Time: 1 hr 12 mins


Let's take a look at some overall aggregate score and time data on a country vs country basis. Numbers of test scores vary, so this list does not imply that one country is 'better' than another. It's interesting to see what kind of data is generated in different parts of the globe.

Revit Structure Fundamentals

Country	Score (%)	Time (secs)
Australia	68.7	4191.2
Canada	72.4	3886.3
Denmark	70.0	4375.0
Hong Kong	72.2	5414.2
India	62.4	4231.5
New Zealand	67.4	3891.7
Pakistan	64.4	3725.0
Philippines	67.5	3587.5
Poland	63.7	3537.5
Qatar	57.7	3521.5
Saudi Arabia	67.5	3927.5
Singapore	60.8	3775.6
South Africa	63.7	3582.5
UAE	63.0	5095.9
UK	71.1	4887.5
USA	73.7	4218.2

We also capture data on a per State basis, for our US customers. The list below represents those States where the volume of data makes for meaningful data sharing.

State	Score (%)	Time (secs)
USA all	73.7	4218.2
California	80.3	3531.0
Colorado	74.0	4210.0
Florida	78.3	4033.8
Georgia	69.8	4323.0
Illinois	80.5	5365.0
Missouri	81.7	4093.9
New York	73.0	5067.5
Oregon	75.4	5197.0
Tennessee	67.5	3327.5
Texas	68.3	3857.5
Virginia	63.5	3595.0

Revit Structure Advanced

2015
Average Score: 57.2%
Average Time: 1 hr 4 mins

Country	Score (%)	Time (secs)
Australia	61.5	4465.3
Canada	62.4	4665.6
Egypt	62.3	3563.3
Hong Kong	54.3	4433.6
India	51.4	3738.8
New Zealand	59.5	3700.9
Qatar	52.5	4800.8
South Africa	53.0	1950.0
UAE	54.8	4182.5
UK	63.7	3905.7
USA	64.6	4598.1

State	Score (%)	Time (secs)
USA all	64.6	4598.1
California	66.4	5435.0
Colorado	63.3	4365.7
Florida	63.0	4542.0
Georgia	66.0	4595.1
Illinois	63.4	4365.6
Pennsylvania	68.3	4623.4
Tennessee	63.6	4242.5
Texas	62.5	4607.5

Revit Structure is a Top 5 KS test topic, across our global customers. The impact of BIM on the AEC industry is significant and Revit is the dominant design tool, in terms of overall market share and breadth of supply chain resources.

R