I recently read an articulate and informed blog post by Carillion's Technical BIM Lead, John Ford, about the business of measuring BIM skills.
Read the full post here: 'Are we Underestimating the Value of BIM Assessments?'
John makes a number of perceptive points about the organisational and process challenges facing AEC businesses, on their journey to BIM excellence.
There is no 'right or wrong' way to capture useful business intelligence, if it feeds into a wider BIM improvement plan. As Ford points out in his article, there are many effective methods for analysing data and metrics around the topic of BIM experience and capability, both within an organisation and across the wider industry supply chain.
A mix of formal assessment, self-assessment, survey responses, RAG tables and industry certification offer AEC firms all the tools they need, to create a clearer picture of BIM experience and credentials.
KnowledgeSmart has been working consistently in this space for the past 5 years, creating software and library content designed to help AEC businesses measure BIM maturity, skills gaps, training needs and technical expertise.
Here is a link to a recent KS webcast, which provides all the necessary tools and benchmark data for Global AEC businesses to create their own tailored BIM excellence plan.
To slightly misquote management guru, Peter Drucker, 'If you can't measure it, you can't manage it'. Wise words.
R
Showing posts with label AEC. Show all posts
Showing posts with label AEC. Show all posts
Tuesday, 23 January 2018
Wednesday, 25 October 2017
KS Webinar - Demonstrating Individual and Team BIM Level 2 Capability
Earlier today, we recorded our first Webinar to support the upcoming launch of our new KS library material: The presentation is titled, "Demonstrating Individual and Team BIM Level 2 Capability".
The session runs for 45 mins, plus 15 mins delegate Q&A at the end. Here is the agenda:
The session was co-presented by KnowledgeSmart CEO, Rory Vance, and Evolve Consultancy Principal, Nigel Davies.
The session runs for 45 mins, plus 15 mins delegate Q&A at the end. Here is the agenda:
The content is aimed at AEC business leadership, BIM professionals and technical management. Here is a link to a recording of the session.
R
Friday, 21 April 2017
How To Explain Skills Assessments To Your Teams
A common question asked by KS customers is how to present the topic of skills testing to your teams.
Here is an example of a recent customer mail:
"Do you have experience with encouraging employees to take the assessments? We want to encourage staff without being off-putting or making it seem like the point of the exercise is to determine their tenure within the organization.
What do you think is the best approach to engage our staff? Do you give it a different name besides assessment?
Your assistance is greatly appreciated."
Here is our reply...
The key to your communication is to emphasize the 'What's in it for me?' aspect of the assessments. No-one likes to take tests. As human beings, we'll avoid it where at all possible! But, conversely, people are curious as to how well they are doing. And they want to improve. Also, they don't know what they don't know. Or how they measure up to their peers and wider industry. So you need to focus your message on the training needs side of the learning path. You can't take them on a focused training journey, if you don't know their starting off point. And you can't assume everyone knows the basics, therefore deserves advanced training. They need to prove it.
When you have a clear firm benchmark, you can also hire more confidently and intelligently. Which is good news for incoming staff and existing teams.
So don't make the assessments optional. This needs to be managed and mandatory, but with a light touch. And the results need to be analysed and fed into a relevant training plan.
Also, people have a natural fear of failure. So make it clear that nothing bad will happen if they screw up. They don't need to feel apprehensive or nervous. This is about positive improvement, not picking on people who happen to have gaps in their knowledge.
And make it clear that the results are confidential. Only to be accessed by people with a position of responsibility for leadership and staff development.
Here is a useful User FAQ guide, which will help you shape your communications in this space:
WHAT IS KNOWLEDGESMART?
KnowledgeSmart is a web-based skills assessment tool, which enables architectural and engineering practices to measure the software skills of their teams. Since 2003, by carrying out in excess of 30,000 individual assessments globally, a helpful independent benchmark has been established for basic proficiency, across a range of subjects.
Here is a microsite which provides helpful industry benchmark data for our Top 5 skills assessment topics: http://www.2dim4bim.com.
The KnowledgeSmart tools help to identify skills associated with design and engineering software applications, such as AutoCAD, Civil 3D, Plant 3D, Revit, Inventor, MicroStation, ProjectWise, 3ds Max, ArchiCAD, Primavera P6, Adobe Photoshop, Adobe InDesign, SketchUp and Rhino. The process aims to assess current performance in order to identify areas where individuals can benefit from additional training.
WHAT IS THE POINT OF KNOWLEDGESMART AND WHY SHOULD I DO IT?
This process is not about a big-brother style test to check up on unsuspecting users. Nor is it a matter of pointing out who is the more knowledgeable software user. Rather, the point of the exercise is to highlight how much each individual currently knows – in order to identify those areas where focused training can improve basic knowledge.
Everyone uses software in a different way. Some people use these tools full time in their role; others may only use the tools from time to time. The objective is not to teach a veteran user how to perform basic tasks. But the fact remains, a more detailed knowledge of basic commands and operations can alleviate apprehension in using technical software applications and assist in speeding up routine tasks.
IS IT THE SAME ASSESSMENT FOR EVERYONE?
There are a number of assessment modules to choose from, created by a team of expert authors. Firms can also create their own material in-house. For Revit skills, as an example, there is a choice of material, ranging from introductory level, all the way up to more advanced concepts. Summary reports take into consideration whether an individual is a full time user, or someone who only uses the software occasionally as part of a wider role. Follow-up comparisons and training recommendations should be reflected accordingly.
WHAT KIND OF TOPICS DOES EACH ASSESSMENT COVER?
KnowledgeSmart assessments typically break down complex technical software applications into a series of stages or modules.
For example, the fundamentals assessments (for AutoCAD & MicroStation) cover a range of basic commands, including; basic object/element creation, layers/levels, blocks/cells, annotation, referencing & printing, UCS/ACS and preferences.
Assessments typically comprise 20-30 questions and present scores as a percentage, based on the accuracy of the answers presented. There are 2 question types and 9 question styles. Question types are either task-based (i.e. you need to perform a task using the software in order to work out the answer) or knowledge-based (i.e. you are being evaluated on your understanding of certain software functions, without having to actually use the tools). Question styles include; Free text, Multiple choice, Pick list, Order list, True or false, Complete the blank, Matching list, Matrix and Essay.
WHAT IF I DO NOT SCORE VERY WELL?
This exercise is not about who is the best. You will identify areas where you can improve your own performance with the help of more targeted training workshops.
Following the assessment everyone receives an overall result and summary report with feedback on dropped marks, training recommendations and coaching notes.
WHEN AND WHERE WILL THE ASSESSMENT TAKE PLACE?
Your administrator will have details of your assessment program. If you are unavailable to make the date or time arranged please contact your administrator to reschedule.
HOW LONG DOES IT TAKE?
There is no limit to how long the exercise will take. The average time is approximately an hour, but this is not a strict rule.
Try to work at your usual speed. Don’t rush, as many simple mistakes are made by people not reading the instructions carefully. Conversely, don’t take too long to complete each task, as your score plus time taken will be looked at in the overall results analysis.
WHEN DO I GET MY RESULT?
You will receive your score straight away when you finish the assessment. You may also be emailed with a link to your assessment report, including feedback on any marks which have been dropped.
ADDITIONAL INFORMATION
The amount of time you take will not affect the accuracy of your score.
Before you start, you may be presented with a zip folder containing files which you’ll need to refer to during your assessment. Save the zip folder locally on your machine and extract the sample data sets from the zip. And remember where you save it – as you will need to navigate back to this folder during your assessment!
Please read the written instructions carefully for each question. This is also an assessment of your ability to follow instructions. Often, marks are dropped through candidates simply not doing what they have been asked to do.
During your assessment session, you can answer the questions in any order, using the KnowledgeSmart question navigator. You can also edit your answers during the session and review your work once more at the end, before you select Finish.
The time taken to complete an assessment varies depending on the individual, but an average is about an hour. You don’t have to complete the assessment in one sitting; you can log out and return later if you prefer. When you resume, your previous answers will have been saved.
R
Here is an example of a recent customer mail:
"Do you have experience with encouraging employees to take the assessments? We want to encourage staff without being off-putting or making it seem like the point of the exercise is to determine their tenure within the organization.
What do you think is the best approach to engage our staff? Do you give it a different name besides assessment?
Your assistance is greatly appreciated."
Here is our reply...
The key to your communication is to emphasize the 'What's in it for me?' aspect of the assessments. No-one likes to take tests. As human beings, we'll avoid it where at all possible! But, conversely, people are curious as to how well they are doing. And they want to improve. Also, they don't know what they don't know. Or how they measure up to their peers and wider industry. So you need to focus your message on the training needs side of the learning path. You can't take them on a focused training journey, if you don't know their starting off point. And you can't assume everyone knows the basics, therefore deserves advanced training. They need to prove it.
When you have a clear firm benchmark, you can also hire more confidently and intelligently. Which is good news for incoming staff and existing teams.
So don't make the assessments optional. This needs to be managed and mandatory, but with a light touch. And the results need to be analysed and fed into a relevant training plan.
Also, people have a natural fear of failure. So make it clear that nothing bad will happen if they screw up. They don't need to feel apprehensive or nervous. This is about positive improvement, not picking on people who happen to have gaps in their knowledge.
And make it clear that the results are confidential. Only to be accessed by people with a position of responsibility for leadership and staff development.
Here is a useful User FAQ guide, which will help you shape your communications in this space:
WHAT IS KNOWLEDGESMART?
KnowledgeSmart is a web-based skills assessment tool, which enables architectural and engineering practices to measure the software skills of their teams. Since 2003, by carrying out in excess of 30,000 individual assessments globally, a helpful independent benchmark has been established for basic proficiency, across a range of subjects.
Here is a microsite which provides helpful industry benchmark data for our Top 5 skills assessment topics: http://www.2dim4bim.com.
The KnowledgeSmart tools help to identify skills associated with design and engineering software applications, such as AutoCAD, Civil 3D, Plant 3D, Revit, Inventor, MicroStation, ProjectWise, 3ds Max, ArchiCAD, Primavera P6, Adobe Photoshop, Adobe InDesign, SketchUp and Rhino. The process aims to assess current performance in order to identify areas where individuals can benefit from additional training.
WHAT IS THE POINT OF KNOWLEDGESMART AND WHY SHOULD I DO IT?
This process is not about a big-brother style test to check up on unsuspecting users. Nor is it a matter of pointing out who is the more knowledgeable software user. Rather, the point of the exercise is to highlight how much each individual currently knows – in order to identify those areas where focused training can improve basic knowledge.
Everyone uses software in a different way. Some people use these tools full time in their role; others may only use the tools from time to time. The objective is not to teach a veteran user how to perform basic tasks. But the fact remains, a more detailed knowledge of basic commands and operations can alleviate apprehension in using technical software applications and assist in speeding up routine tasks.
IS IT THE SAME ASSESSMENT FOR EVERYONE?
There are a number of assessment modules to choose from, created by a team of expert authors. Firms can also create their own material in-house. For Revit skills, as an example, there is a choice of material, ranging from introductory level, all the way up to more advanced concepts. Summary reports take into consideration whether an individual is a full time user, or someone who only uses the software occasionally as part of a wider role. Follow-up comparisons and training recommendations should be reflected accordingly.
WHAT KIND OF TOPICS DOES EACH ASSESSMENT COVER?
KnowledgeSmart assessments typically break down complex technical software applications into a series of stages or modules.
For example, the fundamentals assessments (for AutoCAD & MicroStation) cover a range of basic commands, including; basic object/element creation, layers/levels, blocks/cells, annotation, referencing & printing, UCS/ACS and preferences.
Assessments typically comprise 20-30 questions and present scores as a percentage, based on the accuracy of the answers presented. There are 2 question types and 9 question styles. Question types are either task-based (i.e. you need to perform a task using the software in order to work out the answer) or knowledge-based (i.e. you are being evaluated on your understanding of certain software functions, without having to actually use the tools). Question styles include; Free text, Multiple choice, Pick list, Order list, True or false, Complete the blank, Matching list, Matrix and Essay.
WHAT IF I DO NOT SCORE VERY WELL?
This exercise is not about who is the best. You will identify areas where you can improve your own performance with the help of more targeted training workshops.
Following the assessment everyone receives an overall result and summary report with feedback on dropped marks, training recommendations and coaching notes.
WHEN AND WHERE WILL THE ASSESSMENT TAKE PLACE?
Your administrator will have details of your assessment program. If you are unavailable to make the date or time arranged please contact your administrator to reschedule.
HOW LONG DOES IT TAKE?
There is no limit to how long the exercise will take. The average time is approximately an hour, but this is not a strict rule.
Try to work at your usual speed. Don’t rush, as many simple mistakes are made by people not reading the instructions carefully. Conversely, don’t take too long to complete each task, as your score plus time taken will be looked at in the overall results analysis.
WHEN DO I GET MY RESULT?
You will receive your score straight away when you finish the assessment. You may also be emailed with a link to your assessment report, including feedback on any marks which have been dropped.
ADDITIONAL INFORMATION
The amount of time you take will not affect the accuracy of your score.
Before you start, you may be presented with a zip folder containing files which you’ll need to refer to during your assessment. Save the zip folder locally on your machine and extract the sample data sets from the zip. And remember where you save it – as you will need to navigate back to this folder during your assessment!
Please read the written instructions carefully for each question. This is also an assessment of your ability to follow instructions. Often, marks are dropped through candidates simply not doing what they have been asked to do.
During your assessment session, you can answer the questions in any order, using the KnowledgeSmart question navigator. You can also edit your answers during the session and review your work once more at the end, before you select Finish.
The time taken to complete an assessment varies depending on the individual, but an average is about an hour. You don’t have to complete the assessment in one sitting; you can log out and return later if you prefer. When you resume, your previous answers will have been saved.
R
Saturday, 25 February 2017
KS Wishlist update
We receive a regular flow of new feature ideas from our customers in different parts of the world. The best ideas find their way onto the KS software update wishlist and then, in turn, into the live site.
Here is what we've been up to in recent months...
Update survey Invite > History page with new options for managing invites.
Add a new method to quickly import user list directly onto the invites page (for both tests and surveys). Add new option to revert to default invite text.
R
Here is what we've been up to in recent months...
Update survey Invite > History page with new options for managing invites.
Add a new method to quickly import user list directly onto the invites page (for both tests and surveys). Add new option to revert to default invite text.
You'll find a formatted csv template in the help notes box on the right hand side of the page.
Add more options for managing survey results data.
Tidy up logic on invite re-send tool, where expiry dates have been amended.
Tidy up csv export for dynamic groups in results page.
Tidy up logic to prevent creation of blank groups in users and results pages.
Change presentation of results graphs. Split into two sections: 'Performance' and 'Comparisons'. Add new section to display charts for exam results.
Change results csv export to include exam results, as well as assessment results.
Include option to retrospectively apply exam grade to legacy test results. Swap gap analysis report into exam certificate on user page. Move result from My Assessments to My Exams.
Allow admins to send a link to an exam certificate and/or a gap-analysis report to users who have completed an exam.
Allow admins to choose default landing account when logging in.
Add number of results to Results > Data page. Add user count to footer of training requirements chart.
Coming soon..
Here's a brief summary of what we're working on in the next 6-12 months..
Test UI refresh - new mobile responsive design for the KS test-taking interface
Test report refresh - new design for KS test report
New charting options for the KS Query Tool
Use of CDN for managing test files and downloads
New KS support tool
Tidy up system mails logic (for invite, reminders and results mails)
Bulk upload of questions in library > draft questions area
Option to set an expiry date on the main survey invites page
Reminder mails option for survey invites
Add dynamic groups to survey invite > history
If a group is created in Users (static or dynamic), include option to display it in Results automatically.
Auto reports - Email summary report of KS account activity for admins
Searchable skills matrix
Reporting and charting templates
R
Wednesday, 2 December 2015
Benchmark Stats for Popular KS Test Titles
Benchmarking software capability at individual, team and office level is a key part of the KnowledgeSmart experience.
Our library of skills assessments for the AEC industry covers a variety of popular software tools, used on projects on Construction projects worldwide.
Our results data comes from a wide variety of countries and territories, including: US, UK, Australia, Canada, New Zealand, South Africa, Singapore, Vietnam, India and the Middle East.
Here is the overall average results data for our most popular topics (% score and time to complete):
| Test Title | % Score | Time |
| 3ds Max fundamentals | 54.5 | 57m |
| Adobe InDesign occasional | 64 | 45m |
| Adobe Photoshop occasional | 62 | 48m |
| AutoCAD 2D fundamentals | 58.5 | 1h 13m |
| AutoCAD 2D occasional | 73.1 | 36.5m |
| AutoCAD Civil 3D fundamentals | 67.2 | 1h 3m |
| AutoCAD Civil 3D advanced | 52.2 | 45.5m |
| AutoCAD Plant 3D fundamentals | 63 | 53m |
| Bentley ABD fundamentals | 51.5 | 1h 28m |
| Bentley ProjectWise fundamentals | 59 | 1h 3m |
| BIM Management | 68.5 | 46m |
| InfraWorks 360 fundamentals | 58.5 | 1h 2m |
| Inventor fundamentals | 64 | 1h 15m |
| MicroStation 2D fundamentals | 59.3 | 1h 24m |
| MicroStation 3D fundamentals | 52.5 | 58m |
| Navisworks fundamentals | 58.4 | 43m |
| Revit Architecture fundamentals | 68.5 | 1h 8m |
| Revit Architecture advanced | 57.1 | 1h 24m |
| Revit Content Creation | 57.8 | 1h 3m |
| Revit MEP fundamentals - Electrical | 63.7 | 1h 6m |
| Revit MEP fundamentals - Mechanical | 62.2 | 1h 7m |
| Revit MEP fundamentals - Plumbing | 70.1 | 1h 11m |
| Revit MEP advanced | 53.3 | 1h 2m |
| Revit Project Process | 56.4 | 46.5m |
| Revit Structure fundamentals | 67.1 | 1h 12m |
| Revit Structure advanced | 57.2 | 1h 4m |
| Rhino | 61.2 | 44.5m |
| SketchUp | 53.4 | 50m |
KnowledgeSmart independent skills testing enables businesses to compare performance and capability against industry standards and benchmark their organisation against peers, competitors and partners. Measuring and demonstrating technical software experience and BIM capability is a key differentiator for AEC firms when bidding for new projects.
R
Thursday, 19 December 2013
Results Analysis - A Detailed Overview
The main point of KnowledgeSmart is to provide AEC firms with detailed skills gap and training needs analysis data. So let's take a proper look at the process.
To begin, we will assume that you have already gone through the process of adding users, inviting users, assessing your teams and you now have a healthy selection of results in your KS dashboard.
Don't forget, you control the amount of feedback that users get to see at the end of their test session, by selecting your preferred option on the Settings > Test Feedback page.
And you can also choose to display (or hide) the 'Skip Question' (I don't know the answer) button, the 'Request Training' (I'm not sure about my answer) button and the test logout button, via the Settings > Test UI Options page. This will have an impact on the types of results data you are able to capture and analyse later on.
For the purposes of this exercise, we're using dummy results data, to protect the (incompetent?) innocent!
When your teams have completed their assessments, you can view your KS results in the Results > Data area of your dashboard.
Use the 'Results per page' drop-down to change the number of results displayed on screen.
Use the account drop-down menu to view results data across linked accounts.
Click on the score link to view a detailed summary of each test report. The report opens up in a new window.
You can display links to in-house or third party learning resources in the test report, which builds a bridge between the testing and training parts of your learning journey.
You have a range of searching & grouping tools available, for filtering and analysing your KS results in detail.
Step one
Select the 'Show search' link in the orange bar, at the top of the Results page.
There are a variety of different search parameters. You'll also see the five user datafields on the right hand side of the search box (you can add your own custom labels on the Settings > User Datafields page). Use these values to add more depth to your results searching & filtering options.
Step two
Enter the information in the relevant search field and hit 'Search' to filter your results data. Use the check boxes to select your results records for the new group.
Selecting the 'Training requests' filter will display a list of all users who have selected the 'Request training' button during their test session.
Click on each report to see which questions have been flagged.
Step three
Select the 'Show groups' link in the orange bar.
Step four
Use the 'Create New Group' tool to create a new sub-group of your results, based on the output of your search. Once you have selected your results, type in the name of the new group and hit 'Create'.
Step five
Use the 'View group' drop-down menu to navigate between sub-groups of your results data.
Dynamic Groups
Use the dynamic grouping tool to create new groups which will automatically update, when new test results are added to your dashboard.
Step one
Select the 'Show dynamic groups' link in the orange bar.
Step two
Use the 'New Group' tool to create a new dynamic grouping of your results. Enter the group name in the field provided.
Step three
Use the 5 datafield drop-downs to create the rules you require, for adding existing and future results to your group. Save your new dynamic group, by selecting Save Group.
For example, a results group has been created based on datafield 1, City: London. The next time a new result is added to the database and the associated user record includes datafield 1, City: London, then the results record will automatically be added to the existing group.
Use the 'View dynamic group' drop-down menu to navigate between sub-groups of your results data.
Exporting Results
You have a variety of options for exporting your KS results data.
Select the 'Export results to csv' button to generate a spreadsheet of overall results data, including a personal curriculum of training topics for each user.
Select the 'Export training info to csv' button to generate a spreadsheet of training workshops, together with a corresponding list of users who have been identified as potentially needing to attend each workshop.
Select the 'Export training requests to csv' button to generate a spreadsheet of users who have flagged questions for further training. The report lists the highlighted questions and corresponding training tags.
Select the 'Export skipped questions to csv' button to generate a spreadsheet of users who hit the 'Skip question' button during their test session. The report lists the skipped questions and corresponding training tags.
Charting & Reporting KS Results
There are a range of charting and management reporting options available in your KS dashboard.
Performance Scatter
This chart displays group results in performance quartiles. The upper left quadrant (Q1) contains results, where the users completed their assessment accurately and fast.
The upper right quadrant (Q2) shows results which have high accuracy, but slower completion times. Bottom left (Q3) represents lower scores, but in a fast time.
Lastly, the bottom right quartile (Q4) shows test scores which are inaccurate and slow.
Training Requirements
This chart highlights training topics for a given group of test results. The logic analyzes all of the results for a group, references the training tags assigned to questions presented during a test and lists those tags in priority order.
Red indicates the tasks which have been answered incorrectly by most people in a given group. Orange is the next highest priority, followed by Yellow; green training tags are the topics which have been answered correctly by most of the group, so represent the least urgent issues.
For example; 10 people are presented with one or more questions, which include the tag, ‘Rooms’. If 7 of the 10 people answer one or more ‘Rooms’ questions incorrectly, then the keyword, ‘Rooms’, will be flagged at 70% and appear in Red on the chart.
You can also view charts for Training Requests and Skipped Questions on this page.
Question Performance
This chart looks at how each individual question in your library has performed, in any given test.
The logic analyses all of the results for each test and presents an aggregate percentage score for each question, divided by the total number of results for that test on the account.
For example; 10 people answer a question called ‘Doors’. If 7 of the 10 people answer the question 100% correctly, 1 person scores 50% and the other 2 score 0%, then the question will score an aggregate of 75% and appear in Yellow on the chart.
You can also create a chart for the average time taken to answer each question, by changing the radio button at the foot of the chart.
+chart.JPG)
You can quickly view the relevant question data by clicking on each question name in the 'Question List' box, on the right hand side of this page.
Select the 'Export question performance info to csv' button to generate a spreadsheet which presents a breakdown of per question time & score values for each user.
Select the 'Display question answer info' button to view a table of 'required' vs 'submitted' answer values for each user. This is an easy way to identify an common mistakes or errors across multiple results.
We will shortly be adding a new page called 'Module Performance', which displays a per module breakdown of KS results data.
Group Scores
This chart displays user performance for any given group, in descending order. The X-axis shows the % score attained and the Y-axis displays user names.
Group Comparisons
This chart allows firms to compare performance, from one group to another, across (up to) 9 sets of data at a time. Group vs group comparisons can be used to compare a range of results data.
For example; pre and post-training performance, different project teams, offices from different geographic locations, data representing different job titles or industry disciplines, in-house data vs interview candidates, and so on.
Global Comparisons
This chart allows firms to compare in-house performance against a wider set of anonymous industry aggregate results data, for selected tests.
We will be adding new searching and reporting options in a future release, so you can analyse the user background info and demographics in greater detail. Regional benchmark stats and user profiles are a key theme for the global comparisons section.
So, there you have it, a comprehensive strategy for capturing, analysing and reporting on skills gap and training needs data for your organisation.
R
Subscribe to:
Posts (Atom)
Posts
