8 Steps to More Effective Training Needs Analysis

OK, so you've captured a range of test scores in your firm. What next?  The KnowledgeSmart dashboard offers you a number of options for reporting on your firm's results and identifying skills gaps across your teams.

Here are some steps you can take to interpret your test data..

Step one
Use the searching tools in the 'Results' page of your dashboard, to filter your raw test data.  You have a range of search parameters available.  For example, min score, max score, elapsed time, user name, test name & ID, date ranges, training meta tags, plus 5 extra datafields for categories of your own choosing.

You can use the 'List all training tags' tool, in the 'Your Tests' page of your dashboard, to export a master list of training workshop topics, for each test in your library.  You can then search on these keywords, to identify users who require training on each category.

Step two
When you have run your search, create a new sub-group of your data, using the grouping tools.  For example, all users who scored 60% or less on basic AutoCAD; all users who took longer than 90 mins to complete a basic MicroStation test; all users who require training on worksets in Revit Architecture; all users in your Kansas City office who scored 80%+ on Adobe Photoshop, and so on.

Step three
Use the charting tools to create management reports for each group and highlight priority training issues.  The Training Requirements chart gives you an easy at-a-glance way of identifying the most urgent training topics, for a given group of users.

The Question Performance chart helps you to monitor question scores for each test in your library and further recognise any potential problem areas.  Use the export to PDF tools to create hard copies of your charts.

Step four
Use the Group Comparison charts to analyze and compare results across a range of search results.  For example, the same group of users, pre-and-post training (to measure performance improvement); multi-office results comparison; interview scores vs employee scores, and so on.

The Global Comparison chart offers you an external benchmark, for our most popular test titles.

Step five
You also have the option to export your results data to csv.  When you export results, you will see on the right hand side of the sheet, a field called, 'Training Tags'.  This is a personal summary of training topics for each user, representing keywords relating to questions where, for whatever reason, they didn't score 100% on the question.

Step six
It's a good idea to schedule a brief one-to-one sit down with your users, to discuss their test results and training recommendations summary.  Disregard tags relating to questions where they made a silly mistake, or misread the instructions, etc.  What you are left with is a list of training workshop topics that you - and they - both agree represent genuine skills gaps for a given software topic.

Step seven
Follow up with modular training workshops, which specifically target the highlighted skills gaps. If possible, try to group people together according to experience and ability.  For example, avoid training novice users and expert users in the same session.  You can use the KS meta tags to link to corresponding modular learning content from third parties (i.e. Evolve Consultancy, White Frog, 4D, Bentley Learn, ATC's, etc.) and your own in-house training collateral.

Step eight
Identify your most experienced users for the most significant software titles used by your firm.  They will typically be the people appearing in Quartile 1 or Quartile 2 on your Performance Spread charts, or the people featuring in the top third of your Group Scores charts.

Challenge your leaders to help create bite-sized, modular training material, which you can then use to share knowledge and best practice across the firm.  This is an effective way to turn 'tacit' knowledge (i.e. the information locked away in people's heads), into 'explicit' knowledge (i.e. documented, searchable and accessible to your organisation). Tag your learning material, so it can be searched and easily found via your firm's intranet, LMS, or similar.

Using the KS tools in this way, you can identify both problem areas and examples of excellence and best practice, across your firm.  And target the main productivity issues with focused, relevant training and learning resources.

R

KS Test Library Update

It's fair to say that the KS test library is a work in progress.  It has also been mentioned (on more than one occasion!) that creating up to date test material for the plethora of design, engineering, modelling, civil, structural, building services and analysis tools used by AEC firms - is a job for life! :)

However, we like a challenge, so ever onwards do we march with our library roadmap!

Progress to date has been very good, with almost 30 test titles currently available to AEC firms, across a range of leading vendors.

Here's an up to date list, as of the end of March:

• 3ds Max – modules 1-4
• Adobe InDesign for occasional users
• Adobe Photoshop for occasional users
• AEC Layers Standards – Autodesk
• AutoCAD 2D fundamentals
• AutoCAD 2D – Xpress
• AutoCAD 2D for occasional users
• AutoCAD Civil 3D (part 1)
• AutoCAD Civil 3D (part 2)
• Design Review fundamentals
• Revit Architecture fundamentals
• Revit Architecture – Xpress
• Revit MEP fundamentals (Mechanical)
• Revit MEP fundamentals (Plumbing)
• Revit Structure fundamentals
• Revit Structure – Xpress
• AEC Levels Standards – Bentley
• Bentley Architecture fundamentals
• Bentley GEOPAK (Drainage)
• Bentley InRoads (Roadway Design)
• Bentley View fundamentals
• MicroStation 2D fundamentals
• MicroStation 2D – Xpress
• MicroStation 2D for occasional users
• MicroStation 3D fundamentals
• Rhino fundamentals

We also have a pretty hectic 'work in progress' list, with a variety of new modules at various stages of completion.

Here's what's on the drawing board at the moment:

• Revit Families 
• Revit Project Process 
• USACE ‘Attachment F’ 
• Ecotect 
• Navisworks Manage 
• Revit Architecture 2012 (additional questions) 
• Revit MEP fundamentals (Electrical) 
• AutoCAD Civil 3D (part 3)
• Revit MEP 2012 (additional questions) 
• Google SketchUp 
• AutoCAD Architecture
• Revit Structure 2012 (additional questions)
• Bentley ProjectWise
• Bentley Navigator
• MicroStation V8i (Select Series 1-3)
• IES Virtual Environment
• Tekla Structures
• Bricscad
• Codebook

So by the time we get through that little lot, we'll have close to 50 titles in the KS library.  Well, that's our goal for the end of 2011, anyway!

In addition, we're seeing quite a bit of activity in customer dashboards, with some high quality custom test material being authored in many AEC firms.

So, that's a quick summary of where things currently stand with our test library.  As always, we welcome your comments, feedback and requests for new material.  If it isn't on the list, please let us know and we'll make sure it gets added to the workflow.

R

Customising your Test Invite Text

KS admins have two main options for adding custom text to their test invites.

Option one
The default invite email text appears in the 'Invites' page of the KS dashboard.

From this page, a KS admin can overwrite the default text, in the 'Email subject line' field and in the 'Your message' field, on a per invite basis.  You have a maximum of 800 characters for your test invite text.

Option two
Alternatively, KS admins can write their own custom invite text, which overwrites the KS default text.  To do this, go to Settings > Test Invite.

Here, a KS admin can enter their custom invite text and email subject header.  You have a maximum of 800 characters for your test invite text.  Click 'Save' to register the changes.  This custom text will now replace the KS default message on all test invites sent from the account.

The custom invite text appears on the main test invite page.

To revert to the default KS invite text, at any time, a KS admin simply has to delete the custom text from the Settings > Test Invite page of the dashboard.

R

Building your List of Users

KS admins can upload a list of users into their dashboard and capture additional data against each user record, which can later be used for searching and grouping test results.  Here's how the process works:

Step one
Log in to your admin dashboard and select 'Users' from your navigation tools.

Step two
Click on the 'Upload User Data' link and take a look in the 'Help' notes box on the right hand side.  Tucked away in paragraph 2 is a link to a sample csv file, which you can use to build a list of your users.

Step three
Copy your user info from Excel, into your csv file.  You need the following fields; Email, First name, Last name.  You can also capture user status and 5 additional data fields for information of your own choosing.  For example, employee ID, project team name, office location, job title, industry discipline, etc.  Think of information you might like to use, when searching and filtering your results data later on.

Step four
Browse for your completed csv file and use the 'Upload' tool to add your user list to your KS dashboard.

Step five
Click on the magnifying glass icon, next to the user name (far right of the row), to add to or edit the 5 user datafields.

That's it - you've built your KS user list!  If you subsequently upload a new csv, with additional user records, then the new list will merge with the existing user list.

Searching your user datafields
You can use the information captured in these 5 datafields, to filter your results data in the 'Results' area of your dashboard.  Select the 'Show search' link in the orange bar, at the top of the Results page.  You'll see the five datafields on the right of the search box.

Enter the information in the relevant datafield to filter your data, then select the 'Show groups' link in the orange bar, to create new sub-groups, based on the content of your user datafields.

Batch inviting
Once you have built your user list, you can invite groups of people at the same time, to take a test (provided they are all taking the same test!).  Check the boxes next to the names of your invitees and hit the 'Invite selected users' button at the bottom of the page.

This takes you to the test invite page and pre-populates your user fields.  It is important to remember to select the test you want from the library dropdown at the top.  The default list displays A-Z, with 3ds Max at the top. (If you forget to change the test title, before sending your invites, you may have some confused users asking why you're asking then to take a 3ds Max test!  Unless, of course, that's the one you wanted! :) ).

When you're happy with your invite text, just hit the 'Send Email' button to send your batch invites.

R

Test Feedback Options

KS admins can set different levels of feedback for users at the end of a test session.  It is also possible to select different levels of feedback for interview candidates and in-house users.  Sometimes firms don't want interview candidates to receive a test report immediately after their test session.  So now you have the choice of when to provide user feedback - and just how much information to share.

It's a straightforward process to choose your feedback settings:

Log in to your KS dashboard and click Settings > Test Feedback.

There are 4 choices:

1) No report
2) Basic report
3) Feedback on report and
4) Feedback & coaching on report.


The default setting is as follows:

Interview candidates - basic report

In-house users - full report with feedback and coaching notes

When you've made your selection, hit 'Save' to register your changes.  This setting applies to all tests taken on your account.  If you want to adjust the amount of feedback for different groups, just amend the setting in your dashboard, before you send out the test invites.

R

Branding your Test UI and Reports

We recently added a light touch of corporate branding for KS customers.

It is now possible to replace the KS logo on all user-facing test UI's and reports, with your own company logo.  We recommend that you do this, as it makes the whole experience for users feel more like an in-house training needs analysis exercise, rather than a scary third party skills test!

It's a really simple process..

Step one
Choose a copy of your company logo.  Keep the file size sensible, as it is designed to fit in a comparatively small space, at the top of your admin dashboard, test UI and test reports.  About this size should do just fine.

Step two
Log in to your admin dashboard and select Settings > Branding from your navigation tools.

Step three
Use the browse and upload tools to select your logo and upload a copy to the KS dashboard.

Step 4
Check the boxes to display your logo in 3 places; your KS dashboard, your test UI and your test reports.

Click the 'Preview' buttons, to view how your logo will display in the test UI and test report.  You'll probably note that we've removed the KS orange colouring from the test reports, replacing it with a more neutral light grey - to avoid any unfortunate clashes with your own colours!

Sample Test UI

Sample Test Report

We're gradually dialling back on the KS branding, in favour of your own corporate colours.  In future, you'll see a smaller, 'Powered by KnowledgeSmart' logo, appearing at the footer of most of our collateral.

R

Setting up Test Sessions from a Browser

It's not always possible for users to access a test invite via email.  For example interview candidates, or in-house candidates who are using a training room, where access to their email account is not available.

For this reason, we have introduced a new feature, which allows KS administrators to set up a new test session straight from a browser.

Here's how it works:

Step one
Admin selects the link called, 'Admin Test Setup Login', on the dashboard landing page.

Step two
Admin enters username and password to access their account.

Step three
Admin selects account from the dropdown (for firms with multiple accounts) and selects the required test title from their library dropdown.  This list includes KnowledgeSmart OTS (off the shelf) tests and self-authored tests, displaying in A-Z order.  Test ID numbers are also included, so admins can easily identify which test they want from their library.

Step four
This is where the candidate takes over.  They enter their name and email address, select their status (employee or interview candidate) and proceed to the test landing page.

Step five
This is the test landing page.  This is the same page as user's see when they click on the test URL in their email invites.  From here, the process for taking a test is the same, irrespective of how they were invited.  When they have read their test instructions and downloaded their sample data files, they hit the green 'Start' button to begin their test session.

R