Taking a KS Assessment

It's extremely important for AEC firms to explain to their users precisely why they are being asked to take a technical software skills assessment.  The best firms always handle this well.  As with all things in life, a little bit of good communication goes a long way.

This is not all about who is the smartest, fastest, or best Revit user.  Or the most experienced MicroStation wizard.  Although these are useful things to know.  The real value in the KS tools is trying to help users, at an individual level, identify what they know and also what they need to brush up on next, in order to improve their skills, with a particular piece of software.

Over the past 8 years, we have facilitated over 20,000 individual test sessions, across hundreds of AEC firms.  Experience has shown that, in general, people don't like the idea of taking a test - at least that's usually how they feel at first.  However, experience has also shown that, for most people, when they complete their assessment, they realise that it wasn't half as bad as they thought it would be! Further, they often learn new skills as they go through the process of answering the test questions.  And lastly, they find the test feedback and coaching notes extremely helpful, in giving them an unbiased appraisal of their skills using a particular software application - relative to their peers in industry.  It's important that users have the opportunity to review their test scores and feedback with their team leader or BIM manager.  And it's also important that firms explain to their users that all test scores are confidential.  Best practice would recommend that only senior administrators should have access to firm-wide results data.

Provided firms explain to their teams that the main point of rolling out an assessment program, is to gather valuable training needs analysis data, rather than a big-brother style exercise in peeking over shoulders, then most users feel OK about the idea.  And when firms follow through with more focused, modular training workshops, which target recognised skills gaps, then it's a win-win for users and firms alike.

So let's take a look at the user journey, when asked to complete a KnowledgeSmart skills assessment.

We'll assume the firm has already done the pre-assessment prep - and explained to their teams what is happening and why they are being asked to take an assessment.

The next step is for a user to receive their test invite.  (Here is a separate article on how a KS admin sends out test invites: http://the-knowledgesmart-blog.blogspot.com/2011/04/inviting-users-to-take-ks-test.html).

Admins can set up test sessions straight from a browser, but most invites are sent by email.  Either way, the user ends up in the same place.  Here is a typical KS test invite:

The system assigns a username and password to each user.  They access their assessment by clicking the unique URL at the bottom of their invite mail.  This takes them to the assessment login page, where they will be prompted to enter their username and password.  (If a user loses or forgets their password, they can hit the 'Forgot details?' link to receive a new one).

When they log in, they arrive at the main assessment landing page.  Here they will see a summary of the test they are scheduled to take and any special instructions from their administrator.

There are 5 types of questions, which typically comprise a KS assessment.  Let's take a brief look at each one..

Free Text 
This type of question requires a specific answer, which needs to match the answer stored in the KS database.  Look out for precise instructions on how to format your answer, for example, number of decimal places, case sensitive answers, accurate spelling, and so on.

Multiple Choice
Here, you will be offered a selection of possible answers - and you need to select the best one.  Watch out for 'distractor' entries, which are designed to make you think before selecting your answer.

Pick List
This question type offers you a range of options, which could represent a correct answer.  Choose the best answers from the list provided, using the check boxes, to the left of the answers.  Partial marks are often awarded for entering some of the correct answers on pick list questions.

Order List
This question type requires you to place a number of items in the correct sequence.  Drag and drop the answer options, until they are in the right order.

True or False
This question type is pretty straightforward!  Read the statement or question and decide if you think it is true or false.  Enter your answer using the radio buttons.

OK, so when the user arrives at the landing page, before they begin, they need to download the sample data files that they will be using during their assessment.  There are a couple of ways they might have to do this.  The usual way is to select the orange, 'Download files' button.

This prompts the user to save the files locally on their work station.  They will need to navigate back to this folder during their assessment, so most users choose their desktop, or My Documents, etc.  The data sets do not need to be saved during a test session, so they can simply be deleted after the assessment has been completed.

Sometimes firms choose to zip up all sample test files and host them locally, on their own servers.  If this is the case, then the orange button will not display.  Instead, the admin will provide special instructions on the assessment landing page, explaining which file path to follow, in order to find the sample data sets.

Either method is fine - and the user will achieve the same result!  They simply need to know which folder to navigate to, in order to access the files for their randomly generated set of test questions.

When they have their sample test files and they have read any other instructions from their administrator, they can hit the green 'Start' button and their assessment will begin.

Each assessment is timed, although the time taken to finish does not impact the overall score. The test user interface (UI) is easy to navigate.  All test sessions start on question 1.  There is also a small timer, which allows users to keep track of how long they are taking during their session.

Each question has a name and brief summary, which explains to users the main features of the software being addressed.

If a question is task-based, i.e. the user has to use the software in order to answer the question, then the corresponding sample data set can be found in the zip folder they downloaded at the start.  The orange instruction box explains where to find the files for each question.

Users can answer the test questions in any order.  The 'Question Navigator' allows users to move around the questions and generally familiarise themselves with the exercises.  The simple colour code helps users to keep track of their progress.  Green squares indicate questions for which an answer has been submitted.  Orange represents the current question.  Red squares are questions that have yet to be attempted.

If a user makes a mistake, they can re-visit the green squares at any time, to edit their answers.  A test session finishes when all the questions have been answered.  Users have one last opportunity to review and edit their answers, before hitting the green 'Finish' button, which triggers their assessment to be graded.

(Sometimes, an admin will set a time limit for an assessment, in which case the session will automatically close after that time has elapsed.  Answers submitted will be marked and any questions which have not been attempted will be scored as zero).

When the session is finished, the user may receive a test report, with feedback on any dropped marks.  The level of feedback received can vary, depending on the settings prescribed by the account admin.

A helpful summary of the test questions, feedback on incorrect answers, coaching notes and training tags may be viewed by the user, by selecting the 'Show question detail' link at the top of their summary report.

The training tags for questions where score = < 100%,  form a personal curriculum of suggested training workshop topics.

R