KS Help Notes_Results 02_Charting Results

CHARTING & REPORTING KS RESULTS DATA 

There are a range of charting and management reporting options available in your KS dashboard.

Search & group results

The first task is to filter the raw results data into more manageable blocks. There are a range of searching and grouping options available, including; score ranges, date ranges, elapsed time, test topic, user names, training keywords, plus 5 additional custom datafields.

Performance Scatter

This chart displays group results in performance quartiles.  The upper left quadrant (Q1) contains results, where the users completed their assessment accurately and fast.
The upper right quadrant (Q2) shows results which have high accuracy, but slower completion times.  Bottom left (Q3) represents lower scores, but in a fast time.
Lastly, the bottom right quartile (Q4) shows test scores which are inaccurate and slow.

Training Requirements

This chart highlights training topics for a given group of test results. The logic analyzes all of the results for a group, references the training tags assigned to questions presented during a test and lists those tags in priority order.
Red indicates the tasks which have been answered incorrectly by most people in a given group. Orange is the next highest priority, followed by Yellow; green training tags are the topics which have been answered correctly by most of the group, so represent the least urgent issues.

For example; 10 people are presented with one or more questions, which include the tag, ‘Lines’. If 7 of the 10 people answer one or more ‘Lines’ questions incorrectly, then the keyword, ‘Lines’, will be flagged at 70% and appear in Red on the chart.

Question Performance

This chart looks at how each individual question in a library has performed, in any given test.
The logic analyzes all of the results for each test and presents an aggregate percentage score and time for each question, divided by the total number of results for that test on the account.

For example; 10 people answer a question called ‘Lines’. If 7 of the 10 people answer the question 100% correctly, 1 person scores 50% and the other 2 score 0%, then the question will score an aggregate of 75% and appear in Yellow on the chart.

Group Scores

This chart displays user performance for any given group, in descending order. The X-axis shows the % score attained and the Y-axis displays user names.

Group Comparisons

This chart allows firms to compare performance, from one group to another, across (up to) 9 sets of data at a time.  Group vs group comparisons can be used to compare a range of results data.

For example; pre and post-training performance, different project teams, offices from different geographic locations, data representing different job titles or industry disciplines, in-house data vs interview candidates, and so on.

Global Comparisons

This chart allows AEC firms to compare in-house performance against a wider set of anonymous industry aggregate results data, for selected tests.

Results are currently available for: AutoCAD 2D, MicroStation 2D, Revit Architecture, Revit MEP and Revit Structure.  KS plans to publish additional benchmark figures in future.

Exporting Results

Use the csv export tools, to capture KS results for further analysis in Excel.

When you export results, you will see a field called, Training Tags; a personal summary of recommended training topics for each user.

R

KS Help Notes_Results 01_View, Search & Group Results

VIEWING, SEARCHING & GROUPING YOUR KS RESULTS DATA 

You can view your KS results in the Results > Data area of your dashboard. Use the Results per page dropdown to change the number of results displayed on screen.

Use the drop-down menu to view results data for linked accounts.

Click on the score link to view a detailed summary of each test report. The report opens up in a new window.

Filter your results by searching on a variety of fields, including; user name, score range, time range, date range, test info, module info, question info, user status, training keywords and the information captured in your 5 user datafields.

Step one
Select the Show search link in the orange bar, at the top of the Results page.

You'll see the five user datafields on the right hand side of the search box.

Step two
Enter the information in the relevant search field and hit Search to filter your results data. Use the check boxes to select your results records for the new group.

Step three
Select the Show groups link in the orange bar.

Step four
Use the Create New Group tool to create a new sub-group of your results, based on the output of your search.  Once you have selected your results, type in the name of the new group and hit Create.

Step five
Use the View group dropdown menu to navigate between sub-groups of your results data.

Dynamic Groups

Use the dynamic grouping tool to create new groups which will automatically update, when new test results are added to your dashboard.

Step one
Select the Show dynamic groups link in the orange bar.

Step two
Use the New Group tool to create a new dynamic grouping of your results.  Enter the group name in the field provided.

Step three

Use the 5 datafield dropdowns to create the rules you require, for adding existing and future results to your group.  Save your new dynamic group, by selecting Save Group.

For example, a results group has been created based on datafield 1, City: London.  The next time a new result is added to the database and the associated user record includes datafield 1, City: London, then the results record will automatically be added to the existing group.

Step four

Use the View dynamic group dropdown menu to navigate between sub-groups of your results data.

Exporting Results

Use the csv export tools, to capture KS results for further analysis in Excel.

When you export results, you will see a field called, Training Tags; a personal summary of recommended training topics for each user.

R

KS Help Notes_Invites 03_Set up Test from a Browser

SETTING UP AN ASSESSMENT FROM A BROWSER

It's not always possible for users to access a test invite via email. For example interview candidates, or in-house candidates using a training room, where access to their email account is not available.

For this reason, you will occasionally need to set up a test session, straight from a web browser. Here’s how it works:

Step one
Select the Assessment link, on the dashboard landing page.

Step two
Enter your KS admin User name and Password and hit Log in to access your account.

Step three
Select your account from the dropdown menu (for firms with multiple accounts) and select the required test title from your library list. This list includes KnowledgeSmart OTS (off the shelf) titles and self-authored tests, displaying in A-Z order.  Test ID numbers are also included, so you can easily identify which test you want from your library.

Step four
This is where the candidate takes over. They enter their name and email address, select their status (employee or interview candidate) and proceed to the next page.

Step five (optional)

You have the option to present candidates with an additional information page, where they can submit a brief user profile.

Step six

Here is the test start page. This is the same page users’ see when they click on the test URL in their email invites. From here, the process for taking a test is the same, irrespective of how they were invited.  When they have read the instructions and downloaded their sample data files, they hit the green Start button to begin their test session.

R

KS Help Notes_Invites 02_Inviting Users

INVITING USERS TO TAKE AN ASSESSMENT

Administering the KS tools is a pretty straightforward process. The system is web-based, so there is no software to install, maintain and upgrade. In general, it's a low-impact system, from an IT administration point of view.

Let's take a look at the process for inviting users to take a test. There are three different options, depending on your requirements.  Let's review each one:

Option one – Inviting a small number of users to an assessment

Step one
Select the test you want your user(s) to take, from your library. Each test has a small 'envelope' icon, next to the test name.

Click on the envelope icon, which takes you to the test invite page. Your selected test will automatically appear in the test list dropdown.

Alternatively, you can select the Invites link, from the top navigation bar of your dashboard. (You'll end up in the same place).  However, bear in mind that if you do use the top navigation route (rather than the envelope icon next to a specific test title) then your required test will not automatically get selected in the dropdown menu. (It presents your test library in A-Z order.  You need to select the required test from the dropdown list of titles.

Step two
Enter the email, first name and last name of your test taker(s) in the fields provided.  You can also capture user status, i.e. employee, interviewee, student, etc.  Click Add to confirm your candidate's details.

Step three
Decide on your invite text. You have a couple of different options for customising your invites. You can either add custom text on a per invite basis (by overwriting the default text in the Your message box each time), or alternatively, go to Settings > Test Invite to create your own personalised message.

Step four (optional)
You can set an expiry date on your test invites. After this date has passed, your users will no longer be able to access this test session. If they haven't completed their test before the expiry date, you will need to edit the expiry date or issue a new invite.

Step five
OK, so you have completed the following steps; selected the correct test title from your library; entered all your user data; edited your invite text; maybe set an expiry date. The final step is to hit the Send Invite button.  This step prompts the following routines to take place, in the background: it generates your users' profile(s) on the KS system, adds them to your user list (in the Users page of your dashboard), creates a username/password for each user and sends out their individual test invite emails.

You can view a list of sent invites and completed tests on the Invites > History page.

Option two – Inviting a group of users to an assessment

You have the option to batch invite larger groups of users to take a test (provided they are all taking the same one). Rather than entering their user details one at a time, as outlined in the process above, you can invite them all at once, from the Users section of your dashboard.

Step one 
Once you have built your user list, check the boxes next to the names of your invitees and hit the Invite selected users button at the top of the page.

Step two
This takes you to the test invite page and pre-populates your user fields. It is important to remember to select the test you want from the library dropdown at the top. The default list displays A-Z.  Test ID numbers are also included, so you can easily identify which test you want from your library.

When you're happy with your invite text, hit the Send Invite button to send your invites.

Option three – Setting up an assessment session from a browser

It's not always possible for users to access a test invite via email. For example interview candidates, or in-house candidates using a training room, where access to their email account is not available.

For this reason, you will occasionally need to set up a test session, straight from a web browser. Here’s how it works:

Step one
Select the Assessment link, on the dashboard landing page.

Step two
Enter your KS admin User name and Password and hit Log in to access your account.

Step three
Select your account from the dropdown menu (for firms with multiple accounts) and select the required test title from your library list. This list includes KnowledgeSmart OTS (off the shelf) titles and self-authored tests, displaying in A-Z order.  Test ID numbers are also included, so you can easily identify which test you want from your library.

Step four
This is where the candidate takes over. They enter their name and email address, select their status (employee or interview candidate) and proceed to the next page.

Step five (optional)
You have the option to present candidates with an additional information page, where they can submit a brief user profile.

Step six
Here is the test start page. This is the same page users’ see when they click on the test URL in their email invites. From here, the process for taking a test is the same, irrespective of how they were invited.  When they have read their test instructions and downloaded their sample data files, they hit the green Start button to begin their test session.

R

KS Help Notes_Invites 01_User Guide to Taking an Assessment

TAKING A KS ASSESSMENT

Most assessment invites are sent by email. Or you might log directly into an assessment session via a browser. Either way, you will find your way to the start page.

Here is a typical KS invite:

The KS system assigns a username and password to each user. You can access your assessment by clicking the link at the bottom of the invite mail.  This takes you to the assessment login page, where you will be prompted to enter your username and password.
(NB if you lose or forget your KS password, hit the Reset password link to receive a new one by email).

Hit the Log in link and you may be presented with an additional information page, where you can submit a brief user profile.

Click Continue and you arrive at the main assessment start page. Here you will see a summary of the assessment you are scheduled to take and any special instructions from your system administrator.

At this point, open up your copy of the software, for which you are taking an assessment.

There are 9 types of questions, which typically comprise a KS assessment. Let's take a brief look at each one:

Free Text
This type of question requires a specific answer, which needs to match the answer stored in the KS database. Look out for precise instructions on how to format your answer, for example, precision, case sensitive answers, spacing, accurate spelling, and so on.

Multiple Choice
Here, you will be offered a selection of possible answers - and you need to select the best one. Watch out for 'distracter' entries, which are designed to make you think before selecting your answer.

Pick List
This question type offers you a range of options, which could represent a correct answer. Choose the best answers from the list provided, using the check boxes, to the left of the answers. Partial marks are often awarded for entering some of the correct answers.

Order List
This question type requires you to place a number of items in the correct sequence. Drag and drop the answer options, until they are in the right order.

True or False
This question type is pretty straightforward. Read the statement or question and decide if you think it is true or false.  Enter your answer using the radio buttons.

Matching List
This question type asks the user to correctly identify, or match the relationship between 2 lists of data.

Complete the Blank 

This question type asks the user to fill in one (or more) blank fields, or select from a list of available options in a sentence or paragraph.

Matrix 

This question type asks the user to review a list of short words or phrases and select a radio button option that relates to the ‘correct’ description. 

Essay 

This question type asks the user to write a longer answer to a question. This will not be automatically marked and requires manual scoring by system admins.

OK, so when you arrive at the start page, before you begin, you might need to download some sample data files to use during your assessment. The usual way is to select the orange, Download files button.

Clicking the orange button prompts you to extract and save the files locally on your work station. You will need to navigate back to this folder during your assessment, so most people choose their desktop, or My Documents, etc. The data sets do not need to be saved during a session, so can simply be deleted after the assessment has been completed.

Sometimes firms choose to zip up all sample test files and host them locally, on their own servers. If this is the case, then the orange button will not display. Instead, your administrator will provide special instructions on the assessment start page, explaining which file path to follow, in order to find the files.

Either method is fine - you simply need to know which folder to navigate to, in order to access the files for your set of assessment questions.

When you have opened your software, located your sample files and read any other instructions from your administrator, hit the green Start button and your assessment will begin.

Each assessment is timed, although the time taken to finish does not impact your overall score. The assessment user interface (UI) is easy to navigate. All sessions start on question one. There is also a small timer, which allows you to keep track of how long you are taking during your session.

Each question has a name and brief summary, which explains the main features of the software being addressed.

If a question is task-based, i.e. the user has to use the software in order to answer the question, the corresponding sample data set can be found in the zip folder you downloaded at the start. The orange instruction box explains where to find the files for each question.  (Quick tip: the files for task based questions share the same name as the question, to which they belong).

You can answer the questions in any order. The 'Question Navigator' allows you to move around the questions and generally familiarise yourself with the exercises. The simple color code helps you to keep track of your progress. Green squares indicate questions for which an answer has been submitted. Orange represents the current question. Red squares are questions that have yet to be attempted.

If you make a mistake, you can re-visit the green squares at any time, to edit your answers.

Some system administrators include the option to ‘skip’ questions and/or ‘request training’.  You will see additional buttons appearing on the test UI, if either feature is enabled.

An assessment finishes when all the questions have been answered. (Sometimes, an administrator will set a time limit for an assessment, in which case the session will automatically close after that time has elapsed. Answers submitted will be marked and any questions which have not been attempted will be scored as zero).

You have one last opportunity to review and edit your answers, before hitting the green Finish button, which triggers your assessment to be graded.

Next, you may be presented with another user profile page, as your assessment is being marked.

When the session is finished, you will receive a summary report, with feedback on any dropped marks. The level of feedback received can vary, depending on the settings prescribed by your account administrator.

A helpful summary of the questions, feedback on incorrect answers, coaching notes and training tags may be viewed, by selecting the Show question detail link at the top of your summary report.

The training tags (for questions where marks were dropped) combine to form a personal curriculum of suggested training workshop topics.

R