KS Help Notes_Admin 03_User FAQ

CANDIDATE INTRODUCTION AND FAQs

WHAT IS KNOWLEDGESMART?

KnowledgeSmart is a web-based skills assessment tool, which enables architectural and engineering practices to measure the software skills of their teams. Since 2003, by carrying out in excess of 25,000 individual assessments globally, a helpful independent benchmark has been established for basic proficiency, across a range of subjects.

The KnowledgeSmart tools help to identify skills associated with design and engineering software applications, such as AutoCAD, Civil 3D, Plant 3D, Revit, Inventor, MicroStation, 3ds Max, ArchiCAD, Adobe Photoshop, Adobe InDesign, SketchUp and Rhino. The process aims to assess current performance in order to identify areas where individuals can benefit from additional training.

WHAT IS THE POINT OF KNOWLEDGESMART AND WHY SHOULD I DO IT?

This process is not about a big-brother style test to check up on unsuspecting users. Nor is it a matter of pointing out who is the more knowledgeable software user. Rather, the point of the exercise is to highlight how much each individual currently knows – in order to identify those areas where focused training can improve basic knowledge.

Everyone uses software in a different way. Some people use these tools full time in their role; others may only use the tools from time to time. The objective is not to teach a veteran user how to perform basic tasks. But the fact remains, a more detailed knowledge of basic commands and operations can alleviate apprehension in using technical software applications and assist in speeding up routine tasks.

IS IT THE SAME ASSESSMENT FOR EVERYONE?

There are a number of assessment modules to choose from, created by a team of expert authors.  Firms can also create their own material in-house. For Revit skills, as an example, there is a choice of material, ranging from introductory level, all the way up to more advanced concepts.

Summary reports take into consideration whether an individual is a full time user, or someone who only uses the software occasionally as part of a wider role. Follow-up comparisons and training recommendations should be reflected accordingly.

WHAT KIND OF TOPICS DOES EACH ASSESSMENT COVER?

KnowledgeSmart assessments typically break down complex technical software applications into a series of stages or modules.

For example, the fundamentals assessments (for AutoCAD & MicroStation) cover a range of basic commands, including; basic object/element creation, layers/levels, blocks/cells, annotation, referencing & printing, UCS/ACS and preferences.

Assessments typically comprise 20-30 questions and present scores as a percentage, based on the accuracy of the answers presented.  There are 2 question types and 9 question styles.  Question types are either task-based (i.e. you need to perform a task using the software in order to work out the answer) or knowledge-based (i.e. you are being evaluated on your understanding of certain software functions, without having to actually use the tools).  Question styles include; Free text, Multiple choice, Pick list, Order list, True or false, Complete the blank, Matrix, Matching list and Essay.

WHAT IF I DO NOT SCORE VERY WELL?

This exercise is not about who is the best. You will identify areas where you can improve your own performance with the help of more targeted training workshops.

Following the assessment everyone receives an overall result and summary report with feedback on dropped marks, training recommendations and coaching notes.

WHEN AND WHERE WILL THE ASSESSMENT TAKE PLACE?

Your administrator will have details of your assessment program. If you are unavailable to make the date or time arranged please contact your administrator to reschedule.

HOW LONG DOES IT TAKE?

There is no limit to how long the exercise will take. The average time is approximately an hour, but this is not a strict rule.

Try to work at your usual speed. Don’t rush, as many simple mistakes are made by people not reading the instructions carefully. Conversely, don’t take too long to complete each task, as your score plus time taken will be looked at in the overall results analysis.

WHEN DO I GET MY RESULT?

You will receive your score straight away when you finish the assessment. You may also be emailed with a link to your assessment report, including feedback on any marks which have been dropped.

ADDITIONAL INFORMATION

The amount of time you take will not affect the accuracy of your score.

Before you start, you may be presented with a zip folder containing files which you’ll need to refer to during your test. Save the zip folder locally on your machine and extract the sample data sets from the zip.  And remember where you save it – as you will need to navigate back to this folder during your assessment.

Please read the written instructions carefully for each question. This is also an assessment of your ability to follow instructions. Often, marks are dropped through candidates simply not doing what they have been asked to do.

During your assessment session, you can answer the questions in any order, using the KnowledgeSmart question navigator.  You can also edit your answers during the session and review your work once more at the end, before you select Finish.

The time taken to complete an assessment varies depending on the individual, but an average is about an hour.  You don’t have to complete the assessment in one sitting; you can log out and return later if you prefer.  When you resume, your previous answers will have been saved.

Here’s a short movie which will tell you more: KS user movie.

R

KS Help Notes_Admin 02_Planning your Assessments

ADMINISTRATOR NOTES FOR PLANNING AN ASSESSMENT PROGRAM

COMMUNICATING YOUR PLANS

It is important to explain to candidates precisely why you are asking them to undertake a skills assessment. You might do this by including the assessment in a wider discussion about training needs and more efficient use of technology within your company, including training needs, recruitment, induction and annual appraisals processes.

Improving technical software skills is a continuously improving process – the KnowledgeSmart tools simply highlight a likely starting point. As with most things, good communication goes a long way. An effective strategy is to hold a team meeting and explain your overall plans for improving performance within the business. This can include a Q&A session, where people have an opportunity to clarify any aspects of the assessment process – and understand how it fits into the bigger learning picture.

It is important that people don’t feel that the results of their skills evaluation may be used in a negative context, particularly if they don’t perform well first time around. Ensure they understand this is a designed to be a learning process – and everyone has to start somewhere! By taking a snapshot of each individual’s current skills at a basic level, you can feed this data into your training plans moving forwards.

This is not all about who is the smartest, fastest, or best Revit user.  Or the most experienced MicroStation guru; although these are useful things to know.  The real value in the KS tools is trying to help users, at an individual level, identify what they know and also what they need to brush up on next, in order to improve their skills, with a particular piece of software.

Since 2003, KS has facilitated over 25,000 individual test sessions, across hundreds of AEC firms.  Experience has shown that, in general, people don't like the idea of taking a test - at least that's usually how they feel at first.  However, experience has also shown that, for most people, when they complete their assessment, they realise that it wasn't half as bad as they thought it would be! Further, they often learn new skills as they go through the process of answering the questions.  And lastly, they find the feedback and coaching notes extremely helpful, in giving them an unbiased appraisal of their skills using a particular software application - relative to their peers in industry.  It's important that users have the opportunity to review their test scores and feedback with their technology leaders or learning & development leaders.  And it's also important to explain to your users that all scores are confidential.  Best practice would recommend that only senior administrators should have access to firm-wide results data.

Provided you explain to your teams that the main point of rolling out an assessment program is to gather valuable training needs analysis data, rather than a big-brother style exercise in peeking over shoulders, then most users feel OK about the idea. And when you follow through with more focused, modular training workshops, which target recognised skills gaps, then it's a win-win for users and management alike.

Evidence shows that better overall understanding of basic commands can also benefit the implementation of in-house standards. Once you have presented to your teams the case for measuring and improving skills, you need to set a timeframe for rolling out your assessments.

PREPARE A TIMESCALE

You need to decide over what period of time you want to complete your assessments. This will depend largely on 3 variables; 1) the number of people to be assessed 2) the number of offices in your organisation and 3) general day-to-day work pressures.
You also need to decide what your preferred assessment environment will be. Some companies prefer their teams to use their own work stations, whilst others favour a more formal setting. In truth, there is no right or wrong answer to this question; you want people to feel comfortable and remove any potential for apprehension or pre-assessment nerves.

A formal assessment environment can be helpful, where people simply take an hour or so, away from their desk, at a specified time, to sit their assessment. This way, they aren’t distracted by project work, email, ringing phones or interruptions from colleagues.

For multi-office firms, it is best to start with one location and, when this office is done, gradually ramp up the level of activity until the assessment program is complete. You will need the support of your regional technology co-ordinators if this is to be a success.

PREPARE YOUR TEAMS

We recommend you brief your teams prior to their assessments. You might like to share with them the KS FAQ document and/or KS user video, or perhaps present your plans at a team meeting.  It is best to circulate the information at least day or two before people are due to take their assessments.

SEND YOUR ASSESSMENT INVITES

Use the inviting tools in the KS admin dashboard to manage your assessment program.  You can send invites individually, or create a longer list of users and invite them in groups.

Next, you may customise your welcome message in the invite mail and on the assessment start page.  You might wish to present users with specific instructions on where to locate their sample files (if you have chosen to store them on your own network, for example).

COMPLETE YOUR ASSESSMENTS

Monitor your teams as they complete their assessments. Administrators receive an email with a link to test results, as and when they are completed.

You can use the invite history page of your admin dashboard to track invites and completed assessments.  You can also re-send invites from here.

AFTER THE ASSESSMENTS

KS administrators can decide how much feedback to present to their users.  At the end of their assessment, candidates may receive an instant result and full summary report with a breakdown of their score, training keywords and coaching notes. In addition, they may be sent an email containing a link to their report.

Account administrators can access all assessment results, feedback, training data and coaching notes – for each candidate – in the KS admin dashboard.

Via the admin dashboard, it is possible to search results, view performance charts and export data into Excel for in-house reports. It is important to provide the opportunity for individuals to access a structured training program, after their skills have been evaluated. The results will highlight areas where future training workshops can be focused for maximum improvement.

It can be helpful to schedule a follow up meeting with KnowledgeSmart, to analyze your results data.  If you work with an external training partner, it is a good idea to ask for their recommendations on post-assessment training activity.

R

KS Help Notes_Admin 01_Checklist

KS ADMIN CHECKLIST

Here is a checklist for KS admins to use, when planning their skills assessment program.

• Sign KS subscription agreement/renewal
• Identify Technology, HR, L&D and Management sponsors
• Agree on your goals; how will you measure your success?
• Identify KS admins; who will coordinate your assessments?
• Decide on account hierarchy; one account or several?
• Schedule KS admin web trainings
• Decide on your library content; ‘off the shelf’ or customize?
• Agree on your message to users; clear communication is important
• Feedback and coaching options
• Include assessments for new hires at interview?
• Send out your invites
• Monitor your assessment activity
• One-month review
• Post-assessment review; analyzing your results, charting and reporting
• Follow up activity and training
• Post-training assessments; measure improvements 
• New features?
• New assessment titles?

Sponsors

The first task is to identify who your main contacts will be across your organisation, who will help support your skills assessment activity. Experience shows that it is helpful to gain support from the following key business areas:

• Technology & BIM

• HR and Learning & Development

• Management

Goals – Measuring your Success

Think about your goals.  What do you want to achieve from your assessment program?  Is it better visibility of training needs? Is it a benchmark for minimum performance? Is it a team by team breakdown? Do you want to compare different offices? Or different countries? What are the most important assessment topics for your organisation? Do you want to make the assessments optional or mandatory? (Historical evidence suggests that it will help your % completion rates if there is a clear message from management, that this is a mandatory exercise).

How will you measure your success?  What training resources do you have access to (in-house or 3rd party)?  Do you have a formal training partner? Do you have access to E-Learning material?  Do you have a LMS or HR database?  Do you have a company Intranet?

Your Account Structure

Think about your KS account hierarchy and any additional administrators who will be responsible for coordinating assessments for their teams or offices.

For example, you can administer your assessments from a single account, or you can set up multiple accounts, representing different offices or business groups, within your organisation.

You can decide which admins will have ‘global’ access to your assessment data and which will have more restricted admin permissions.

Admin Training & Support

Schedule some web trainings for your KS admins. There are a range of help guides available for using the KS tools, which will help you and your team quickly get up to speed.  

It is helpful to schedule a getting started webinar, to walk through the system, answer any questions and share some ideas on good practice for successfully managing your assessment program.  The KS team will be available to help you, as you get up and running.

Library Content

Think about your assessment library.  Do you want to include any custom content, or simply use the KS library unchanged?  Most firms tend to use the material ‘off the shelf’, at least to begin.  Over time, as you become more familiar with the library questions, you’ll likely make some changes and add your own material.

Communicating Your Plans

Think about how you wish to communicate your plans to your teams. Good communication is essential in explaining to your users what you want to achieve. Review the KS guides on managing your assessment program, together with the user Q&A information and video.

How much feedback do you want your users to see at the end of their assessment? Do you want to include coaching notes or links to learning?  Do you want to assess new hires at interview?

Review Your Progress

One month review; schedule an interim update call with KS, to make sure everything is on track. It’s still early days, so it sometimes helps to share any technical issues or queries you may have at this stage.

Sending Your Invites

Send out your invites and commence your assessment program.  Monitor progress and chase up any missed appointments.  The KS team will be on hand to assist with anysupport issues you encounter.

Analyzing Your Results

Post-assessment review; schedule an update call with KS, to discuss progress and analyze your results in detail. Look at options for charting and reporting your data. Discuss skills gaps and training needs. Will you involve a training partner at this stage?

Training

Feed the skills gaps data from your assessments into a continuous improvement program of training, lunch & learns, seminars, E-Learning, mentoring.

Post-Training

Post training assessment. Use the KS tools to measure the effectiveness of your training and how much information people have retained.

Feedback

New features & feedback; please share your thoughts with us on what you would like us to develop or improve in the KS dashboard.  

New library material; do you want us to add any new titles to the KS library?

R

KS System - next release

We're just going through the final round of testing on our staging site, before we move the next release of the KS tools to the live site.  The beauty of a web based system is that all KS customers get to enjoy the latest features, as soon as they go live, without worrying about physically un-installing or re-installing software.

A number of firms have asked for a heads up, as to what will be appearing in the March release.  We'll be talking about this in detail at the upcoming KS user group, in London, but for now, here's a brief overview of the new features, plus a look ahead to the point-release currently in development, which rolls out around May.

The main theme of the next release is to introduce account hierarchies, for firms with two or more locations in their organisation.  Large firms, with multiple KS accounts in place, will now be able to view and manage data, across linked accounts.  This impacts on almost every area of the KS admin dashboard.

KS Library

A common request is to enable firms to share custom test material, which has been created in one KS account, with the related accounts in an organisation.  The 'Test Content' pages have been re-branded as the 'Library' area of the dashboard, with a new menu structure now appearing. The 'Your Tests / Modules / Questions' area now appears as 'Draft Tests / Modules / Questions'. When a KS OTS (off the shelf) test is imported, the editable copy now appears in 'Draft Tests'.  The individual editable stages appear in 'Draft Modules' and the editable questions appear in 'Draft Questions'.

When you are happy with your new content, (either material you have written in-house, or an OTS test that has been modified), you now have the option to 'Publish' your modules and tests.  This effectively locks down that content and allows you to share it across linked accounts.  Draft content can not be shared between accounts.  Published modules and tests can be un-published at any time, for further changes.

Viewing Linked Accounts

Another key change to the KS dashboard operation, is the introduction of a hierarchy of linked accounts.  Admins can now view their account structure of 'parent', 'child' and 'sibling' accounts.  For admins with permission to access multiple accounts, a new 'Switch account' feature allows simple navigation between accounts. For viewing data across linked accounts, a new sub-menu dropdown is available in the Library, Invites > History, Results > Data and Users pages.  This sub-menu can be used to drill down to individual accounts, or to view information from all linked accounts, organisation-wide.

Creating New Accounts and Administrators

It is now possible to add new KS accounts to an organisation hierarchy, and also to create new account administrators for each account.  In addition, we have created a new type of admin profile, a Sub-administrator.  Full administrators can assign their preferred status to new admins and also change the status of existing admins.  Sub-admins can only set up KS test sessions from a browser; they cannot access the full admin dashboard. This is particularly useful for delegating the task of setting up test sessions for interview candidates, without the need for a full KS administrator to be available every time.  It also opens up the option for firms to create a general admin profile, which they can make centrally available, for users to 'self-invite' themselves to take a KS test or re-test, via a browser (i.e. skip the need to issue an email invite).

Full admins can also assign account permissions (i.e. decide how high up the organisational hierarchy new admins are allowed to access company data) and flag additional admins to receive system emails (i.e. invite mails, results mails, etc.).

Managing Content

We've added a selection of new tools, which allow Full admins to share KS data, across linked accounts.  For example, moving user records between accounts, moving test results, sharing modules and tests between accounts and applying changes to account settings, across linked accounts.

We have also tweaked the logic, relating to usernames and passwords appearing on test invite mails. Before, this data was only appended to the first instance of an invite. Subsequent invites only included the test name and URL.  Now the username and password is included in every instance of an invite, which will reduce the need for users to re-set their passwords (i.e. if they have lost or deleted the original password).

So, as you can see, quite a few changes to the overall structure of the KS system, but offering greater flexibility and control to customers, to manage data across their organisation.


Here's a sneak peek at what's currently being worked on, from the KS user wish list..

General update of the KS system to a cloud based server environment, including an updated operating system, more memory and greater flexibility for future growth.

Automatic retrospective grouping of new user and results data, based on existing groups.

New charting options and improvements to existing results data. New exporting options for results data. Linking of question footprint information, to Question Performance chart.

New options for admins to select level of precision on free text answers.

Update to the rich text editing tools, for creating and editing questions.

General improvements to the 'Invites' area of the dashboard, including auto emails for expired tests, reminder mails for users and capturing data for Issued/In Progress/Completed tests.

Updated 'Resources' area for admins, on how to utilise the KS tools and manage their assessment program.


This little lot will be finding its way into the live site in the next 3 months.  We'll be discussing all this and more at the KS user group, in London, on April 26.  We will also be mapping out the technical roadmap (sometimes referred to as, the 'Jane List') for the second half of 2012 and beyond.  Themes include; capturing and displaying additional benchmark statistics, personal dashboard pages for users, survey questions, links to training and e-learning providers and links to HR systems and learning management systems.

R

KS User Group - April 26, London

We’re hosting the next KS user group meeting, in London, in April.  Here are the details:

Date:

Thursday 26 April

Location:

Smeaton Room, Buro Happold, 17 Newman Street, London

Start:

Tea & coffee at 08.30, for a 09.00 start

End:

Last session wraps up at 13.30

Cost:

Free (breakfast & lunch included)

We will be reviewing the KS tools in some detail, including a look at what’s coming up in the next 3-6 months.  Plus a guest presenter or two.  Full agenda to follow, as the date gets closer.

Places will be limited to 25, on a first come, first served basis.  As of close of business Tuesday 28 Feb, we have just 6 slots remaining.

Hope you can make it!

R

2012 Priorities for AEC Firms

I read an interesting White Paper from Deltek last week, looking at where AEC firms are focusing their energy this year.  Conducted in November and December 2011, Deltek’s survey polled 430 decision makers at primarily US based AEC firms.

Not surprisingly, Marketing & Business Development and Cost Control are the two key themes for AEC firms in 2012.  70% of respondents said that increasing business development is their biggest priority this year.

This is something we are seeing across all regions.  Most AEC firms have now begun to implement a BIM strategy, but the majority are still in the early stages of communicating a joined-up message.  One of the areas the Deltek paper addresses, is how architects and engineers can make the most of Business Intelligence, in order to win more work. 

A trend we are noticing, is the need for AEC firms to demonstrate their BIM credentials.  This is important for a number of reasons.  The more data firms can gather on how proficiently they can deliver on a project, the more confidence they can instil in clients, prospective clients and their supply chain partners. It's one thing to say, 'Oh yes, we can deliver this project to BIM Level 2'.  It's quite another thing to actually be in a position to back up the words with hard evidence. This includes metrics from past or current BIM projects and performance data on individuals in the team, who will be hands on delivering the project information and working alongside the client and the GC.

And when this information is identified, it is vitally important that the business development teams are fully briefed on what it all means.  More and more firms are being requested to complete BIM-readiness questionnaires these days and, as clients become more knowledgeable about BIM, those forms will be requesting ever increasing levels of detail.  The phrase 'BIM Execution Plan' is already becoming a familiar refrain and this is not going to go away.

No longer can AEC firms simply trade on their past reputations.  They need to come up with a new message, one which truly differentiates them from their numerous competitors.  We came into this recession, at the end of 2008, on the back of a near 10-year boom.  We should now be bracing ourselves for a 7-10 year down- cycle, which is currently only 3 years old.  It is those firms who can capture relevant, objective, BIM-related performance data, who will be the real winners in the continued lean years, which lie ahead.

Here is a link to the Deltek paper:
http://www.deltek.com/pdf/whitepapers/AE-Firms-2012-Priorities-WP.pdf.

R

Demonstrating Professional Credentials

A common theme, here at KnowledgeSmart, is that of skilled professionals being able to demonstrate their capability and credentials, whatever the subject.  And it is a topic which is being discussed at the highest levels, here in the UK, as 2012 rolls around.

Back in 2009, I posed the (rather tongue-in-cheek) question; what do bus drivers and welders have in common that CAD & BIM professionals don't?  The answer is simple; bus drivers and welders need to demonstrate their professional credentials on a regular basis throughout their careers, in order to be allowed to do their job.

Bus drivers (and lorry drivers) have to pass a 'certificate of professional competence' in order to drive their vehicles in the UK.  The CPC was developed as a requirement of a European Directive, designed to improve the knowledge and skills of professional LGV and PCV drivers throughout their working life. There are two parts to the legislation:

- An Initial Qualification that must be achieved by new (LGV and PCV) drivers along with their vocational licence to enable them to use their licence professionally.
- Periodic Training, which involves all professional drivers undertaking 35 hours of training every 5 years.

Meanwhile, welder certifications are specially designed tests to determine a welder's skill and ability to deposit sound weld metal. The tests consist of many variables, including the specific welding process, type of metal, thickness, joint design, position, and others. Most certifications expire after a certain time limit, and have different requirements for renewal or extension of the certification. Once a welder passes a test (or a series of tests) their employer will certify the ability to pass the test, and the limitations or extent they are qualified to weld, as a written document (welder qualification test record, or WQTR).

So if the driving industry has a recognised benchmark for performance and if the welding industry won't allow its members to pick up a blow-torch without first demonstrating they have the requisite skills, then why on earth does the AEC industry let just about anyone gain access to a CAD or BIM software license?

I remember reading a story on ENR.com, a while back, highlighting the dangers of unqualified staff working on projects. A warehouse in Philadelphia suffered a major structural collapse, resulting in damages of $3.5M USD. A building-collapse expert concluded, 'The team committed several engineering errors of "amazing proportions" that caused the Philadelphia warehouse to fail under the weight of snow'. He found that the warehouse had only one-third of the steel roof framing it needed.

Someone using analysis software grossly underestimated the quantity of materials, the error went unnoticed, the building was constructed, the snow fell... and they firm got hit with $3.5M in costs. Plus a number of people were hurt when the roof fell in. Ouch! Suddenly a few extra dollars on testing a person's proficiency using engineering software, plus a few hours training, seems like a good use of the company resources!

A CIO at a large US design firm once said to me; “The minute you give someone a BIM license on their desktop – you are effectively giving them full access to your end product. You want to be sure they know what they are doing!”. This firm doesn't allocate a license to someone's desktop unless they have first met the minimum threshold competence level for performance, as identified by the firm's BIM administrators (using KnowledgeSmart, in this instance).

A bus company doesn't give someone the keys to its shiniest double-decker without making absolutely sure they have the current skills required to drive the vehicle. Isn't it time that the AEC industry recognised the need for clear skills certification in core authoring, BIM, Civil and analysis software packages? And a continuous improvement program for design and engineering professionals to demonstrate their credentials to their employers on an ongoing basis?

Well, hopefully, that is precisely what is happening, as the UK Government mandates use of BIM Level 2 on all appropriate construction projects, by 2016.  Despite (or perhaps because of) the ongoing economic hardship around the world, many firms are revisiting their technology assessment and training programmes, to make sure that they can measure the skills of key personnel.  Firms who can demonstrate to clients that they have the requisite skills to deliver a BIM project safely, on time and budget, will have a definite competitive advantage, as the clock counts down to 2016.

R