AUGI Top DAUG 2012 - The Results

Building on the success of last year's contest at AU, KnowledgeSmart and AUGI once again teamed up to provide an interactive skills assessment, across 8 popular Autodesk software tracks: 3ds Max, AutoCAD 2D, AutoCAD Civil 3D, Inventor, Navisworks, Revit Architecture, Revit MEP and Revit Structure.

Here's a brief history of the TD contest, from AUGI.  And here's a summary of last year's competition.

Once again, we had some cool prizes up for grabs.  Track winners walked away with a $100 USD Amazon voucher (a combination of high scores and fastest times determined the winner of each track) and the overall winner won a HP laptop and a free pass to AU2013.

Here's a shot of the contest rules board:

We spent several hours on the Monday, setting up 18 networked PC's in the exhibition hall, at the AUGI stand. We got off to a slow start when the exhibition hall manager decided that the PC's were facing the wrong way round. So we dismantled all 18 PC's, turned them around 180 degrees and wired them all back in again! (Which, of course, we were delighted to do!).

Now properly oriented, all 144 copies of  Autodesk software needed checking, then all the KS sample data sets had to be uploaded to each machine and checked again.  Big thanks to Joe Croser and Mike Johnson for their help in accomplishing this rather onerous task!

Here's how we looked when we were all nicely set up:

The competition ran over 2 days (1 x 3 hour slot on day one, then 1 x 2 hour and 1 x 3 hour slot on day two). Contestants had to answer 10 questions, using the 2013 version of each software title. Each session was limited to just 10 minutes, so people had to work fast!  Special thanks to awesome AUGI team members, Kristin, Bob, Michael, Donnia and Richard, for briefing users and making sure everything ran smoothly.

Here are the Top DAUG's in action:

By the end of the contest, we posted 336 results, with 0% failed tests. Considering the hotel web connectivity was, at times, positively creaking with the competition for bandwidth in the main hall, we are rather proud of the resilience demonstrated by the KS system.  Particularly when we saw some users juggling a beer in one hand, a mouse in the other and a bag of popcorn balanced somewhere in the middle! (Editor note: When you're writing technical specs for software, somehow you never quite anticipate all of the live scenarios your code might end up experiencing! :) ).

Throughout the competition, we posted a rolling list of the top 10 contestants for each track, on the AUGI big screen.

The Results

Congratulations to the following contestants, who won their respective software tracks:

And a special mention to the overall winner of AUGI Top DAUG 2012:

Fred Wismer

Analysis

So, let's take a detailed look at the results of this year's Top DAUG competition.

Overall

No. of Tests Completed:  336

Overall Average:  41% in 9 mins 26 secs

(NB the average score for 2011 was 53%).

Track 1 - 3ds Max

Track Winner: James Clarke (NB James is the Max track winner for the second year running).

Winning Score: 75% in 10 mins 0 secs

Top 10 Contestants:

No. Completed: 13

Group Average: 17% in 9 mins 34 secs

Track 2 - AutoCAD 2D

Track Winner: Glenn Sinclair

Winning Score: 100% in 8 mins 5 secs

Top 10 Contestants:

No. Completed: 113

Group Average: 37% in 9 mins 45 secs

Track 3 - AutoCAD Civil 3D

Track Winner: Fred Wismer

Winning Score: 100% in 7 mins 30 secs

Top 10 Contestants:

No. Completed: 46

Group Average: 61% in 8 mins 20 secs

Track 4 - Inventor

Track Winner: Tracy Chadwick

Winning Score: 60% in 10 mins 0 secs

Top 10 Contestants:

No. Completed: 16

Group Average: 22% in 9 mins 50 secs

Track 5 - Navisworks

Track Winner: James Austin
Winning Score: 80% in 10 mins 0 secs

Top 10 Contestants:

No. Completed: 19
Group Average: 42% in 9 mins 34 secs

Track 6 - Revit Architecture

Track Winner: Michael Patrick

Winning Score: 98% in 9 mins 30 secs

Top 10 Contestants:

No. Completed: 84

Group Average: 55% in 9 mins 20 secs

Track 7 - Revit MEP

Track Winner: Maxime Sanschagrin

Winning Score: 53% in 10 mins 0 secs

Top 10 Contestants:

No. Completed: 26

Group Average: 34% in 9 mins 50 secs

Track 8 - Revit Structure

Track Winner: Eric Bernier

Winning Score: 95% in 10 mins 0 secs

Top 10 Contestants:

No. Completed: 19

Group Average: 43% in 9 mins 32 secs

Popular Tracks

The most popular tracks, in order of completed tests, were as follows:

AutoCAD 2D - 113 results

Revit Architecture - 84 results

AutoCAD Civil 3D - 46 results

Revit MEP - 26 results

Navisworks - 19 results

Revit Structure - 19 results

Inventor - 16 results

3ds Max - 13 results

Range of scores

Interestingly, across the 8 tracks, we saw scores ranging from 0% to 100%. Here is a summary of both ends of the performance scale:

4 x 100% scores (2 x AutoCAD 2D, 2 x AutoCAD Civil 3D).

12 x 0% scores (1 x AutoCAD 2D, 4 x AutoCAD Civil 3D, 3 x Navisworks, 3 x Revit MEP, 1 x Revit Structure).

Honourable mentions

Along with our track winners, the following contestants deserve a special mention, for their performance in the competition:

Brian Mackey - yet another stellar performance from our 'Mr Consistent' competitor. Just pipped for 2nd place in the RST track, top 3 finish for Navisworks, plus a very good Revit Architecture score.

Brent McAnney - a winner in last year's contest, placed top 3 in both AutoCAD and Civil 3D tracks.

Glenn Sinclair - winner of the AutoCAD 2D track, top 10 for Revit Architecture and 2nd place overall in the contest.

Tracy Chadwick - winner of the Inventor track, top 5 finish for AutoCAD 2D, plus a very good Revit Architecture score.

Kate Morrical - another consistent performer, placing top 3 in both AutoCAD 2D and Revit Structure tracks.

Rebecca Frangipane - track winner from last year, top 5 finish in Revit Structure, plus a very good Revit Architecture score.

John Fout and Ben Downey for completing 4 tracks apiece.

Boys vs Girls

Finally, let's take a look at the demographic breakdown of the competition. Out of 336 contestants, 303 were male and 33 female. The average overall performance for each group breaks down like this:

Girls: 49% in 9 mins 18 secs

Boys: 41% in 9 mins 29 secs

So, that's Top DAUG done and dusted for another year. A thoroughly enjoyable 3 days at AU2012. 336 completed tests, across 8 popular Autodesk software applications. Once again, the overall standard was extremely high, with some outstanding individual performances from our track winners and top 10 contestants. And once again the Civil 3D track takes the overall prize!

Congratulations to all our winners. Thanks to the AUGI team for all their support, with a special mention for AUGI President, David Harrington and past-President, Mark Kiker. Lastly, a big thank you to everyone who took part in this year's contest.

R

KS server update

Quick technical heads up for this Sunday. We're moving the KS system from the existing Rackspace local server to a cloud based server. Nothing especially earth shattering. But an important update to our back-end hosting, which sets us up well for next year and beyond. The nice thing about cloud based hosting is the scalability. If we need to ramp up our RAM or increase overall capacity in future, then we'll have greater capability to do so.

So for 2-3 hours on Sunday 09 December, around 12pm EST / 5pm GMT, you might experience a little down-time in the live system.  We'll do everything we can to minimise the impact of this work.

R

KS Devs Update

As Q4 rolls around, we thought it would be useful to jot down a brief summary of what we're working on at the moment.

KS Library

All of the existing (Autodesk) library titles have been updated to 2013 format.  Our favourite task of the year. (Not really).  This includes: Revit (x3), AutoCAD, Civil 3D, Navisworks, Design Review and 3ds Max.

We're over half way to finishing an expanded Inventor 2013 question set.  Working with ace US author, John Evans, we are covering 4 key areas: Assemblies, Part Modeling, Drawings (and presentations) and Welding and sheet metal.  Our goal is to complete the set in time for AU2012.

We're compiling a larger set of general BIM related questions, for the KS Community area of the library.  These will be a FREE addition to the KS library.  Should be ready in time for AU, or thereabouts.

We have started work on a set of ArchiCAD fundamentals questions - our first foray into the Graphisoft world.  Covering 4 levels of difficulty, this set will be available in the New Year.

Advanced RAC, advanced RST and Revit for occasional users are all works in progress. Our deadline for these sets has slipped a bit, due to author commitments, but we're aiming for something on all 3 fronts by AU2012.

USACE has put together a working group to check the Attachment F assessment material.  We're awaiting the green light from the Corps, to release this free question set into the general KS library.

On the drawing board (can we still use that phrase?) for the next round of updates, include the following topics: Advanced RMEP, advanced Civil 3D, Bentley ProjectWise, AutoPlant and Plant 3D.

A variety of vendors are currently discussing writing test material for their own software products.  These include: CADS (CADS RC), Tekla (Tekla Structures), Synchro (Synchro Professional), Savoy (AutoTrack) and Codebook (Codebook).


KS System

The latest release of the KS tools hit the live site in early October.  This completed our program of works for the summer, within two weeks of our original goal, which is excellent.

Next up, we have two point-releases in the works, which address the remaining items on the KS user group wish list.  These include: capturing additional user data for more detailed statistical & benchmarking analysis; update to the test report format; additional library searching & grouping; tidy up of the users data page (including option to edit and delete master records); prep for supporting different languages in the test library;  update to account hierarchy options (including option to delete and move accounts); new charting option for flagged training request data, plus a few other smaller tidy-up tasks.

We'll also be moving to a new cloud server next month, which gives us a greater capacity to grow.  We anticipate a small amount of down-time, during the move, so we'll keep you updated.

Looking ahead, we'll be discussing the following additional features with our user group, in the coming months: User dashboard pages, new question types, survey questions & charting, links to third party systems, more detailed regional benchmark stats, additional charting & reporting options and a web based skills matrix.


Out and About

We'll be attending a few events in the next couple of months.  Autodesk host their annual BIM conference, in London, on 19 November.  The same week sees a free BIM learning and networking conference, hosted by Ramboll, in London, on 22 November.

Plus, KS is once again co-hosting the AUGI Top DAUG skills contest at Autodesk University.

So hopefully we'll have a chance to cross paths at some point this quarter.  Once again, thanks to our ever-enthusiastic user group, for a steady stream of ideas for new system features and library additions.

R

KS Help Notes_Invites 05_Setting up a Mock Test

Occasionally, KS firms ask if it is possible to set up a practice or mock test environment for users, so they can familiarise themselves with the KS test UI, format, question types, file download process, and so on, before taking a test 'for real'.

The answer is yes.  And it's pretty easy to set up.  Here's what you need to do...

Step One
Set up a Practice Test Account

First, set up a new account, called 'XYZ Engineers Practice Account/Sand Box/Mock Test Account', or similar.

See this link for more detailed notes on how to set up a new KS 'child' account.  Don't add any KS OTS tests when setting up this account.  Leave the KS test list blank for now.

Next, set up a new Full-admin profile for your new account.  Go to the Accounts > Your Accounts page of your dashboard and select the 'Administrators' icon next to your new practice test account.

Use the 'Add administrator' fields to add a new Full-admin to your account.

Use a general user name, i.e. XYZAdmin and a 'catchall' email address, such as training@xyzengineers, learning@xyzengineers, or similar.

Look out for the system mail with your new admin's login details.  Or, alternatively, re-set the password on your new admin profile.

Log in as your new administrator.  Now, hit the 'Change password' link, to create a new generic password, i.e. XYZ123.  These login details will be made available to all users, so should be easy to remember.

Last, change your new admin profile from Full-admin to Sub-admin status.  Don't forget, a Sub-admin user profile can only access a KS test from a browser.  It cannot be used to access your main KS dashboard.

Now that you have set up your new practice test account, you need to create a mock test, for your users to log into.


Step Two
Set up a Mock Test

Choose a suitable topic for your mock test, i.e. AutoCAD, Revit, Civil 3D, MicroStation, etc.  Log into your main KS 'parent' account and create a new practice test from scratch.  See this link for more detailed notes on how to create your own KS modules and tests.  See this link for more detailed notes on how to import and edit a KS OTS test.

Set up your new test so that you only have a single module, with 3 or 4 sample questions presented in your mock test.  Save your changes and publish your new test.  See this link for notes on how to publish draft KS content.

Now, go to the Accounts > Manage Content page and copy your new mock test across to your practice test account.  See this link for notes on how to copy published tests from a 'parent' account to a 'child' account.

Now, any user can log in to a mock test session, using the generic Sub-admin profile, to get a preview of what a live KS test session looks and feels like.

You might want to create some simple user instructions on how to log in to a KS test from a browser.  This link will give you some useful info to share with your team.

R

KS user passwords

This query comes up from time to time, so I thought it worth documenting on the KS blog.

Some invite mails contain a username and password, but others do not.  Why is this?

The answer is pretty straightforward.  The KS system appends a username and password on invite mails for regular users (i.e. non-admins).  Further, if a user receives two or more test invites in close succession, the latter invite(s) will overwrite the password(s) from earlier mails. Which means they need to use the most recent password for all active test sessions.

All users can re-set their KS password at any time, by clicking the 'Forgot details' link on the KS dashboard login page, which takes you here.

Enter your KS username and the system will email a new password.

System admins can set their own password, so the KS system does not include usernames and passwords on test invite mails for administrators.

KS admins can set their own password, by logging in to their KS dashboard and selecting the 'Change password' link.

Copy the original password into the first field, then confirm your own password and save the changes.

R

AUGI Top DAUG 2012

The AUGI Top DAUG competition was introduced at Autodesk University, back in 2000. The contest originally featured AutoCAD 2D knowledge. This year, once again, AUGI is teaming up with KnowledgeSmart to expand the range of topics to include 8 software tracks.

Here are some guidelines for this year's contest..

All contestants must attend Autodesk University in order to compete.

The contest includes modules based on the following software versions:

- 3ds Max 2013
- AutoCAD 2013
- AutoCAD Civil 3D 2013
- Inventor 2013
- Navisworks Manage 2013
- Revit Architecture 2013
- Revit MEP 2013
- Revit Structure 2013

Contestants may participate in multiple modules, but only take each module once.  (Any contestant taking the same module more than once will be disqualified).

A combination of test scores and fastest times determines the winner of each module and the overall winner of the competition.  In the event of a tie, the winner of the contest will be decided by a coin toss.

The following are not allowed to be used in the Top DAUG contest area:

- Cell phones, smart phones or any electronic devices
- Reference books and/or websites
- Autodesk materials, training courseware or related materials
- Personal 1 on 1 coaching from Brian Mackey (just kidding, Brian! ;) )

A range of prizes will be awarded for the best performers.  See you all at the Mandalay Bay, in a couple of months!

R

KS Help Notes_Library 10_Precision on Free Text Questions

KS admins can determine answer precision on free text type questions.  To edit a question, go to Library > Draft Content > Draft Questions and hover your mouse over the 'Edit question' icon. Select 'Provide the answer' in the flyout menu.

If an answer is numeric, admins can allow a variance in the value presented in the answer field by users, assigning full or partial marks for 'nearly right' answers.

Admins can provide user hints for answer precision (i.e. the number of decimal places required in the 'correct' answer).

For non-numeric answers, we have included the option to disregard incorrect case and white spacing in the answers submitted by users.  So, for example, if the 'correct' answer to a question was 'Top DAUG', but a user entered an answer of 'Top  daug', the admin can decide whether to assign full, partial or no marks.

As a default setting, KS OTS (off the shelf) tests assign full marks to answers with minor variance in case and white spacing.  Precision hint and variance for numeric answers is not enabled.

R

KS FAQ's

Here are the most commonly posed questions, about the KS admin dashboard and assessment journey.

Q - What's the difference between a Sub-admin and a Full-admin?
A - Sub-admins can set up test sessions from a browser, using the 'Admin Test Setup Login' link, but not access the main KS dashboard. Full-admins can log in to the KS dashboard and use the browser test setup tools.

Q - How do I re-set or change my KS password?
A - All users can re-set their KS password at any time, by clicking the 'Forgot details' link on the KS dashboard login page, which takes you here. Enter your KS username and the system will email a new password.
KS admins can set their own password, by logging in to their KS dashboard and selecting the 'Change password' link. Copy the original password into the first field, then confirm your own password and save the changes.

Q - How do I add a new KS administrator to my account?
A - Go to the Accounts > Your Accounts page of the KS dashboard and click on the 'Administrators' tool (small people icon). From here you can add new Sub-admins or Full-admins, using the 'Add administrator' fields.

Q - I get an error message when I try to import my user list. Why is this?
A - First, make sure that you are using a .csv file format, not .xls or .xlsx. (See help notes on the Users > Upload User Data page for a link to a formatted .csv template). Also, make sure that you have 10 data columns in your .csv file.

Q - Do I need a new username and password for the KS Support Ticket System?
A - Not really. You can use the same login details as you use to log in to the main KS dashboard, or you can choose a different username and password to access the support ticket tools. Either will work OK.

Q - My invite mails are not getting through to users every time. Why not?
A - Sometimes KS invite mails are blocked by spam prevention software. If the KS mails are not in junk mail, then your corporate security might not be allowing the mails through. This happens from time to time. A combination of the following steps usually resolves the issue:
‘White flag’ the domain @knowledgesmart.net.
‘White flag’ the domain @ks-server.net.
‘White flag’ mails from system@ks-server.net.
That should allow the mails to work OK. However, if they are still getting blocked, there is one more thing to try. Our hosting company (1&1) sometimes uses a relay server in Germany, called kundenserver.de.
‘White flag’ the domain @kundenserver.de.
That should sort everything out, from a corporate mails point of view.

Q - Some invite mails contain a username and password. Others do not. Why is this?
A - The KS system appends a username and password on invite mails for regular users (i.e. non-admins). If a user receives two or more test invites in close succession, the latter invite(s) will overwrite the password(s) from earlier mails. KS admins can set their own password, so the KS system does not include admin usernames and passwords on test invite mails.

Q - I can't log into my test session. A message refers me back to the system administrator. What's happening?
A - This could be a number of things. Check that the test invite doesn't have an expired date that has passed. Check that the test doesn't have one or more modules with '0' in the 'assigned questions' box (i.e. an empty module). Check that the original invite hasn't been deleted, by viewing the Invites > History page.

Q - What happens if a 'Whoops Gremlins!' message appears during a test session?
A - If you lose a network or web connection during a test session, an error message might display. This sometimes happens as you move between questions. Click the back button once to return to the previous screen and wait for a few seconds. The test session should resume as normal.

Q - I am seeing one or more error messages onscreen during a test session, when I navigate between questions. This is not caused by loss of web connection. What else could it be?
A - If the same question has been assigned in a test more than once, this can cause an error message to display. Check to make sure that you have not duplicated the same question in one or more modules. (Quick tip: when assigning questions to modules, use a single mouse-click. A double-click can cause the same question to display twice).

Q - I can't answer the 'order list' questions in my test session because the drag & drop feature isn't working. Why not?
A - We support the following browsers: IE8, IE9, Google Chrome, Mozilla Firefox and Apple Safari. We are aware that the 'order list' type questions occasionally cannot be completed in IE6 or IE7. There is an easy workaround. Simply log out of the session, then copy and paste the original invite link into one of the other browsers listed (or use the 'Resume Test' tool in the Invites > History page of your dashboard). Enter your user details and click the 'Resume' button to log back in to the session (all existing answers will be saved). Complete the remaining question(s) and click 'Finish', then click the link to view the test report.

Q - If I log out of a test session, will my answers be saved?
A - Yes. All your answers will be saved if you resume a test session, at a later date.

Q - My test result did not upload correctly at the end of a test session. Can it be rescued?
A - Yes. If you lose your web connection at the end of a test session, occasionally a result will not upload to the dashboard successfully. Just raise a support ticket and the KS team will manually retrieve the score and add it to your dashboard. Your work will not be lost.

Q - How can I tell which version (ID number) of a test someone took? For example, I have two different Revit fundamentals tests, but I don’t know which one a specific user took.
A - When the invite mail goes out, the test ID is appended after the test name. Results mails also include the test name and ID number.
If you can’t locate the original test invite mail, you can track individual question ID numbers in the test report at the end of the test. Select a question ID from the report (any question will do), then go to Library > Draft Content > Draft Questions and enter the question ID number into the ‘Search Questions’ box. Hit ‘Search’ and you’ll filter the question library, to display the question which carries the relevant ID number. Click on the question name and view the green popup panel. About a third of the way down, you will see a section called ‘Used In (Modules)’ and another called ‘Used In (Tests)’. Here, you can read the ID number of the parent module and/or parent test, for that question.

R

KS Help Notes_Invites 04_Invites History

Admins can view a full list of sent invites, including test status, on the Invites > History page.

Use the account dropdown menu, to view individual accounts or all linked accounts.

Use the Filter by invite status tool to sort the view for all sent invites, into 3 groups: Not Started / In Progress / Completed.  

Search your invites by selecting the ‘Show search’ link and entering user name, test name or invite sent date in the relevant field.  Then hit the ‘Search’ button to filter your data.

Sent invites data can be exported to CSV for further analysis.

Re-send multiple invites at the same time, by selecting the relevant invite(s) using the check box(es) and clicking the Re-send invites button.

Delete multiple invites at the same time, by selecting the relevant invite(s) using the check box(es) and clicking the Delete button.

Change the expiry date for one or more invites, by selecting the relevant invite(s) using the check box(es) and clicking the Extend invites button.

Use the Resume Test Session tool, to log back into user test sessions.

R