Testing software skills of new hires (in the US)

I've blogged on this subject in the past, but it just keeps coming up in conversation with HR professionals, Stateside; is it OK to test the basic CAD or BIM skills of prospective employees in the USA? A common conundrum surrounding skills assessments is whether you are legally permitted to screen the technical software ability of prospective employees and contract staff, before they join your team.

The short answer is yes, but only under the right circumstances. Let’s examine some of the evidence.

The most important issue underlying the use of pre-employment assessments is validity. The question we need to ask is this; ‘Is the test valid for this intended purpose; does it support the decisions that are going to be made?’

So what is 'validity'? Validity measures how appropriate a test is for a specific purpose. Simply put, a test may be considered valid for one use and invalid for another. Why do pre-employment tests need to be validated? In 1978, the Equal Employment Opportunities Commission (EEOC) created guidelines to ensure that the knowledge gained from testing is applied with impartiality to protect minority applicants from discriminatory employment procedures. What is the best method of validation? The EEOC guidelines do not state that one method is better than another; the method used must fit the needs of the business or organization.

There are three methods of validation set forth by the EEOC:

1) Criterion Validity: If data demonstrates that a test is significantly correlated with a vital measure of job performance, the test is said to demonstrate criterion validity. For example, if all the users that scored highly on a selected test to measure CAD skills completed their projects accurately and on time, the test would demonstrate criterion validity.

2) Construct Validity: The term construct is a technical term for personality traits like intelligence and creativity. Construct validity is demonstrated if a test measures traits that have been found to influence successful performance of a job. A test that measures the interpersonal communication skills of a potential customer service representative would demonstrate construct validity.

3) Content Validity: This is demonstrated if the questions that make up an assessment are representative of content that is required to perform a particular activity or task. A test made up of algebra questions given to an applicant for a maths teacher's position would demonstrate content validity.

Many employers use employment tests and other selection procedures in making employment decisions. Examples of these tools, many of which can be administered online, include the following:

- Cognitive tests assess reasoning, memory, perceptual speed and accuracy, and skills in arithmetic and reading comprehension, as well as knowledge of a particular function or job;

- Physical ability tests measure the physical ability to perform a particular task or the strength of specific muscle groups, as well as strength and stamina in general;

- Sample job tasks (e.g. performance tests, simulations, work samples, and realistic job previews) assess performance and aptitude on particular tasks. NB CAD & BIM skills assessments fall into this category;

- Medical inquiries and physical examinations, including psychological tests, assess physical or mental health;

- Personality tests and integrity tests assess the degree to which a person has certain traits or dispositions (e.g. dependability, cooperativeness, safety) or aim to predict the likelihood that a person will engage in certain conduct (e.g. theft, absenteeism);

- Criminal background checks provide information on arrest and conviction history;

- Credit checks provide information on credit and financial history;

- Performance appraisals reflect a supervisor’s assessment of an individual’s performance; and

- English proficiency tests determine English fluency.

An important item to remember when interpreting assessment scores is to put the results in context, and to compare them to an external performance benchmark. For example, a 58% score does not reflect ‘failure’. If this score is presented by a CAD user against an office average of 60%, it is likely the candidate will comfortably fit in to the team.

Test questions should also have a recognizable skill level; basic, intermediate or advanced. In this way, the benchmark data can be reliably used to compare performance and make sound hiring decisions.

In conclusion, for a CAD or BIM skills test to be valid, it must contain content that reflects a representative sample of the target skill. A reliable skills test should comprise sample job tasks (e.g. performance tests, simulations, work samples, and realistic job previews) and assess performance and aptitude on particular tasks.

Skills evaluations should not discriminate according to; race, colour, national origin, sex, religion, age (40 or older), or disability.

Lastly, employers should ensure that employment tests and other selection procedures are properly validated for the positions and purposes for which they are used. The test or selection procedure must be job-related and its results appropriate for the employer's purpose.

R

Business Benefits of Benchmarking

Benchmarking is the process of comparing the cost, time or quality of what one organization does against what another organization does. The result is often a business case for making changes in order to make improvements. Also referred to as "best practice benchmarking" or "process benchmarking", it is a process used in management where organizations evaluate various aspects of their processes in relation to best practice, usually within their own sector. This then allows organizations to develop plans on how to make improvements or adopt best practice, with the aim of increasing performance.

The most prominent methodology is the 12 stage approach by Robert Camp (who literally wrote the book on benchmarking in 1989). It consists of;

1. Select subject 2. Define the process 3. Identify potential partners 4. Identify data sources 5. Collect data and select partners 6. Determine the gap 7. Establish process differences 8. Target future performance 9. Communicate 10. Adjust goal 11. Implement 12. Review / recalibrate.

Benchmarking can take various guises:

Process benchmarking - a firm focuses its observation and investigation of business processes with a goal of identifying and observing the best practices from one or more benchmark firms. Activity analysis will be required where the objective is to benchmark cost and efficiency.

Financial benchmarking - a company performs a financial analysis and compares the results in an effort to assess overall competitiveness.

Performance benchmarking - allows a firm to assess their competitive position by comparing products and services with those of target firms.

Product benchmarking - the process of designing new products or upgrades to current ones. This process can sometimes involve reverse engineering competitors’ products to find strengths and weaknesses.

Strategic benchmarking - involves observing how others compete. This type is usually not industry specific, meaning it is best to look at other industries.

Functional benchmarking - a company will focus its benchmarking on a single function in order to improve the operation of that particular function, i.e. Human Resources, Finance and ICT.

Internal benchmarking - involves benchmarking businesses or operations from within the same organisation (e.g. business units in different countries).

External benchmarking - analysing outside organisations that are known to be best in class provides opportunities of learning from those who are at the ‘leading edge’.

International benchmarking - best practitioners are identified and analysed elsewhere in the world; globalisation and advances in information technology are increasing opportunities for international projects.

Benchmarking involves four key steps:

1) Understand in detail existing business processes

2) Analyse the business processes of others

3) Compare own business performance with that of others

4) Implement the steps necessary to close the performance gap

Benchmarking should not be considered a one-off exercise. To be effective, it must become an ongoing, integral part of an ongoing improvement process with the goal of keeping abreast of ever-improving best practice.

Why Bother?

There are many benefits of benchmarking; the following list summarises the main ones:

• provides realistic and achievable targets

• prevents companies from being industry led

• challenges operational complacency

• encourages continuous improvement

• allows employees to visualise improvement which can be a strong motivator for change

• creates a sense of urgency for improvement

• confirms the belief that there is a need for change

• helps to identify weak areas and indicates what needs to be done to improve.

So how does all this apply to us - what are the benefits of benchmarking CAD & BIM performance? In summary:

• Gain visibility of core CAD & BIM skills

• Identify individuals’ strengths & weaknesses

• Implement better CAD & BIM training plans for staff

• Improve recruitment processes

• Share performance data across an organisation

• Promote collaborative working between teams & offices

• Measure the performance of outsourcing or off-shoring partners

• Provide ‘best practice’ for CAD & BIM development

• Offer clearer staff inductions and more meaningful staff appraisals

• Enjoy better skills resourcing for projects

• Develop a continuous improvement process for CAD & BIM

• Save time and money on construction projects and offer best value for clients

In the end, it all comes down to better visibility of your teams and their real ability to use often complex technology and tools for maximum effect. If you can't measure it, you can't manage it.

R

UK Construction - The Impact of Recession (so far)

During the summer, I was invited to present to a group of AEC professionals about the impact of the recession on the UK Construction Industry. Whilst still a wee bit premature to be looking back with the benefit of hindsight - let's face it, we're not out of the woods just yet! - it is possible to gather some useful and interesting data on what has been happening during the past 3 years of economic downturn.

First, the basics; a recession is a significant decline in economic activity spread across the economy, lasting more than a few months, normally visible in production, employment, real income, and other indicators. A recession begins when the economy reaches a peak of activity and ends when the economy reaches its trough.

The first of 6 successive quarterly falls in UK GDP started in Q2 2008. The commonly accepted indicator that a recession has begun used to be, “two consecutive quarters of negative growth in GDP”. However, it's not quite so simple!

According to the National Bureau of Economic Research – UK (the official arbiter of recessions) the current recession began as early as December 2007.

5 key measurements NBER UK points to, with respect to a recession:

- GDP (a measure of a country's economic activity, namely of all the services and goods produced in a year)
- Retail/wholesale sales figures
- Overall employment
- Personal income
- Industrial production

Meanwhile, the Business Cycle Dating Committee of the National Bureau of Economic Research (NBER) – USA, maintains a chronology of the beginning and ending dates (months and quarters) of U.S. recessions. The committee determined that a peak in economic activity occurred in the U.S. economy in December 2007.

So whilst many firms have reported that the UK economy really went into a nose-dive in Q4 2008, it was actually nearly a year earlier that the problems began!

Some interesting data was quoted in a Building Magazine editorial, back in June 2010;

Experian predicts a slight upturn for 2011 and 2012 – depending on recovery in the private sector and no more than £5bn cuts in the public sector. (RV - hmmm not sure that they could foresee the current round of cuts when that was written!)

FT review of Office of National Statistics data found the following unemployment figures:

• 720% increase for architects
• 655% increase for QS’s
• Similar figures for Planners, Construction Managers & Engineers (500%+)

Earlier in the year, the Chartered Institute of Building (CIOB) published the results of their third annual survey, where they polled opinion from 1182 UK construction firms.

• Results show that the industry is still suffering a skills shortage despite the recession and downturn in construction demand.
• 77% of respondents believe there is a skills shortage in construction and 78% of those feel that the loss of skills will hinder the industry’s recovery when the economy improves.
• 54% state that their company has had to make redundancies and 14% expect further redundancies to occur.
  
  

Similar comments echoed by Construction Skills, who observed; 'Forecasts demonstrate the severe effect that the recession is having on the UK construction industry, with rising job losses risking huge skills deficits in the long term.

The Construction Skills Network (CSN) shows the prognosis for the UK construction industry in the short term (2009 to 2010) is poor. The industry is expected to contract in construction output terms by at least 12% in 2009, followed by a more marginal decline of around 2-4% in 2010.

Consistent recovery is not forecasted until 2011 and even then, it is likely to be a slow and steady return to moderate levels of growth as confidence returns to the market.'

Construction News published the following figures;

• UK Construction employment began to fall in 2008 with a decline of 1% over the year, however, CSN forecasts a much larger drop of 15% across 2009 to 2010, with the largest losses (around 13%) expected in 2009. The total number of construction job losses from 2009 to 2010 could be up to 450,000 if output contracts by the suggested higher rate of 13% in 2009. The slow recovery forecast from 2011 to 2014 is set to create around 125,000 jobs.
  
  
• Coupled with the large number of workers who are set to leave the industry in the next 10 years through retirement, the need to increase the flow of apprentices and graduates into the industry and retain as many existing skilled workers as possible is vital if the UK is to sustain a skilled construction workforce.

Data from the CIOB survey, noted the following points about Apprentices & Graduates;

• Only 37% are sure their companies are still employing apprentices.
• 11% stated that their companies usually employ apprentices, but cannot afford to in the current economic climate.
• 12% of respondents are aware of their companies recruiting more graduates.
• Only 1% are recruiting the same number of graduates as before the recession.

So what are the lessons we can take from all of this? What mistakes have firms made and what are the opportunities for growth when things pick up?

Mistakes:

• Panic firing/firing the wrong people, i.e. firing the best software users or even the CAD/BIM Manager!
• Hoping things will get better sooner; waiting too long to make key strategic decisions
• Freeze on technology investment
• Freeze on all hiring
• Freeze on all training & learning budgets
• Lack of cash flow management

Opportunities:

• Removal of ineffective staff (who perhaps shouldn’t have been hired in the first place?)
• Better users remaining
• More time to spend on IT strategy & ‘housekeeping’
• Better hiring practices in future
• Better training & learning plans in future
• Better cash flow procedures
• Better, more realistic business plans
• More focused roadmaps for technology, marketing and business strategy
• Better relationships with remaining clients
• Leaner working practices
• More enthusiastic adoption of BIM

A final word from an article which appeared in the FT in the summer; “Those businesses that survive a recession of this magnitude, are the ones who are best placed to ride the wave back up again!”.

R

UK BIM Roundtable

Recent months have seen a plethora of forums and conferences about this year's numero uno hot potato, namely, BIM. I've attended a number of these sessions and, whilst the networking is always useful, the content has varied tremendously. So many meetings still focus on the big picture of BIM, almost selling the idea that BIM is the future and we all need to get on board. I think, for most firms, they've already made that leap. They kind of get it already, that we can all do better in terms of delivering more efficient projects and harnessing the latest technology to make it happen.

The question now is, how exactly do we, as individual firms and collectively an industry, sally forth and turn all the rhetoric into BIM-reality? If you speak to the marketing teams of many UK design & engineering firms, the story is that BIM is altogether sorted, that they are 100% BIM proficient and can deliver projects in any technology, at any time, with maximum efficiency, no fuss and zero downtime! :) Talk to the operations teams, who are tasked with actually making this all happen, and the story isn't quite the same. And this is in no way a criticism of these hard-working people. Quite the opposite. But, ironically, when firms speak openly about this stuff, the reality is that they are all wrestling with the same challenges!

To this end, we've been working with a group of connected people this year, to put together a programme of round table meetings, whose sole aim is to provide detailed answers to some key questions about BIM - at a practical level. The nice thing about this initiative is that no-one is forcing an agenda onto the proceedings. Whilst we all have a vested interest in the UK Construction industry picking itself up off the floor, the rest is all about using whatever connections or influence we may have individually, and pooling our resources to bring together the right people, at the same time, to figure out some new answers to these questions.

This Thursday in London, we have the first UK BIM Roundtable. All being well, we anticipate that this could develop into a regular series of half-yearly, or even quarterly, forums, each addressing a different area of BIM.

The first meeting focuses on the dynamics of the relationship between Architects and Engineers and General Contractors. Discussion will be based around the following topics:

• What are the needs and requirements of consultants and what are the needs and requirements of contractors, when undertaking a BIM project?

• How can we negate conflicts occurring at the interface of a design-based BIM as it undergoes the transition to the construction phase?

• Getting BIM deliverables in sync throughout the project; exploring commonly accepted workflows and identifying the requirements of each discipline.

The meeting is by invitation and all 30 available places were filled very quickly. The following firms will have representation at the session;

Consultants: AECOM, WSP, Buro Happold, Halcrow, Hoare Lea, Capita Architecture, Zaha Hadid, Atkins, Levitt Bernstein, Foster + Partners, Davis Langdon, Waterman, IBI Nightingale, HOK, Tribal, Jacobs, Ramboll UK, Scott Wilson, BDP.

Contractors: Skanska, TPS Carillion, Carillion, Bovis Lend Lease, Mace, Laing O'Rourke, Costain, BAM, Vinci, Robert McAlpine, plus UKCG.

We also anticipate having people from TfL, Crossrail and BIS in attendance.

This is designed to be an interactive discussion forum. There will be an agenda and specific objectives for the session, steered by co-chairs representing both groups, but the rest is up to the delegates. We have no commercial sponsors, nor formal presentations from pre-arranged speakers. The minutes of the roundtable will be documented and issued in the form of a short report following the meeting. Thanks to Buro Happold for providing the venue for this meeting.

R