As someone currently in the midst of making vacation scheme applications, I can’t help but wonder why different firms choose such different application processes. Whilst some firms, such as Slaughter and May, use only a CV and cover letter, others, such as Herbert Smith Freehills, require applicants to pass an online test before they will even look at their application forms. However, one form of assessment that is fairly common across the legal sector is the Watson Glaser test. Not only is it used by several large solicitors’ firms, including Clifford Chance and Hogan Lovells, but it also forms the basis of the Bar Course Aptitude Test used as part of the application process for bar courses.

On one level, this makes perfect sense. The Watson Glaser test, which includes sections on identifying whether a conclusion follows logically from a passage, grading the probability that a conclusion is true given the contents of a passage, deciding whether a generalisation is justified, deciding whether a particular assumption is made within a text and deciding whether an argument provides support for a particular conclusion, is designed to assess critical thinking skills. These skills are directly relevant to work as a lawyer, which often involves assessing arguments critically and sceptically. Moreover, from a firm’s perspective, using an online or written test to screen applicants dramatically reduces the number of hours that HR teams must spend wading through CVs and cover letters or administering assessment days.

However, some have argued that the Watson Glaser test isn’t quite as useful as it might seem. The logician Kevin Possin points out that it does not even attempt to test some fairly crucial elements of critical thinking. It doesn’t include arguments by analogy, nor does it test people’s ability to see past “informal fallacies” where a flawed argument is disguised by rhetoric (for example, where an argument relies on the fact that a word has a double meaning). He also claims that people with greater prior knowledge of critical thinking may be less likely to perform well in the test, as they will be more confused by instructions that don’t make it clear whether arguments are supposed to be assessed by the standards of inductive or deductive reasoning. Perhaps the worst problem, however, is that four of the Watson Glaser test’s five sections involve choosing one of just two multiple choice answers, meaning that there’s a very high probability of randomly guessing correct answers.

This is not to say that the Watson Glaser test is useless. In fact, as TalentLens (which administers the BCAT) are keen to point out, it is a surprisingly good predictor of performance in the Bar Professional Training Course. Those who achieved an Outstanding result in the BPTC achieved an average of 67.6 in the BCAT, whilst those who achieved the lowest grade, “Referred”, scored an average of 43.2. This makes it a better predictor of course success than A-level points or degree class.

However, it is perhaps interesting to consider the strengths and weaknesses of the tests used in different areas of the legal industry, especially as more and more firms move away from the Watson Glaser test and towards Situational Judgement Tests.