Situational Judgement Tests vs. ShadowBox: Surface similarities mask essential differences
Have you ever wondered how ShadowBox's scenario-based training method differs from other training and evaluation approaches? In this week's featured insight, Gary Klein compares and contrasts ShadowBox training to Situational Judgment Tests, highlighting key distinctions that put ShadowBox in a league of its own.
Situational Judgement Tests (SJTs) and ShadowBox exercises are both scenario-based methods for testing and evaluating people.
SJTs have been around for over 50 years and have an impressive track record of helping organizations make personnel selection decisions. SJTs give job candidates a brief scenario describing a choice to be made, followed by a multiple choice set of options about what to do. The options have been evaluated by subject-matter experts (SMEs) so that the candidates can be assessed on how closely their response choices match those of the experts.
I am sometimes asked how the ShadowBox approach my colleagues and I have developed is different from SJTs. ShadowBox also uses scenarios, usually with multiple choice options, and vetted by SMEs. Sounds like the same thing with a new name.
However, closer inspection shows a number of critical differences.
Objectives. SJTs are designed to help organizations select job applicants, and I respect the effectiveness of SJTs in this regard. ShadowBox scenarios are designed for training, primarily cognitive skills training.
ShadowBox can be used for a variety of purposes such as knowledge management, leadership development, and even personnel evaluation, and I expect that SJTs can similarly be adapted for different uses. However, the primary use of ShadowBox is for cognitive training, just as the primary use of SJTs is for personnel selection. I imagine that SJTs could be repurposed to provide training, but I haven’t seen anything in the literature describing the use of SJTs for training.
The concept of cognitive skills training emphasizes ways to help trainees build more powerful mental models, shift to more effective mindsets, and gain skills for better decision making and sensemaking. The ShadowBox scenarios are specifically designed for these kinds of outcomes. ShadowBox evolved from an earlier technique called Tactical Decision Games that have been widely used by the US Marine Corps for training decision making abilities.
Scenarios. For several reasons, including efficiency, the SJT scenarios tend to be brief and to the point, whereas ShadowBox scenarios provide the background, the story of how the incident unfolded, and important contextual features of the situation, sometimes even including some branching.
The short SJT scenarios present a decision point with several possible options for which course of action to select. ShadowBox scenarios cover several decision points, usually 4-7 of them. More important, the ShadowBox decision points are not only choices about courses of action. They can cover decisions about which goal to prioritize, which types of information to seek, which concerns to be most worried about, and which interpretation of the ambiguous situation to adopt. The ShadowBox scenarios support cognitive skill development by capturing all of these issues.
SJT response options are designed so that there are no clearly wrong choices. Because of its emphasis on training, ShadowBox response options include specially designed foils that will be likely to be chosen by trainees who have flawed mental models, as a way to help the trainees discover what is wrong with their thinking.
Further, ShadowBox approaches are not restricted to scenarios. ShadowBox Lite relies on statements and lets trainees compare their reactions, what they agree with and disagree with, to the reactions of experts. And the Expert-Eyes (Cue Detect) version of ShadowBox uses short videos along with a patented method for the trainees to spotlight the specific cues that are of interest.
Debrief. The process of debriefing is central to ShadowBox training, whereas it is not relevant to the SJT personnel selection process. ShadowBox debriefs are intended to help trainees restructure their thinking. Therefore, trainees have to describe the rationale behind their choices and get to compare their rationale statements with those of the experts.
ShadowBox also allows for a minority view, because it is not intended to teach a right answer. There is value in showing how the SMEs might disagree, and letting the trainees study the contrasting rationale statements provided by the SMEs. The ShadowBox trainers don’t encourage the trainees to uncritically accept the SME choices and rationale statements, but the trainees do have to study these statements as a way to expand their mental models.
With an SJT, the expert answers are intended to help the evaluator in making a selection. The candidate doesn’t get any feedback on individual questions. In a ShadowBox scenario, the expert answers are intended to help the trainee to learn. They are not for the benefit of the trainer. The orientations are opposite.
Presentation. Both SJTs and ShadowBox materials can be offered as hardcopy booklets to be filled out in a classroom, and both can be offered online. ShadowBox scenarios have dedicated software systems, but are also presented using off-the-shelf systems such as Articulate and its Storyline mode.
Further, the online ShadowBox version includes a facilitator mode to allow trainers who are not co-located with the trainees to gather their information and conduct meaningful debriefs. The online ShadowBox version can be used for synchronous training sessions, especially with the facilitator mode, but also asynchronously by providing trainees with the SMEs ranking and rationale statements.
In conclusion, the SJT and ShadowBox approaches both seem to have great value. Each is designed for a different use — personnel selection and cognitive skills training. Although they share a number of surface similarities, the underlying objective of the two approaches is quite different and the design of the methods is accordingly quite distinct.