|EXPERT STATUS REPORT
|Year : 2014 | Volume
| Issue : 3 | Page : 123-125
Test blueprinting for multiple choice questions exams
Imran Siddiqui, James Ware
Department of Medical Education and Postgraduate Studies, Saudi Commission for Health Specialties, Riyadh, Saudi Arabia
|Date of Web Publication||31-Jul-2014|
Department of Medical Education and Postgraduate Studies, Saudi Commission for Health Specialties, Riyadh
|How to cite this article:|
Siddiqui I, Ware J. Test blueprinting for multiple choice questions exams. J Health Spec 2014;2:123-5
The test development process includes an overall plan, content definition and test specification. The overall plan of an assessment will then include the identification of the target group, and the purpose of the exam such as selection, classification, placement, diagnosis or certification. Furthermore, the details of the test design, assembly and production, administration, scoring, standard setting, reporting results, item banking, post hoc analysis and logistic requirements must all be decided.  Once the purpose of the test is identified then the most crucial step in the development of a test is to ensure that the test should measure what it is supposed to measure, known as the test validity. Constructing a test blueprint enables stakeholders to have a bigger picture of the exam and determine relative weights of individual content. 
There are a number of important uses for a test blueprint, such as:
- Reduce validity threats which occur due under or oversampling bias of the subject matter
- Target item writers to give a clear message in which content area they need to write items to assess specific learning outcomes at the prescribed level of complexity
- Guide examinees to focus on the important areas of the subject matter. Since the test blueprint is a tool that ensures content validity of the exam, the examinees will get a clear message that their exam is balanced
- Define the acceptable evidence to infer the areas of the subject matter
- Give constructive feedback to examinees and faculty
The blueprint is a general term that means an outline of the detailed plan or a plan of action. Originally, this term was introduced in the late 19 th century in architecture for structural designs on prints composed of white lines on a blue background or blue lines on a white background before building a house.  Test blueprints which are sometimes called the table of content specifications, define the assessment instrument of the exam items allocated to each learning objective or content, along with the specific item character (e.g., with or without a scenario) given a certain weight to be included in the exam. It serves as a tool to align learning objectives with assessment items of the subject matter. 
The test blueprint could be developed in three forms, (a) the overall high level blueprint in which graduate outcomes are laid down against assessment tools, (b) the year or phase blueprint which is designed to assess learning outcomes of a year or phase and hence suitable for promotion or graduation exams, (c) while the "single examination" blueprint is for licensing exams. The test blueprint should be constructed by a small group of educators or trainers who are directly involved in the teaching or training during the programme but ideally the people who constructed the programme should propose the test blueprint.
A basic test blueprint is made up of a table called the matrix. It is a two-dimensional matrix chart, with an X and Y axis, which describes the sample of items to be selected in the test. The preferable format uses a spreadsheet which allows easy calculations and fine tuning of the proportions or number of items for an exam. Learning targets that could be termed as sections are laid down in rows, section headings are entered on the Y axis, and the columns are used to write domains which are entered on the X axis with specific tasks across each of the sections [Table 1]. The learning targets or sections of a programme or course will depend upon its specific nature and philosophy of the subject matter. These are usually driven by the curricular content, learning objectives or in some cases from pre-determined common clinical presentations or problems.
The next thing to do is to make a list of tasks from the learning outcomes, which would be expected to be achieved by the examinees. These tasks or learning outcomes could be termed as domains. Despite the variations in courses, subject matter and learning objectives, the domains may be common to many different blueprints, like recall or application of knowledge in basic sciences, appreciation of pathophysiology, identification of appropriate investigations, making the correct diagnosis, managing the case appropriately, and so on. The correct identification of domains is as important as selecting the correct sections, therefore, it is usually desirable to make two separate lists of these two because examinees may need to react in the same way (make a diagnosis) to different sections of the subject matter and he or she may need to behave in different ways (diagnose, manage or order investigations) to the same section of the subject matter.  For instance, an examinee may show poor performance in the pathophysiology of various diseases or may show poor performance in the whole of one section.
One has to note that the content and learning objectives should be written down in greater detail for teaching purposes, but in designing the test blueprint they should be grouped in major categories. While creating the test blueprint it is preferable to have 8 - 14 sections and 3 - 7 domains. The simple way to work out the correct number of sections and domains is to first decide the number of test items in the exam. Ideally, the number of cells in the test blueprint which are created in the spreadsheet of the test blueprint should not exceed 70% of the proposed number of items for that exam. This means that if an exam paper shall contain 100 items and the number of sections is 15, the number of domains should be approximately four giving the total number of cells in the blueprint matrix of 60, 60% of the total number of questions required for the exam. This will allow an appropriate representation of the content as compared to the test blueprint having 25 sections and six domains which might result in the production of an exam with no representation of the various subject matter areas.
The next step should be assigning the relative weight to each subject matter sections. The number of questions in each cell of the spreadsheet could be determined by a number of factors such as how important each area is and how much time was devoted to each area during instruction. In medicine, one way of assigning relative weight in cells of the spreadsheet is by considering the importance of the area in terms of impact and frequency of related clinical conditions. For instance, the diagnosis of ischaemic heart disease or diabetes mellitus should be assigned more weight than the diagnosis of Gaucher's disease or Takayasu's arteritis. It is important to note that the decision about the total number of items in the exam should be made prior to assigning the relative weights in the cells. It has been reported that to ensure a higher reliability of the exam the number of items should be at least 80 - 100 with an average discrimination of > 0.3. 
|Table 1: Part of a test blueprint, as an example with 3 sections and four domains. In each cell the proportions of items testing low and high cognition are given, K1:K2 |
Click here to view
Often all six levels of Bloom's taxonomy have been used for test blueprints to distinguish different levels of cognition.  However, this is too complex and experts seldom agree on which level of the taxonomy to assign an item. Therefore, we have called the lowest two levels of Bloom's taxonomy, recall of memorised information and its comprehension, K1. While, higher levels of cognition, including interpretation, analysis, evaluation and the application of knowledge are called K2. If the introduction of a level of complexity in the exam is desirable, then it would be advisable to introduce the levels of cognition (K1 and K2) into the blueprint. The decision about how many K1 or K2 questions should make up an exam could be a general decision for the whole exam, and given as a ratio, such as 30:70, K1:K2 respectively. But it would be more appropriate to assign the desired number of K1 and K2 in each cell of the test blueprint matrix. It should be noted that exams in medicine and allied health sciences should have a higher proportion of K2 items where decisions about fitness to practice will be made. [Table 1] shows an example, from a portion of a test blueprint, with its essential components.
If the test blueprint is correctly aligned with the various areas of the subject matter, the remaining part will be the alignment of the test items with the blueprint. To do this, each and every item has to be classified according to the test blueprint. If there is an existing question bank, those assessment items that match the blueprint can be retrieved, and where there is no question available, a clear instruction could be given to the item writers about how many extra items they should write in a specific area of the subject matter.
In conclusion, constructing a test blueprint includes the following steps:
- Identify the content areas of the subject matter (sections) to be measured by the exam
- Identify the learning outcomes (domains) across each section to be measured by the exam
- Weight the sections and domains in terms of their relative importance
- Construct a spreadsheet in accordance with relative weights by distributing the test items
- Allocate the relevant cells of the spreadsheet with their level of cognition
| References|| |
|1.||Hochlehnert A, Brass K, Möltner A, Schultz JH, Norcini J, Tekian A, et al. Good exams made easy: The item management system for multiple examination formats. BMC Med Educ 2012;12:63. |
|2.||Downing SM. Twelve Steps for Effective Test Development. In Downing SM and Haladyna TM. Eds.). Handbook of test development. Mahwah, NJ: Lawrence Erlbaum Associates 2006; p. 3-25. |
|3.||"blueprint" Shorter English Dictionary. 6 th ed. Oxford University Press; 2007. |
|4.||Grönlund NE. Assessment of student achievement. Allyn & Bacon Publishing, Longwood Division, MA, USA, 1998. |
|5.||Ebel RL. Measuring educational achievement. Englewood Cliffs, NJ: Prentice-Hall, 1965. |
|6.||Bloom BS. Taxonomy of educational objectives. In: Cognitive domain. Vol. 1. New York: McKay; 1956. |