CFA Exam Question Format Changed FAQs

The CFA Institute has announced the following...

"Format Change for 2009 CFA Exams
CFA Institute is committed to maintaining and improving the standards and consistency of the CFA exams. Research and our own experience indicate that the fourth answer option on multiple choice and item set exams is unnecessary to assess a candidate’s knowledge and skills. Three answer choices are sufficient and effective in discriminating between those candidates that possess the knowledge and skills and those that do not. As a result, we are changing the format of the multiple-choice and item set questions on CFA exams from four answer options to three. "

THey have listed the following FAQs to explain the policy
Exam Question Format FAQs

Won’t candidates taking exams in 2009 and later have a statistical advantage over those who took the exams earlier?
There is no fixed passing score on CFA exams. If there were fixed passing scores, all other factors equal, a reduction in the number of answer choices would likely result in more candidates passing the exams. That is not the situation. The passing score of any CFA exam is directly related to its level of difficulty as assessed by CFA charterholders in a formal standard setting process.

The CFA Institute Board of Governors establishes the minimum pass score for each exam. The board relies heavily on the widely accepted modified Angoff standard setting method in evaluating each exam. Standard setting, independently managed and directed by expert psychometricians, is a rigorous process conducted after an exam is administered. Each standard setting session brings a diverse group of approximately 30 charterholders together for 1-2 days to independently evaluate each question and judge whether a minimally qualified candidate would answer it correctly. The goal of standard setting is to recommend to the board a minimum passing score that appropriately distinguishes between those candidates that demonstrated mastery of the curriculum from those that did not. The difficulty of any question is considered by each standard setter in making his/her judgment about the expected performance of the minimally qualified candidate. Plausibility of distractors is a factor considered by standard setters regardless of the number of answer choices. This process considers any potential changes in exam difficulty that result from a reduction in the number of answer choices, and factors this in to the recommended minimum passing score. The standard setting process is an important step in maintaining consistency and fairness in evaluating candidate performance across years.

Further, it is true that for an unprepared candidate in the CFA Program, the probability of successful guessing on any single question will increase. However, the probability that such a candidate could guess correctly enough times to pass the exam is mathematically infinitesimal. This is true whether there are two, three, four or more choices. It is also at least theoretically possible that a candidate with enough knowledge and skills to achieve a reasonably high score might achieve a higher score through successful guessing on just a few questions. Again, it is important to note that there is not a fixed score that candidates need to achieve to pass a CFA exam. Standard setters evaluate the difficulty of each exam in making their minimum pass score recommendation.

It is also worth noting that each of us likely made some educated guesses on our CFA exams. That is in part a function of a rigorous exam that demands mastery of many different topics and that will remain the case. The content of the questions will not be any less rigorous with three-choice questions. The fact that guessing is mathematically better rewarded does not mean that pass rates will be higher or that unqualified candidates will pass the CFA exams. The major reasons for this are the rigor of the questions, the quality of the answer choices, and standard setting.

Why is CFA Institute making the change from questions with four answer choices to questions with three choices?
It is an improvement we have been considering for many years. We have followed the research of respected psychometricians in this field and have consulted with them on the benefits of making this change. In evaluating CFA exams we found that the fourth choice on the majority of selected response questions (multiple choice and item set) is frequently ineffective – attracting relatively few candidates. High-quality questions do not require four answer choices and in fact the fourth choice often performs poorly.

It is worth remembering the last time CFA Institute made a major change in the structure of the exams. In 2000, we began to phase in item-set questions. Some members felt very strongly that the change to item sets represented a certain “dumbing down” of the exams. Item sets are short (approximately 1-2 pages) case studies followed by six selected response questions. Candidates must analyze the information provided in the case to determine the correct answers. Item set questions were gradually phased in to the Level II and III CFA exams. Since 2005, Level II exams have been comprised entirely of item set questions. Half of the Level III exam is comprised of item sets. All evidence points to no degradation in the quality and rigor of CFA exams. Candidate performance on item sets is comparable with performance on constructed response (e.g., short essay) questions. Candidates who sit for these exams and members who review item set questions consistently believe these questions are as difficult or more difficult than constructed response questions. Last, but not least, pass rates indicate the CFA exams have not become easier with the conversion to item set questions. Item sets are an effective method of testing higher order knowledge and skills, including problem-solving and interpretation. Similarly, the change from four to three answer choices will not degrade the quality, content, and rigor of the questions.

How can you assert that the quality of the exam will be as good as or better than exams that utilized four-choice questions?
Put yourself in the position of the test taker. A question with a correct answer and two plausible but incorrect choices is a challenging question. An exam comprised of such questions is a challenging exam. Remember that most four-choice questions include at least one choice that does not appear plausible to candidates and is not often selected by candidates. CFA Institute has decided to focus our efforts on ensuring that every question has two plausible incorrect answers rather that spending the effort to write a third incorrect answer that, despite best intentions, is often not seen as plausible. In addition, the empirical literature in test development reveals minute to often undetectable differences in exam quality when two incorrect choices are offered rather than three. All things considered, we believe that this move will result in a higher quality exam that is equally effective in differentiating between mastery and non-mastery of the material.

What is a plausible incorrect answer?
A plausible incorrect answer is one which cannot be dismissed without careful thought or calculation. It should appear to be a viable answer to a candidate who has not mastered the knowledge and skills taught in the curriculum. For example, on a calculation question, incorrect choices might be the result of calculations that mix up the variables in a formula or represent interim calculations in a situation where two or more operations are required. Incorrect alternatives are often written in accordance with popular misconceptions about a particular topic. After a careful reading of the question and appropriate assessment of the possible answer choices, a well-prepared candidate should not find plausible incorrect answers to be attractive.
Won’t the exams be easier because candidates have fewer answer choices to read and consider?
Although candidates need to carefully manage their time to perform well on CFA exams, the CFA exams are not primarily designed as a test of time management skills. They are designed to comprise a fair sample of the knowledge and skills taught in the CFA Program curriculum. The exams provide a framework in which candidates can demonstrate mastery of the knowledge and skills. You mentioned that research of respected psychometricians supports the decision of CFA Institute to use three-choice questions. Can you give me some examples?
We considered educational research which questions the value of developing more than two distractors (incorrect choices) for multiple-choice items. Dr. Thomas Haladyna and Dr. Steve Downing, two recognized leaders in testing research, argue that most items used in well-developed standardized tests contained only one or two good distractors, suggesting that item writers should focus their efforts on the quality, not the quantity, of the distractors. For discussions, see Writing Test Items to Evaluate Higher Order Thinking (Haladyna) and Handbook of Test Development (Haladyna and Downing). In addition, a 2005 article by Dr. Michael Rodriguez in Educational Measurement: Issues and Practice summarizing 80 years of empirical evidence clearly supports the use of three-option (two-distractor) test items. We consulted other testing experts as well. "