Board of Bar Examiners of the Supreme Court of Delaware
Beginning in 1998, the Board of Bar Examiners (the "Board") of the Delaware Supreme Court (the "Court"), in consultation with experts from the National Conference of Bar Examiners (the "NCBE") and bar examination testing experts, undertook a study of the Delaware Bar Examination. The object of the review was to, among other things: (a) enhance the ability of the Board, through administration of the Delaware Bar Examination, to identify fairly, accurately and effectively those applicants who meet the threshold requirements set by the Court for admission to its Bar; (b) maintain grading procedures that are both consistent and fair to all tested applicants; and (c) promote fairness and objectivity to all applicants in the Board's evaluation of competency.
Following this review in 1998 and 1999, the Board recommended to the Court, and the Court approved in 2000, the following changes to the Delaware Bar Examination, starting with the July 2000 Bar Examination: (a) the Delaware law essay questions were reduced from twelve to eight 45-minute questions; (b) two 90-minute Multi-state Performance Test (the "MPT") essays were added; (c) the three parts of the Delaware Bar Examination (the Multi-state Bar Examination (the "MBE"), the Delaware law essays and the MPT) were weighted 40% MBE, and 60% essay and MPT; (d) the written portions of the Delaware Bar Examination were scaled using the standard deviation method; and (e) a combined score of 145.00 was set as the minimum score required to pass the Delaware Bar Examination.
Along with these administrative and substantive changes, the Board formally adopted grading and scoring procedures to promote standardization and accuracy in grading essays and calculating scores. These changes, used by the Board beginning with the administration of the July 2000 Delaware Bar Examination, are explained below in detail.
II. Essay Drafting and Grading Procedures
In 2000, the Board formalized its essay drafting and grading procedures as set forth below:
A. Drafting Question and Outline of Answer.
1. Assignment of Topics and Designation of Two-Person Drafting Partnerships.
Early in each calendar year, eight Board Members (the "Essay Committee" of the Board), along with their respective Associate Members, are assigned primary responsibility for drafting an examination question and an outline of an answer and point allocations on a topic selected by the Board from among those topics specified in Rule 12 of the Board of Bar Examiners Rules. At the same time that essay question topics are assigned, drafting partnerships are also designated, each consisting of two Board Members from the Essay Committee. Each draft essay question and answer outline is thereafter prepared and carefully reviewed by the assigned Board Member and his or her Associate Member in accordance with the guidelines set forth by the NCBE.
The drafts are then exchanged with the other member of the assigned drafting partnership for review and comment. The purpose of this review is to provide additional objective input and, more specifically, to assure that the question:
- fairly raises and calls for discussion of the issues expected to be addressed by the examinee's responses, consistent with the accompanying answer outline;
- avoids unintended ambiguities and is otherwise appropriate in subject and form;
- does not focus on minutia or unduly reward rote memorization of readily available rules or statutes rather than the quality of the examinee's analytical skills and legal understanding; and
- does not call for applicants to rely on a recent precedent or statutory amendment to answer the question correctly.
Based upon comments received from the drafting partner, the question and outline of the answer are then revised by the drafting Member, in consultation with his or her Associate Member, and the final versions are sent to the Board's Essay Review Committee (as defined below).
Although the MPT is drafted by the NCBE, the Board retains the grading function. Board Members who grade the MPT attend a national grading workshop sponsored by the NCBE on the weekend following administration of the Delaware Bar Examination. The Board Members grading the MPT follow the same grading procedures, explained in more detail below, that are used by Board Members who grade the Delaware law essays.
2. Essay Review Committee.
The essay review committee is composed of the Chair and/or Vice Chair and selected Members of the Board (the "Essay Review Committee"). The Essay Review Committee reviews each question and answer outline and proposes revisions as appropriate. The Essay Review Committee also reviews the draft question for clarity, length and propriety of subject matter. It also looks to make sure that the call of the question is correctly phrased in order to invite the answer expected by the author and to minimize ambiguities, irrelevancies or other perceived defects. This Committee also reviews the proposed answer outlines for completeness, accuracy, clarity and reasonableness of point allocation. Questions and answers are thereafter redistributed by the Essay Review Committee to the drafting partnerships for further comment from the author and revision in accordance with the revisions proposed by the Essay Review Committee. The resulting versions of the questions and answer outlines are thereafter submitted for final review and approval by the full Board. The Essay Review Committee then reviews the examination to ensure proper format and length of the examination as a whole as well as each of its essay sections.
1. General Grading Guidelines.
Essay questions are drafted to be graded on a 100-point scale, referencing the approved point allocation for each question. Any portion of an answer that fails to address the identified issue or that evidences no semblance of issue recognition, analysis or discussion receives no credit. Neither is credit given for discussion of irrelevant issues or statements of general principles of law that are not responsive to the question, even if correct. At the same time, however, no points are deducted because of the inclusion of such discussion or statements in an examinee's answer.
2. Grading Procedures.
The grading procedures are as follows:
a. Initial Grading Calibration Exercise.
Before final grading is undertaken, the Board Member who drafted the original question and his or her Associate Board Member engage in a calibration exercise, randomly selecting fifteen to twenty-five answer booklets and independently assigning grades to each of them. The Board Member and Associate Board Member thereafter compare the grades that each has awarded to these examinations and establish specific standards to govern the grading and resolve any divergence of process between them. A scoring sheet is then prepared that, where necessary, incorporates and accounts for issues properly but unexpectedly raised by examinees but which may not have been included in the original outline of the answer, that properly accounts for any unintended ambiguities in the question that the calibration exercise suggests may have been reasonably perceived by examinees, and that reflects such other equities as may be appropriate based upon this initial review.
b. Independent Grading by Board Member and Associate Member.
After the initial grading calibration exercise, the Board Member and Associate Board Member thereafter independently, and in reverse order, grade each essay answer. Upon completion of the independent grading by the Board Member and Associate Member, the Board Member and Associate Member compare the results each has reached with respect to each answer, discuss and resolve any significant discrepancies by regrading, if necessary, and reach agreement on a final grade for each applicant's examination. Thus, every answer is automatically graded twice and any differences between the grades assigned by the first and second readers are resolved. Those raw scores are then submitted to the Executive Director by anonymous examinee number. Prior to their submission to the Executive Director, all final grades are required to be reviewed carefully to avoid any possibility of inaccuracies in the completion of the form.
To maximize the integrity of the process, the Board does not regrade essays after the initial dual readings from the Board Member and Associate Member. Although regrading was selectively employed as a part of the Board's grading process prior to the July 2000 Delaware Bar Examination (employed specifically with respect to scores that fell within a limited range below the score necessary to pass the Examination), the Board has since concluded, based upon expert statistical and psychometric advice, that such a regrading process is inherently unreliable and that it effectively erodes the objectivity and reliability of the Examination.
The Board determined that its former regrading system should be eliminated because there already is effectively an automatic regrade as a result of having two readers per answer. In addition, the essay scores and the MPT and MBE scores for all applicants in the new process are interrelated in a scaled and weighted scoring process. As a result, any changes in "raw" essay scores caused by a regrading after scaling and combining could affect other scores and could change pass/fail determinations. Obviously, this would affect the consistency, validity and integrity of the entire grading process. Therefore, the Board recommended, and the Supreme Court approved, the elimination of any regrading.
III. Statistical and Accounting Experts Used for Scaling and Combining Raw Scores
Once the Executive Director receives the raw essay scores on the Delaware law essays and the MPT, the scores are transmitted to the Board's statistical expert, Gansk & Associates ("Gansk"). Gansk converts the raw scores on the eight Delaware law essay questions and the MPT to a common scale of measurement, computes the scale scores for the written and MBE portions of the exam, and weights and combines these scale scores to produce "total scale scores" in accordance with the procedures set forth in Board Rule 13:
An applicant shall be deemed to have passed the Bar Examination if, in a single administration of that Examination, the applicant achieves a "total scale score of 145.00 or higher." "Total scale scores" are computed as follows: (a) the raw scores on each of the eight essay questions are converted to a score distribution that has a mean of 50 and a standard deviation of 7, (b) the raw scores on each of the two MPT questions are converted to a score distribution that has a mean of 50 and a standard deviation of 14, (c) a given applicant's "total converted score" is the sum of that applicant's converted essay and converted MPT scores, (d) the distribution of the "total converted scores" is scaled to the MBE to yield a "total written scale score," and (e) the applicant's "total written scale score" is combined with that applicant's MBE scale score (with the written score weighted at 60% and the MBE score weighted at 40%) to produce a "total scale score." The term "scaled to the MBE" as used herein means that the distribution of "total converted scores" is scaled to a distribution that has the same mean and standard deviation as the MBE scale scores in Delaware.
Board Rule 13.
The results are then sent to the Board. To confirm that there are no mathematical errors in the scoring process, the Board sends the results and Gansk's calculations to a Delaware accounting firm. The accounting firm reports to the Board and confirms the results, or, if necessary, identifies any computational errors.
By way of example, the scaling and weighting formulas used for the 2003 Delaware Bar Examination are set forth as follows:
(1) Convert MBE raw scores (i.e., the number of questions answered correctly) to scale scores using the formula provided by NCBE/ACT.  The formula for the July 2003 Delaware Bar Examination was:
MBE Scale = (0.86 x Raw Score) + 31.93
(2) Convert the raw scores on each of the eight essay questions to a score distribution that has a mean of 50 and standard deviation of 7. The attached table shows the converted value for each reader assigned raw score on each question. For example, a reader assigned raw score of 40 corresponds to a converted score of 34.94 on Question #1, 41.38 on Question #2, etc. A new table is constructed for each exam.
(3) Convert the raw scores on each of the two Multi-state Performance Test (MPT) problems to a mean of 50 and a standard deviation of 14. The MPT problems are questions #9 and #10 on the attached conversion table.
(4) Compute the sum of the ten converted scores from Steps 2 and 3. Insert this sum into the formula below to obtain the written scale score:
Written Scale Score = (0.25 x Sum of ten converted scores) + 23.33
(5) Compute each applicant's Total Scale Score as follows:
Total Scale = (.4 x MBE Scale Score) + (.6 x Written Scale Score)
Numerical Example: Suppose an applicant had an MBE raw score of 120. According to the MBE scaling formula for the July 2003 Exam, this raw score converts to a value of 135.13 as follows:
MBE Scale = (0.86 x 120) + 31.93 = 135.13
The sum of this applicant's ten converted written scores was 500. According to the formula for scaling written raw scores on the July 2003 Exam, a total written converted score of 500 has a value of 148.33 as follows:
Written Scale Score = (0.25 x 500) +23.33 = 148.33
This applicant's Total Scale Score is therefore (.4 x 135.13) + (.6 x 148.33) = 143.05
Full Board Review
The Board thereafter meets to review, discuss and approve the Bar Examination results. Upon final approval, the scores are published to the applicants by anonymous number only. This information is released on the Board's website and posted at each of the three County law libraries as promptly as possible following approval of the scores. After the results have been posted by anonymous number only, Board Members go to the safe deposit box where the sealed envelopes containing the anonymous numbers with corresponding applicant names have been stored since the first day of the Bar Examination. Following retrieval of the envelopes, Board Members match the anonymous number with the applicant's name. Shortly thereafter, the names of the passing applicants are posted on the website. Passing applicants are advised of their success in writing. Failing applicants are advised in writing and provided with their scores on each of the various components of the Examination and are advised of their opportunity to secure copies of their essay answers and copies of representative answers to each essay question for comparison purposes. Copies of the essay questions are thereafter provided to law libraries throughout the State.
Elimination of Post-Announcement Review of Examination Scores
Prior to the adoption of the 2000 grading changes, a petition filed pursuant to then Board Rule 20 could result in a review of the petitioner's Examination scores. The Board studied this practice in conjunction with its 1998-1999 review of the Delaware Bar Examination procedures. After consulting with the NCBE and bar examination testing experts, the Board determined that the Rule 20 post-announcement petition process should be eliminated because the essay scores and the MPT and MBE scores for all applicants in the new process are interrelated in a scaled and weighted scoring process. Changes in "raw" essay scores after grade announcement could affect all grades and could change the pass/fail determinations for other applicants, which would affect the consistency, validity and integrity of the grading process. Board Rule 20 was deleted by the Court in 1999. As a result, the Board does not engage in any post-announcement review of specific grades assigned to any individual applicant.
When the Board changed its procedures in 2000, based upon the advice the Board received from the NCBE, the Board decided to round MBE written and total scale scores to two decimal places. The Board recommended that there be no further rounding in the scaling and combining procedures and the Supreme Court approved those procedures.
IV. Appeal Rights.
An applicant who is aggrieved by a final action of the Board may appeal to the Delaware Supreme Court for relief only if such action affects the substantial rights of the person claiming to be aggrieved. However, decisions of the Board with respect to a specific grade or grades assigned to any individual applicant are final and not subject to review by either the Board or the Delaware Supreme Court. (See Board Rule 41 and Supreme Court Rule 52(e).)
 Beginning with the February 2005 Exam, NCBE/ACT will change the method it uses to compute MBE scale scores and will no longer use a simple linear transformation like the one illustrated here. Instead, it will report MBE scale scores to one decimal place. All of the other procedures remain unchanged as described herein.
Board of Bar Examiners of the Supreme Court of Delaware
The Renaissance Centre
405 North King Street, Suite 500
Wilmington, DE 19801
P: (302) 651-3935 | F: (302) 651-3935