Drafting and Grading Guidelines and Procedures
Beginning in 1998, the Board of Bar Examiners (the
"Board") of the Delaware Supreme Court (the “Court”), in consultation
with experts from the National Conference of Bar Examiners (the “NCBE”) and bar
examination testing experts, undertook a study of the Delaware Bar
Examination. The object of the review was to, among other things:
(a) enhance the ability of the Board, through administration of the
Delaware Bar Examination, to identify fairly, accurately and effectively those
applicants who meet the threshold requirements set by the Court for admission
to its Bar; (b) maintain grading procedures that are both consistent and
fair to all tested applicants; and (c) promote fairness and objectivity to
all applicants in the Board’s evaluation of competency.
Following this review in 1998 and 1999, the Board
recommended to the Court, and the Court approved in 2000, the following changes
to the Delaware Bar Examination, starting with the July 2000 Bar Examination:
(a) the Delaware law essay questions were reduced from twelve to eight
45-minute questions; (b) two 90-minute Multi-state Performance Test (the
“MPT”) essays were added; (c) the three parts of the Delaware Bar
Examination (the Multi-state Bar Examination (the "MBE"), the
Delaware law essays and the MPT) were weighted 40% MBE, and 60% essay and MPT;
(d) the written portions of the Delaware Bar Examination were scaled using
the standard deviation method; and (e) a combined score of 145.00 was set
as the minimum score required to pass the Delaware Bar Examination.
Along with these administrative and substantive changes,
the Board formally adopted grading and scoring procedures to promote
standardization and accuracy in grading essays and calculating scores. These
changes, used by the Board beginning with the administration of the July 2000 Delaware
Bar Examination, are explained below in detail.
II. Essay Drafting and Grading Procedures
In 2000, the Board formalized its essay drafting and grading
procedures as set forth below:
A. Drafting Question and Outline of Answer.
1. Assignment of Topics and Designation of Two-Person Drafting Partnerships.
Early in each calendar year, eight Board Members (the
“Essay Committee” of the Board), along with their respective Associate Members,
are assigned primary responsibility for drafting an examination question and an
outline of an answer and point allocations on a topic selected by the Board
from among those topics specified in Rule 12 of the Board of Bar Examiners
Rules. At the same time that essay question topics are assigned, drafting
partnerships are also designated, each consisting of two Board Members from the
Essay Committee. Each draft essay question and answer outline is thereafter
prepared and carefully reviewed by the assigned Board Member and his or her
Associate Member in accordance with the guidelines set forth by the NCBE.
The drafts are then exchanged with the other member of the
assigned drafting partnership for review and comment. The purpose of this
review is to provide additional objective input and, more specifically, to
assure that the question:
· fairly raises and calls for discussion of the issues expected to
be addressed by the examinee's responses, consistent with the accompanying
· avoids unintended ambiguities and is otherwise appropriate in
subject and form;
· does not focus on minutia or unduly reward rote memorization of
readily available rules or statutes rather than the quality of the examinee’s
analytical skills and legal understanding; and
· does not call for applicants to rely on a recent precedent or
statutory amendment to answer the question correctly.
Based upon comments received from
the drafting partner, the question and outline of the answer are then revised
by the drafting Member, in consultation with his or her Associate Member, and
the final versions are sent to the Board's Essay Review Committee (as defined
Although the MPT is drafted by the NCBE, the Board retains
the grading function. Board Members who grade the MPT attend a national
grading workshop sponsored by the NCBE on the weekend following administration
of the Delaware Bar Examination. The Board Members grading the MPT follow the
same grading procedures, explained in more detail below, that are used by Board
Members who grade the Delaware law essays.
2. Essay Review Committee.
The essay review committee is composed of the Chair and/or Vice Chair and selected Members of the
Board (the "Essay Review Committee"). The Essay Review Committee
reviews each question and answer outline and proposes revisions as
appropriate. The Essay Review Committee also reviews the draft question for
clarity, length and propriety of subject matter. It also looks to make sure
that the call of the question is correctly phrased in order to invite the
answer expected by the author and to minimize ambiguities, irrelevancies or
other perceived defects. This Committee also reviews the proposed answer
outlines for completeness, accuracy, clarity and reasonableness of point
allocation. Questions and answers are thereafter redistributed by the Essay
Review Committee to the drafting partnerships for further comment from the
author and revision in accordance with the revisions proposed by the Essay
Review Committee. The resulting versions of the questions and answer outlines
are thereafter submitted for final review and approval by the full Board. The
Essay Review Committee then reviews the examination to ensure proper format and
length of the examination as a whole as well as each of its essay sections.
1. General Grading Guidelines.
Essay questions are drafted to be graded on a 100-point
scale, referencing the approved point allocation for each question. Any
portion of an answer that fails to address the identified issue or that
evidences no semblance of issue
recognition, analysis or discussion receives no credit. Neither is credit
given for discussion of irrelevant issues or statements of general principles
of law that are not responsive to the question, even if correct. At the same
time, however, no points are deducted because of the inclusion of such
discussion or statements in an examinee's answer.
2. Grading Procedures.
The grading procedures are as follows:
a. Initial Grading Calibration Exercise.
Before final grading is undertaken, the Board Member who
drafted the original question and his or her Associate Board Member engage in a
calibration exercise, randomly selecting fifteen to twenty-five answer booklets
and independently assigning grades to each of them. The Board Member and
Associate Board Member thereafter compare the grades that each has awarded to
these examinations and establish specific standards to govern the grading and
resolve any divergence of process between them. A scoring sheet is then
prepared that, where necessary, incorporates and accounts for issues properly
but unexpectedly raised by examinees but which may not have been included in
the original outline of the answer, that properly accounts for any unintended
ambiguities in the question that the calibration exercise suggests may have
been reasonably perceived by examinees, and that reflects such other equities
as may be appropriate based upon this initial review.
b. Independent Grading by Board Member and
After the initial grading calibration exercise, the Board
Member and Associate Board Member thereafter independently, and in reverse
order, grade each essay answer. Upon completion of the independent grading by
the Board Member and Associate Member, the Board Member and Associate Member
compare the results each has reached with respect to each answer, discuss and
resolve any significant discrepancies by regrading, if necessary, and reach
agreement on a final grade for each applicant's examination. Thus, every
answer is automatically graded twice and any differences between the grades
assigned by the first and second readers are resolved. Those raw scores are
then submitted to the Executive Director by anonymous examinee number. Prior
to their submission to the Executive Director, all final grades are required to
be reviewed carefully to avoid any possibility of inaccuracies in the
completion of the form.
To maximize the integrity of the process, the Board does
not regrade essays after the initial dual readings from the Board Member and
Associate Member. Although regrading was selectively employed as a part of the
Board’s grading process prior to the July 2000 Delaware Bar Examination
(employed specifically with respect to scores that fell within a limited range
below the score necessary to pass the Examination), the Board has since
concluded, based upon expert statistical and psychometric advice, that such a
regrading process is inherently unreliable and that it effectively erodes the
objectivity and reliability of the Examination.
The Board determined that its former regrading system
should be eliminated because there already is effectively an automatic regrade
as a result of having two readers per answer. In addition, the essay scores
and the MPT and MBE scores for all applicants in the new process are
interrelated in a scaled and weighted scoring process. As a result, any
changes in “raw” essay scores caused by a regrading after scaling and combining
could affect other scores and could change pass/fail determinations.
Obviously, this would affect the consistency, validity and integrity of the
entire grading process. Therefore, the Board recommended, and the Supreme
Court approved, the elimination of any regrading.
III. Statistical and Accounting Experts Used for Scaling and Combining Raw Scores
Once the Executive Director receives the raw essay scores
on the Delaware law essays and the MPT, the scores are transmitted to the
Board’s statistical expert, Gansk & Associates (“Gansk”). Gansk converts
the raw scores on the eight Delaware law essay questions and the MPT to a
common scale of measurement, computes the scale scores for the written and MBE
portions of the exam, and weights and combines these scale scores to produce
“total scale scores” in accordance with the procedures set forth in Board Rule
An applicant shall be deemed to have
passed the Bar Examination if, in a single administration of that Examination,
the applicant achieves a “total scale score of 145.00 or higher.” “Total scale
scores” are computed as follows: (a) the raw scores on each of the eight
essay questions are converted to a score distribution that has a mean of 50 and
a standard deviation of 7, (b) the raw scores on each of the two MPT
questions are converted to a score distribution that has a mean of 50 and a
standard deviation of 14, (c) a given applicant’s “total converted score”
is the sum of that applicant’s converted essay and converted MPT scores,
(d) the distribution of the “total converted scores” is scaled to the MBE
to yield a “total written scale score,” and (e) the applicant’s “total
written scale score” is combined with that applicant’s MBE scale score (with
the written score weighted at 60% and the MBE score weighted at 40%) to produce
a “total scale score.” The term “scaled to the MBE” as used herein means that
the distribution of “total converted scores” is scaled to a distribution that
has the same mean and standard deviation as the MBE scale scores in Delaware.
Board Rule 13.
The results are then sent to the Board. To confirm that
there are no mathematical errors in the scoring process, the Board sends the
results and Gansk’s calculations to a Delaware accounting firm. The accounting
firm reports to the Board and confirms the results, or, if necessary,
identifies any computational errors.
By way of example, the scaling and weighting formulas used
for the 2003 Delaware Bar Examination are set forth as follows:
(1) Convert MBE raw scores (i.e., the number of questions answered
correctly) to scale scores using the formula provided by NCBE/ACT. The formula for the July 2003 Delaware
Bar Examination was:
MBE Scale = (0.86 x Raw Score) + 31.93
(2) Convert the raw scores on each of the eight essay questions to a score
distribution that has a mean of 50 and standard deviation of 7. The attached
table shows the converted value for each reader assigned raw score on each
question. For example, a reader assigned raw score of 40 corresponds to a
converted score of 34.94 on Question #1, 41.38 on Question #2, etc. A new
table is constructed for each exam.
(3) Convert the raw scores on each of the two Multi-state Performance Test
(MPT) problems to a mean of 50 and a standard deviation of 14. The MPT
problems are questions #9 and #10 on the attached conversion table.
(4) Compute the sum of the ten converted scores from Steps 2 and 3. Insert
this sum into the formula below to obtain the written scale score:
Written Scale Score = (0.25 x Sum of ten
converted scores) + 23.33
(5) Compute each applicant’s Total Scale Score as follows:
Total Scale = (.4 x MBE Scale Score) + (.6 x Written
Numerical Example: Suppose an applicant
had an MBE raw score of 120. According to the MBE scaling formula for the July
2003 Exam, this raw score converts to a value of 135.13 as follows:
= (0.86 x 120) + 31.93 = 135.13
The sum of this applicant’s ten converted
written scores was 500. According to the formula for scaling written raw
scores on the July 2003 Exam, a total written converted score of 500 has a
value of 148.33 as follows:
Scale Score = (0.25 x 500) +23.33 = 148.33
This applicant’s Total Scale Score is therefore
(.4 x 135.13) + (.6 x 148.33) = 143.05
Full Board Review
The Board thereafter meets to review, discuss and approve
the Bar Examination results. Upon final approval, the scores are published to
the applicants by anonymous number only. This information is released on the
Board’s website and posted at each of the three County law libraries as
promptly as possible following approval of the scores. After the results have
been posted by anonymous number only, Board Members go to the safe deposit box
where the sealed envelopes containing the anonymous numbers with corresponding
applicant names have been stored since the first day of the Bar Examination.
Following retrieval of the envelopes, Board Members match the anonymous number
with the applicant's name. Shortly thereafter, the names of the passing
applicants are posted on the website. Passing applicants are advised of their
success in writing. Failing applicants are advised in writing and provided
with their scores on each of the various components of the Examination and are
advised of their opportunity to secure copies of their essay answers and copies
of representative answers to each essay question for comparison purposes.
Copies of the essay questions are thereafter provided to law libraries
throughout the State.
Elimination of Post-Announcement Review of
Prior to the adoption of the 2000 grading changes, a petition
filed pursuant to then Board Rule 20 could result in a review of the
petitioner's Examination scores. The Board studied this practice in
conjunction with its 1998-1999 review of the Delaware Bar Examination
procedures. After consulting with the NCBE and bar examination testing experts,
the Board determined that the Rule 20 post-announcement petition process should
be eliminated because the essay scores and the MPT and MBE scores for all applicants
in the new process are interrelated in a scaled and weighted scoring process.
Changes in “raw” essay scores after grade announcement could affect all grades
and could change the pass/fail determinations for other applicants, which would
affect the consistency, validity and integrity of the grading process. Board
Rule 20 was deleted by the Court in 1999. As a result, the Board does not
engage in any post-announcement review of specific grades assigned to any
When the Board changed its procedures in 2000, based upon
the advice the Board received from the NCBE, the Board decided to round MBE
written and total scale scores to two decimal places. The Board recommended
that there be no further rounding in the scaling and combining procedures and
the Supreme Court approved those procedures.
IV. Appeal Rights.
An applicant who is aggrieved by a final action of the
Board may appeal to the Delaware Supreme Court for relief only if such action
affects the substantial rights of the person claiming to be aggrieved.
However, decisions of the Board with respect to a specific grade or grades
assigned to any individual applicant are final and not subject to review by
either the Board or the Delaware Supreme Court. (See Board Rule 41
and Supreme Court Rule 52(e).)