Education
Consumers Consultants Network
Response
to Education Commission of the States Panel Report of September 23, 2002
*************************************************************************************************************************
The
Controversy Regarding the NBPTS Briefing & Study
May 12,
2002
(updated
October 5, 2002)
The Education Consumers ClearingHouse (ECC) is a paid
subscription service--a Consumers Union for the consumers of public
education. Briefings on educational
research are published as a service to ClearingHouse subscribers by the
Education Consumers Consultants Network (ECCN)--a partnership of consultants
affiliated with the ClearingHouse.
Occasionally they are publicized as a public service.
The
Briefing titled “Value-Added Achievement
Gains of NBPTS-Certified Teachers in Tennessee: A Brief Report” (Vol. 2, No. 5, May 2002, http://www.education-consumers.com/briefs/may2002.asp)
reported that a study by J. E. Stone found that NBPTS-certified teachers in
Tennessee were only average producers of student achievement gains as compared
to other teachers in their school districts.
Stone is the principal partner of the ECCN. Given persistent questions about the value of NBPTS certification
(most recently: Harvard Education
Letter, http://www.edletter.org/current/abstracts.shtml
- a4), policymakers were urged to suspend the program until definitive
research can be undertaken.
The Briefing and the study by Stone have drawn comments from
several organizations:
Linda Seebach in the Rocky Mountain News, October 5,
2002 (NEW):
http://www.rockymountainnews.com/drmn/news_columnists/article/0,1299,DRMN_86_1459977,00.html
“No one need be surprised that the national board is
digging in its heels to protect its very lucrative franchise. But why is the
Education Commission of the States, supposedly an independent body, so eager to
help out?”
Education Week, “ECS Review Discounts Study Critical of Teaching Board,”
October 2, 2002 (NEW):
http://www.edweek.org/ew/ewstory.cfm?slug=05ecs.h22&keywords=Stone
“The review of Mr. Stone's seven-page
study, which he unveiled last spring, does not charge that his conclusions are wrong,
but rather that the methods that led to his conclusions were faulty.”
Thomas Fordham Foundation (NEW): http://www.edexcellence.net/foundation/gadfly/issue.cfm?id=40#581
The Gadfly: Study of NBPTS
certification also scrutinized by a panel of experts
Even more big guns were brought out by the Education Commission of the
States (ECS) to evaluate a small study that examined the effectiveness of
teachers certified by the National Board for Professional Teaching Standards
(NBPTS) in Tennessee. That study (actually a 4-page brief followed by 4 pages
of data), by J.E. Stone of East Tennessee State University and the Education
Consumers’ Clearinghouse, analyzed the value-added achievement gains produced
by NBPTS-certified teachers in Tennessee, as generated by the much-vaunted
Tennessee Value Added Assessment System. [For more about Stone’s study, see
http://www.edexcellence.net/foundation/gadfly/issue.cfm?id=58#846.]
Stone found that none of the 16 Board-certified teachers who teach in
grades 3-8 in Tennessee met a standard for exceptional teaching. (That standard
was producing 115 percent of a year’s academic growth in their local school
system in three core subjects over three years, a standard now used to identify
exceptional teachers in a new incentive program in Chattanooga. [For more about
the program, see
http://www.edexcellence.net/gadfly/v02/gadfly12.html#reading1.])
ECS asked four prominent scholars to examine the Stone study: Dominic
Brewer, Susan Fuhrman, Robert Linn, and Ana Maria Villegas. (We hope ECS will
continue this admirable practice of asking independent scholars to review all
future studies of the effectiveness of the NBPTS, not just short briefs
produced by board critics.) The reviewers complained that it was unclear how
the 16 teachers were selected for the study and expressed concern that the
teachers included in the study might not be representative of Board-certified
teachers in Tennessee, but Stone’s study makes clear that the 16 teachers are
the only Board certified teachers in Tennessee who teach in grades 3-8.
(Value-added scores are not available for Tennessee teachers in other grades.)
The reviewers’ main concern is that teacher value-added scores jump
around from year to year and vary significantly by subject and school district.
Volatility of gain scores does present a challenge for efforts to identify
effective teachers this way. But Stone looks at both averages and scores from
individual years in his analysis, and no matter what angle he used,
Board-certified teachers simply didn’t produce exceptional gains in student
learning. The Tennessee testing system is not perfect, and it would have been
helpful to know more about the distribution of exemplary and deficient scores
among the general teaching population in Tennessee, but it does not appear that
Tennessee’s Board-certified teachers are setting records for the value they add
to student achievement.
While none of the Board-certified teachers in Tennessee was able to meet
the standard set by Chattanooga for exceptional teaching, at least four
teachers of 4th or 5th grade in Chattanooga did meet that standard, according
to Ken Jordan, a special assistant to Mayor Bob Corker. Those four teachers
applied and were selected to teach in high-need elementary schools in the
district after they submitted evidence that they had produced gains of at least
115 percent of one year’s growth in three core subjects over three years.
School Reform News (NEW):
Noted education researcher Eric Hanushek, a senior fellow at
the Hoover Institution, told School Reform News that, contrary to the
NBPTS’ insinuations, “Stone’s study follows a well-conceived
methodology.” He added that “Tennessee is not an aggressive NBPTS state,”
and therefore Stone had no choice but to rely on a mere 16 teachers in the database.
“This weakness is not Stone’s fault, or his choice. It simply represents
the available universe of teachers.”
Hanushek observed, “John Stone’s provocative study
underscores one extremely important feature of U. S. education: Widely
acclaimed and expensive policies frequently escape any evaluation in terms of
their true effectiveness. Stone’s study is far from definitive, but it is
[italics in the original] the evidence that is available.
“The NBPTS certification process has become an important
element of policy in several states, and large financial rewards flow to
successful applicants. But all of this happened without a thorough
analysis of its effectiveness.”
States and school districts have awarded pay increases or
bonuses in the range of $5,000 to $7,500 per year for teachers winning national
certification. After paying the NBPTS a $2,300 application fee--a tab
sometimes picked up by their districts--candidates prepare a portfolio of their
work, videotape themselves teaching, and take an all-day examination.
Hanushek said now that Stone has asked “the right question,”
others--including states with a significant financial stake in the NBPTS
process--should look at the evidence also.
“One thinks that, had Stone’s
study of 16 teachers supported the certification program, it would have been
widely publicized and little criticized,” Hanushek commented. (School Reform News, Vol. 6, No. 8, August 2002, http://www.heartland.org/education/aug02/certified.htm)
Comments from a number of other professors, researchers, and
teachers are quoted. They include: Dr. George Cunningham of the
University of Louisville, Dr. Louis
Chandler of the University of Pittsburgh, John Tuepker, a teacher and AFT
official in Mississippi, and Al Haskvitz, a national award winning teacher in
California. (School Reform News, Vol. 6, No. 8, August 2002, http://www.heartland.org/education/aug02/findings.htm)
Center for Education Reform:
This shouldn't be a surprise.
NBPTS certification is really just ordinary certification on steroids -- a
puffed-up assessment of teachers' mastery of conventional certification
standards. Considering the large body of research finding almost no correlation
between certification and teacher effectiveness, it makes sense to expect no
correlation between "super-certification" and student performance.
What one should expect is proportionately inflated rhetoric about the value of
the certification, which is exactly what one finds.
(Education Reform Newswire,
Vol. 4, No. 17, May 7, 2002, http://www.edreform.com/update/2002/020507.html).
Thomas Fordham Foundation:
NBPTS-certified teachers flunk
value-added test in Tennessee
(The Education Gadfly, Vol.
2, No. 19, May 9, 2002, http://www.edexcellence.net/gadfly/index.html).
Education
Week:
(NEW)
To date, research on the value of the voluntary
national credential—especially its effect on student achievement—has been
thin. (“Critical Study
of NBPTS Spurs State Advisory Group to Act,” May 15, 2002).
A
news release by the NBPTS (http://www.nbpts.org) was critical of the Tennessee study
and the Education Commission of the States (http://www.ecs.org/clearinghouse/35/57/3557.htm)
called for a review by a “panel of unbiased, distinguished educators and
researchers.”
National Board for
Professional Teaching Standards news release of May 7, 2002:
The recent report by J. E. Stone about testing results in
Tennessee on National Board Certified Teachers is hardly independent
research. Stone has been a frequent opponent of the National Board and by
his own admission supports marketplace models for teaching. He opposes
certificates, licensure, and credentialing and calls those efforts
"meaningless."
In this review of test scores, Stone uses 16 out of 40
National Board Certified Teachers (NBCTs) in a state with one of the lowest
percentages of NBCTs (.07%). The representation is a minuscule percentage
as pointed out by Education Daily and represents less than .01 of total
NBCTs. Mr. Stone is well aware of the fact that a significant study of
the impact of NBCTs on student achievement is currently underway by William
Sanders who developed the "value added" system in Tennessee.
Mr. Sanders did not approve or authorize Stone's "so called" research
model. A cursory look at the information used for the 16 teachers
suggests there is not enough data to make good judgements about the effects of
NBCTs.
The National Board is looking forward to the results of the
upcoming study that we have commissioned from Sanders that focuses on a much
larger sample of NBCTs in the state of North Carolina. This study will
compare more than 800 teachers; those who achieved National Board
Certification, those who went through the process and did not achieve
certification, and teachers who have not been through the process.
We also welcome the review of the Stone study by the
Education Commission of the States (ECS). ECS is assembling a panel of
"unbiased, distinguished educators and researchers" to conduct an
independent review of the study's procedures, data and findings, and issue a
public report of its conclusions.
[The remainder of the release is
available at: http://www.nbpts.org]
Reply and Comment
Plainly, the Education Consumers ClearingHouse (ECC), its
Consultants Network (ECCN), and the present study represent an independent
viewpoint. For a consumer organization
or the studies it commends to be otherwise would be the proverbial “kiss of
death.” Without independence, the
ClearingHouse would be out of business.
The ClearingHouse would not expect the NBPTS to endorse this
study. Manufacturers and vendors, for
example, frequently do not agree with the assessments of their products by
consumer organizations. The ECC’s
stock-in-trade is credibility with the consumer, not acceptance by the
education community.
The
NBPTS objects to the Tennessee study on the grounds that its author (Stone) has
been a frequent critic of NBPTS; that he supports "marketplace"
models for teaching; and that he characterizes teaching certificates,
licensure, and credentials as meaningless.
These
objections are factual but relevant only from the standpoint of an unreasonably
stringent standard of scholarly impartiality.
Scholars rarely study a subject about which they have no opinion. The question is whether their opinion shaped
the outcome. In any case, it is
apparent that NBPTS and ECS have not applied this standard to the several studies
of NBPTS that have reported supportive findings.
As
to a “marketplace” point of view, in fact, the ClearingHouse and its
Consultants Network are founded on the belief that the educational priorities
of the parties who furnish the children and the money should be respected by
the public servants whose salaries they pay.
It
is also true that Stone, Cunningham, and Crawford (2001, http://www.education-consumers.com/consultants/opinion3.pdf)
have criticized NBPTS, NCATE, and INTASC standards on the grounds that they
urge teaching practices unsuited to the attainment of student achievement—an
assertion vindicated by the Tennessee evidence.
Finally,
in a statement supporting the recent Abell Foundation report (http://www.education-consumers.com/articles/teacher_certification.asp),
the ECCN including Stone have argued that existing teaching certificates,
credentials, and licensure are meaningless with regard to the public’s interest
in protection from unsound and ineffective teaching.
Stone
has remarked, "Consider the fads to which public school students
historically have been subjected by fully credentialed and licensed
educators. Physicians who use questionable
practices are sued and called quacks.
Engineers whose bridges collapse are fined and lose their licenses. Educators who institute fads are called
innovators and given more funding."
The
Size of the Tennessee Study
The
NBPTS's chief complaint about the Tennessee study was that the number of
teachers (16) was small and therefore may not be representative of
NBPTS-certified teachers in general (a point carefully discussed by Stone's
report).
Stone's
reply: "While it is statistically
possible that the 16 teachers available in TN just happen to be the only
average performers in the NBPTS-certified population, don't bet the farm on
it. Surely Tennessee isn't so unlucky
that it got them all."
And
what if there were only 16 mediocre NBPTS-certified teachers out of
16,000? If 16 certified lifeguards were
found to be mediocre swimmers, the finding would not be dismissed as
statistically insignificant.
Like
an IQ score, NBPTS certification is a clinical statement. NBPTS is certifying that each teacher is
qualified, not that the NBPTS-certified group has an acceptable average.
NBPTS’s
and ECS’s Concerns and Priorities
If
the NBPTS’s top priority were ensuring that their teachers are as skilled as
advertised, they would take careful note of the Tennessee study and consider
whether they should suspend their promotional campaign until fuller data is
available. Instead, they seem intent on
discrediting it and defending the status quo.
In
truth, if policymakers continue to presume the validity of NBPTS certification
and commit billions more to NBPTS teacher bonuses, it may become politically
impossible to abandon the program regardless of what future studies show.
This
scenario is reminiscent of the life cycles had by the various teacher
career-ladder plans that were adopted in the early nineteen eighties. They too involved complex teacher assessment
schemes that were similarly unsupported by proof of teacher effectiveness. In Tennessee, for example, all that remains
of its career ladder are the teacher bonuses and tax increases. Policymakers
who are considering adoption of NBPTS bonuses would be well advised to consider
Tennessee's experience and its budgetary consequences.
The
same might be said of the response made by the Education Commission of the
States (ECS) (http://www.ecs.org/clearinghouse/35/57/3557.htm). Instead of alerting its members and noting
that NBPTS certification has never been clearly linked to student achievement,
ECS took the unprecedented step of calling for the Tennessee study to be
reviewed by an 'unbiased, distinguished' review panel. By contrast, ECS failed to express
skepticism about any of the several NBPTS sponsored studies of the past 10 or
so years, some of which included as few as 3 NBPTS-certified teachers [see J.
Hattie, J. Clinton, M. Thompson, & H. Schmidt-Davis, Identifying Highly
Accomplished teachers: A validation study, A Research Report of the
National Board for Professional Teaching Standards Technical Analysis Group
(Greensboro, North Carolina: Center for Educational Research and Evaluation,
University of North Carolina-Greensboro, 1995)].
ECS
has played a valuable role in educational reform but it may have a blind spot
when it comes to NBPTS. Governor James
Hunt--a former ECS chair--was the founding chair of NBPTS and Ted
Sanders--current president of ECS--is one of the founding commissioners of the
NBPTS's greatest supporter, the National Commission on Teaching and America's
Future. Like the state departments of
education of its constituency, ECS tends to represent the education community’s
prevailing views. In fact, NBPTS’s
early endorsement by ECS (see ERIC ED 304444-1988, http://www.ed.gov/databases/ERIC_Digests/ed304444.html)
is emblematic of its mainstream acceptance.
Unless
ECS President Ted Sanders is confident that the forthcoming studies by Dr.
William Sanders and others will not suggest major revisions in NBPTS, we would
urge that he examine all of the NBPTS research and caution the states that the
lack of supporting data may mean that they are committing millions to
unwarranted teacher bonuses. Historically
educational fads have been “discovered” only after the money has been
spent. Perhaps the NBPTS initiative
could become the one exception.