Louisiana's Value-Added Mess

Value Added Methodology (VAM) Stability/Validity in Louisiana

(excuse the formatting - can't make adjustment) 
VAM as it is currently being used has approximately a 10-39% stability/validity rate
Written by:

Wayne Free, Former Assistant Executive Director, Louisiana Association of Educators
In a report entitled The Status of the Development of the Value Added Assessment Model as specified in Act 54 - A report to the Senate Education Committee and the House Education Committee of the Louisiana Legislature, February 25, 2011.” Drs. George Noell and Beth Gleason Strategic Research and Analysis, Louisiana Department of Education (DOE) write: The results show moderate stability across years. Teachers who fell in the bottom 20% in2007-2008 were likely to fall in the bottom 20% of results again (mathematics: 45.3%; ELA: 39.8. They were unlikely to move to the top of the distribution one year later. Teachers who were in the top 20% in 2008-2009 were most likely to fall in that range in 2009-2010 (mathematics: 61.6%; ELA: 55.7%). They were unlikely to move to the bottom of the distribution one year later.

Perhaps I am weak in mathematical analysis but when I read data that is not consistent with the conclusions, I question the reasons.  Over 50% (54.7%) of the teachers that scored in the bottom 20% of the VAM mathematics score in 2008-2009 scored above the bottom 20% in 2009 - 2010  In fact, of the teachers who scored below 10% over 70% raised their score the 2nd year without changing teaching practice.  So much for moderate stability” (ELA scores were even wider). The same can be said of the teachers who scored in the top 20%

A closer look at the report and confirmed by Dr. Noell indicates that teachers who fall in the bottom 10% of VAM scores in 2007-2008 would fall in the bottom 10% in Math 26.8 % of the time and in ELA 22.3% of the time even if they did nothing different the next year.  A teacher who fell in the top 10% in 2007-2008 would fall in the top 10% approximately 45.8% of the time in Math and in ELA 37.5% of the time even if they did nothing different.

VAM as it is being used currently has approximately a 75% error rate (73.2% in Math, 77.7% in ELA) at the bottom 10% level and approximately 57% error rate (54.2 in Math, 62.5% in ELA) at the top 10% level based on the Validity numbers in the departments report.

Additional Data from the Louisiana Department of Education (DOE) reinforces the earlier data and adds additional years to the information. In the new data, the stability from 2009-2010 through 2011-2012 drops to 10% at the Ineffective level over the period and 21% at the Highly Effective level.  Neither number is sufficient to justify the current use of VAM in Louisiana. (See following charts and table 1.1 below… data supplied by DOE 10/12/2012)

Rating Level
2009-2010 original
number Rated
2010-2011 number
remaining in the original
Rating
% Remaining at the original level
% moving up at least one level
% moving down at least one level
1-10% (Ineffective)
539
137
25%
75%
0%
11-49% (Effective emerging)
3085
1659
54%
37%
9%
50-89% (Effective Proficient)
3391
1892
56%
11%
33%
90-99% (Highly Effective)
800
319
40%
0%
60%
Total Individuals Rated
7815







Rating Level
2009-2010 original
number Rated
2010-2011 number
remaining in the original
Rating
2011-2012 number
remaining in the original
Rating
% Remaining at the original level
1-10% (Ineffective)
539
137
52
10%
11-49% (Effective emerging)
3085
1659
944
31%
50-89% (Effective Proficient)
3391
1892
1092
32%
90-99% (Highly Effective)
800
319
169
21%


Others agree: A report by the Economics Policy Institute (EPI) reinforces the instability of VAM scores.  In the announcement of the study results EPI writes, Student test scores are not reliable indicators of teacher effectiveness, even with the addition of value-added modeling (VAM), a new report by leading testing experts finds. Though VAM methods have allowed for more sophisticated comparisons of teachers than were possible in the past, they are still inaccurate, so test scores should not dominate the information used by school officials in making high-stakes decisions about the evaluation, discipline and compensation of teachers. 

A recent report released by Marcus A. Winters, a senior fellow at the Manhattan Institute and an assistant professor at the University of Colorado, Colorado Springs says, VAM-based measures of teacher performance can be quite imprecise. When VAM is used to inform tenure decisions, it is likely that some average and even above-average teachers could be removed from the classroom because of a low VAM score caused by random variation in measurement over the years, rather than their own failures.

All of this would be academic if it werent for the politics of the issue and the rush to judgment about teacher performance using a system that is inappropriate for high stakes” decisions.

If you are interested in further information about VAM research or other issues surrounding teacher effectiveness email me. 



2010-2011
2011-2012
2009-
2010
Rank
Bottom
1% - 10%
Middle
11% - 49%
Middle
50% - 89%
Top
90% - 99%
Bottom
1% - 10%
Middle
11% - 49%
Middle
50% - 89%
Top
90% - 99%

Bottom
1% -
10%
(539 )
25.4%
(137)
57.5%
(310)
16.5%
(89)
0.6%
(3)
38%
(52)
50.4%
(69)
10.2%
(14)
1.5%
(2)

Middle
11% -
49%
(3085 )
9.0%
(278)
53.8%
(1659)
34.2%
(1055)
3.0%
(93)
9.5%
(157)
56.9%
(944)
31.5%
(522)
2.2%
(36)

Middle
50% -
89%
( 3391)
2.2%
(74)
31.0%
(1050)
55.8%
(1892)
11.1%
(375)
2.6%
(49)
29.8%
(564)
57.2%
(1082)
10.4%
(197)

Top
90% -
99%
(800 )
1.6%
(13)
12.1%
(97)
46.4%
(371)
39.9%
(319)
2.2%
(7)
7.2%
(23)
37.6%
(120)
53.0%
(169)

Supporting DATA:

Stability of Teacher Ranking Using Overall Composite Percentiles across 2009-2010 to 2010-2011 to 2011-2012

Table 1.1

Please note: to read the table above the 2009-2010 rank is the starting year. The 2010-2011 columns display where teachers in each 2009-2010 rating fell. The 2011-2012 columns then display where, after the 2010-2011 year, teachers were rated. For example in 2009-2010 there were 539 teachers rated as ineffective. In the 2010-2011 year, 137 of these teachers remained in the ineffective rating. Out of the 137 ineffective teachers, 52 of them remained ineffective for a third year, 2011-2012.
This table displays:

In 2009-2010 539 teachers were rated ineffective, as per value-added results. Out of those 539 teachers, 137 or 25.4% of them remained ineffective for the next year, 2010-2011. Three hundred ten, or 57.5%, of the 539 were rated as Effective: Emerging in 2010-2011. Of those 310; 89, or 16.5%, were rated as Effective: Proficient; and 3, or 0.6%, were rated as Highly Effective. Of the 137 rated Ineffective for 2 consecutive years, 52, or 38% of them remained in this same category for a third year in 2011-2012. Sixty-nine, or 50.4%, were rated as Effective: Emerging, 14, or 10.2% were rated as Effective: Proficient, and 2, or 1.5% were rated as Highly Effective in the third year, 2011-2012.

In 2009-2010 3,085 teachers were rated as Effective: Emerging, as per value-added results. Out of those 3,085 teachers, 1,659 or 53.8% of them remained Effective: Emerging for the next year, 2010-2011. Two hundred seventy-eight, or 9.0%, of the 3,085 were rated as Ineffective; 1,055, or 34.2%, were rated as Effective: Proficient; and 93, or 3.0%, were rated as Highly Effective. Of the 1,659 rated Effective: Emerging for 2 consecutive years, 944, or 56.9% of them remained in this same category for a third year in 2011-2012. One hundred fifty-seven, or 9.5%, were rated as Ineffective, 522, or 31.5% were rated as Effective: Proficient, and 36, or 2.2% were rated as Highly Effective in the third year, 2011-2012.

In 2009-2010 3,391 teachers were rated as Effective: Proficient, as per value-added results. Out of those 3,391 teachers, 1,892 or 55.8% of them remained Effective: Proficient for the next year, 2010-2011. Seventy-four, or 2.2%, of the 3,391 were rated as Ineffective; 1,050, or 31.0%, were rated as Effective: Emerging; and 375, or 11.1%, were rated as Highly Effective. Of the 1,892 rated Effective: Proficient for 2 consecutive years, 1,082, or 57.2% of them remained in this same category for a third year in 2011-2012. Forty-nine, or 2.6%, were rated as Ineffective, 564, or 29.8% were rated as Effective: Emerging, and 197, or 10.4% were rated as Highly Effective in the third year, 2011-2012.

From: George Noell [mailto:gnoell@lsu.edu] 
Sent: Thu 5/10/2012 10:20 AM
To: Free, Wayne [LA] 
Subject: RE: Report to legislative education committees

Wayne, 
Answers are below, based on the mathematics data (exact numbers vary slightly between content areas.

1. If a teacher scores in the lowest 10% of the VAM score the first year and does nothing different the next year what is the likelihood they will fall in the lowest 10% the second year and remain "ineffective".
26.8%

2. If a teacher scores in the lowest 10 - 20% range of the VAM score the first year and does nothing different the next year what is the likelihood they will fall in the lowest 10% the second year and become "ineffective".
14.8%

3. If a teacher scores in the lowest 20 - 30% range of the VAM score the first year and does nothing different the next year what is the likelihood they will fall in the lowest 10% the second year and become "ineffective".
I don't have those data and I don't have the to run that since I am back at LSU.

4. If a teacher scores in the highest 10% of the VAM score the first year and does nothing different the next year what is the likelihood they will fall in the highest 10% the second year and remain "highly effective".
45.8%

5. If a teacher scores in the highest 10 - 20% of the VAM score the first year and does nothing different the next year what is the likelihood they will fall in the highest 10% the second year and become "highly effective".
22.1%

6. If a teacher scores in the highest 20- 30% of the VAM score the first year and does nothing different the next year what is the likelihood they will fall in the highest 10% the second year and become "highly effective".
I don't have those data and I don't have the to run that since I am back at LSU.

7. I guess what I'm actually asking is what is the stability range across years based on a 10% differential each year and not the top to bottom analysis given in the report
Numbers are above.

Hope that helps.
George

_______________________
George Noell, PhD, BCBA
Professor
Department of Psychology
Louisiana State University
Baton Rouge, LA 70803
office: (225) 578-4119
FAX: (225) 578-4125


Graph # 1













Graph # 2
 









Graph # 3
 






Graph # 4
 








Graph # 5
















No comments:

Post a Comment

Your Comments Welcome