How my school gamed the stats

I was read­ing the Slate Star Codex re­view of Rule of the Smart. Part of it dis­cusses char­ter vs state schools and the alle­ga­tions of fraud of var­i­ous kinds un­der­min­ing char­ter schools record of bet­ter achieve­ment. Read­ing it, I re­al­ized that I took for granted that pub­lic schools en­gage in sys­tem­atic fraud in a va­ri­ety of ways. I don’t think this is some­thing ev­ery­one un­der­stands, hence this post.

I went to a state school in the UK. State schools are rated on a 1 − 4 scale from un­satis­fac­tory to out­stand­ing. My school was rated good, mean­ing a 3. A few mem­o­ries which stand out. Dur­ing my first week I saw one of the boys in my class who was 11 at the time held up against the wall in a cor­ri­dor while a 16 year old put a shiv to his throat and robbed him. He handed over his wallet and keys. A year or two later and I re­mem­ber see­ing a small boy who strug­gled with de­pres­sion held up by the throat against a locker and slapped in the face by a trou­ble­maker from the same class in front of ev­ery­one just be­fore we went in to the class­room. I re­mem­ber classes which were filled start to finish with peo­ple shout­ing and talk­ing. Nei­ther of the first two events were com­mon but they also weren’t un­com­mon. No one was sur­prised to wit­ness them. It’s worth em­pha­siz­ing again that my school was above av­er­age, in fact quite far above av­er­age, and in a mid­dle class area. It’s also worth not­ing that I was mostly in top abil­ity streamed classes, mean­ing my class­room ex­pe­rience was likely far bet­ter than av­er­age.

There were many ways in which the school and teach­ers gamed the sys­tem to boost their mea­sured perfor­mance. One way was to do ex­ams for stu­dents. I was on a bot­tom set lan­guage class for French. After two years I liter­ally couldn’t speak a sin­gle sen­tence in french and maybe knew 20 words in to­tal. I still passed my ex­ams. How? We did the tests in class. Often the teacher would go through them with us. Liter­ally giv­ing us the test and then go­ing through each ques­tion on the white­board and tel­ling us what to write. A differ­ent year and a differ­ent teacher, this time the teacher would sit next to us and write the an­swers down. Why sit next to us? It was the bot­tom set so peo­ple of­ten wouldn’t even bother to write down the an­swer if they were told it. This kind of thing was nor­mal, so much so that I, and I think most peo­ple there, didn’t re­al­ize any­thing un­usual was hap­pen­ing.

Another way schools game met­rics is to cheat in­spec­tions. A ma­jor com­po­nent of how schools are judged in the UK is through in­de­pen­dent in­spec­tions car­ried out by an in­de­pen­dent quasi-gov­ern­men­tal or­ga­ni­za­tion called Ofsted. Now, you may imag­ine that these in­spec­tions would be unan­nounced, so as to best get a real image of how a school works. Not the case. They’re sched­uled well in ad­vance. Be­fore ev­ery in­spec­tion, a few things would hap­pen in my school:

• The worst trou­ble­maker kids would be taken aside and put in a spe­cial room where in­spec­tors wouldn’t see them. Either that or they would just be told not to come into school at all on that day.

• All of us were told in as­sem­bly that an in­spec­tion was com­ing and to be on our best be­hav­ior on that day. Often teach­ers would have con­ver­sa­tions with less se­ri­ous trou­ble­mak­ers and im­press on them that they would be­have on that day or face con­se­quences af­ter­wards.

• Teach­ers would put a great deal more effort into their les­son plans than was nor­mal. Class­room be­hav­ior man­age­ment would also be far stric­ter. Be­cause of these and other mea­sures my school dur­ing an in­spec­tion was ut­terly differ­ent than my school on a nor­mal day. On some level this isn’t sur­pris­ing. If teach­ers’ pro­mo­tions and man­age­ment’s jobs de­pend on good in­spec­tion re­sults and in­spec­tions are easy to game, peo­ple will game them. In­cen­tives drive be­hav­ior. But it’s still sad.

Another way the stats were gamed was by not record­ing bad be­hav­ior. When a school gives a de­ten­tion or sus­pends/​ex­pels a stu­dent, there’s a record of it. This is es­pe­cially true of sus­pen­sions, stu­dents be­ing sent home or ex­pul­sions. The more of these you have, the worse you look as a school. The solu­tion then is ob­vi­ous, don’t pun­ish peo­ple or pun­ish them in non-recorded ways. Again, in my school it was com­pletely nor­mal for stu­dents in lower sets to swear at the teacher, talk over them or dis­rupt the class for ev­ery­one else. It was nor­mal for some­one to be ag­gres­sive and abu­sive to­wards oth­ers and to face at most a 40 minute de­ten­tion, but even get­ting a de­ten­tion would be un­usual.

I re­al­ize that one data point is not enough to draw solid gen­eral con­clu­sions. My own per­cep­tion is that this kind of fraud wasn’t spe­cific to my school. My cousin went to a state school fairly nearby. He’s 4 years younger than me. Dur­ing one of my win­ters back from un­der­grad we dis­cussed his school and his ex­pe­riences mir­rored mine. His ex­act words re­gard­ing in­spec­tions were “I learned 4 times more that day than any other day that year. It was amaz­ing”. I talked to a few Bri­tish stu­dents at uni­ver­sity, al­though speci­fi­cally the not mid­dle/​up­per class ones who would have gone to pub­lic schools. They had gone to schools similar to mine in differ­ent parts of the coun­try and their sto­ries were similar and of­ten worse. Two par­tic­u­larly funny ex­am­ples from my friends’ ex­pe­riences stand out. A teacher in year 9 walked up to a stu­dent who was talk­ing, picked them up and threw them out of an (open) first floor win­dow. My friend sit­ting in class no­ticed two boys mak­ing fun of him and then pro­ceeded to get up in the mid­dle of class while the teacher was talk­ing, walk to their table, flip the table up­wards to hit them in the face be­fore go­ing to sit down again when the teacher told him to. (Re­mem­ber, my friend was a stu­dious, sporty Asian kid and not a trou­ble­maker. This kind of thing is nor­mal in that en­vi­ron­ment). Comedic sto­ries aside, my ex­pe­riences in school, while not uni­ver­sal, seem fairly com­mon in the UK and from what I’ve read of the statis­tics, bad US schools are far, far worse.

I’m un­sure what my point here is. I think I have two:

• Char­ters may cook their books in var­i­ous ways. In the UK, State schools do too. I would be sur­prised if it wasn’t also the case in the US.

• I think that I feel like a lot of com­men­ta­tors on places like SSC have fairly mid­dle class ex­pe­riences of fairly good schools and that bleeds into how their com­par­i­son be­tween state vs char­ter schools. It’s just good to re­mem­ber that it’s not those nice mid­dle class schools that char­ters typ­i­cally re­place.

Cross­posted to my blog at https://​​dis­sent.blog/​​2021/​​02/​​20/​​how-my-school-gamed-the-stats/​​