The contest and its aftermath have more or less reached their conclusion. After a few weeks of silence, waiting for things fall into place, I'll tell you the story.
It turns out that I was the one with the most judge's points at the end of the two days of finals. Now, I could have lost the contest by playing poorly, and I could have lost it by not distinguishing myself enough from the other candidates and getting muddled in the mix. But I didn't. I was the jury's choice, the people's choice, and even the evening news reported me as the expected winner. But when the awards were announced, I was ranked 4th. The music committee did an after-the-fact evaluation of that scoring algorithm and determined that it produced illegitimate results. Unfortunately, it was too late to do much at that point. Give me the Al Gore Presidential Medal of Honor – I lost it in the counting.
Of Mice and Men -or- It seemed like a good idea at the time:
I'll try to explain the problem with the scoring system as best I understand it.
Objectivity was the buzzword around this competition. There would be no undue influence, there would be no strong-arming in jury discussions, there wouldn't even be jury discussions, there would only be points and averages. To go even further, a special point averaging system was put into play to prevent any judge from influencing the final averages by giving extreme point totals. If a judge gave a grade which differed from the average grade by more than 10 (out of 100) points, their grade was eliminated from the final average. The intention was that this would keep any judge from trying to singlehandedly eliminate a candidate by giving a 0, or singlehandedly promote a candidate by giving a 100. But it did a whole lot more.
Notice some strange side effects. In the Olympics, everybody loses their highest and lowest score. In this system, I might lose only my highest, while my opponent might lose only her lowest. That means even if we have similar averages to begin with, they can be skewed apart by the point system.
Here's how bad it gets. Even if every judge gives candidate A higher points than candidate B, there's no guarantee that A will win. The system can overturn a unanimous jury. Check it out:
Candidate Boo's Scores: 60, 71, 80 (average = 70.33)
Candidate Yay's Scores: 74, 75, 95 (average = 81.33)
Each judge's higher score is boldfaced. Notice that all three judges give better scores to Yay. But since Boo's lowest score is eliminated, and Yay's highest score is eliminated, the final averages are:
Boo: 75.5
Yay: 74.5
Boo, wins in spite of every judge prefering Yay. (For simplicity, the effect is shown for 3 judges. Its also possible with 6 judges, as was the case with the competition)
This is just one example, but it turns out that the skewing follows this pattern in systematic and predictable ways:
Remember that (especially in an international jury) some judges will tend to judge more generously (80 is bad, 100 is good), than others (70 is bad, 90 is good). The most generous judge is likely to have his grades eliminated when he is pleased (high-generous looks extreme, low-generous looks average), while the strictest judge is more likely to have his grades eliminated when he is displeased (low-strict looks extreme, high-strict looks average). THIS MEANS THE BEST PERFORMERS WILL USUALLY HAVE THEIR MOST GENEROUS GRADES ELIMINATED, WHEREAS THE WORST PERFORMERS WILL USUALLY HAVE THEIR MOST STRICT GRADES ELIMINATED. Poor performers get a boost, good performers are handicapped. And the magnitude of those boosts and handicaps can easily overwhelm whatever original differences there were in the averages.
The event unfolds:
I drew lots for playing order with the rest of the candidates on Wednesday, the first day of the competition.
Over the next 2 days, we each got 20 minutes, 3 pieces, to try to prove our way into the finals. Most of us listened together from the terrace of a cafe below. We were all pretty friendly in spite of the stakes.
When the 5 finalists were announced, I was among them.
The finals went for 2 nights. I played some of my best playing on Saturday, the first night of the finals. Nobody could touch me. The second night (Sunday) everybody rallied and I was a bit less on top of my game. I did fine, and some people still thought I was the best, but most said that we were all about even that night.
When the awards ceremony finally began, several influential carillon teachers made their way towards the area where I was standing. I asked one if that was a good sign. He said that it ought to be.
But the final awards ceremony read like a random number generator. At the time, we didn't understand why. We didn't know anything about the scoring system. The crowd fell silent when my name was announced prematurely. I watched the news footage of the event and they showed the applause after the winner was announced. People were clapping, but they didn't look happy.
Over the next two weeks, things gradually surfaced. In the beginning I figured I just screwed myself by not playing at my top level the last night. But a day or two later I found out that the music committee did an analysis of the judges' points, and that by several reasonable and standard scoring systems, I would have been the winner. Most points. Most judges giving me the top ranking. Etc.
There was a question of whether anything could be done. And I wouldn't have been the only one to benefit from a revision of the results. The first place winner was taking a lot of heat and criticism from this contest. A lot of people were saying he didn't belong in the finals, his playing was awful, etc. They were exaggerating out of anger at the results, but it wasn't doing him much good anyways. A legitimate final ranking would've helped him out just as much, perhaps even more than me.
The music committee issued an apology to me, and I told them that they hadn't really done anything wrong – none of us could have seen this coming. In all fairness, the scoring system works when judges have the same standards for grading, and its still being used in many Belgian conservatories.
A couple weeks later I found out that the music committee was planning on publishing an article in every major carillon publication explaining what went wrong and giving the final rankings as they would have been under a normal scoring algorithm. They couldn't risk the bad publicity of a public overturn, but at least the carillon community would know. That was nice to hear. But when the letter came out, there were no rankings listed, and if you were skimming the article, you probably wouldn't even catch that the results indeed had been illegitimate.
I don't know if I would have been satisfied had they published the results as they would've been. Probably more satisfied. A lot of what I wanted was to be able to say without complication, I won this thing. And some of what I wanted was to have the glory that night. To get the trophy, to shake all the hands, to be interviewed on the news, to be able to receive full compliments unmarred by awkward condolences, to be on top of the world that evening.
Still, I can't complain too much. People who were there knew what happened, and reputation is worth more than a trophy.
It was in the midst of the competition that my playing made a big leap forward. And the process showed me that I could do things I wouldn't have expected. I reached a new level in my musicianship simply by giving the performance I did on Saturday and thereby realizing that I could work at that level.
And on some level, while it would have been nice to get the first place prize, there's a certain reality that comes through in this version. I still have room to grow, up to go and I still ought to stick around and figure out how to be able to play all my performances at Saturday's level. The fact that nobody managed to play better than me is in some sense unimportant compared to recognizing where I came out on the scale of my own potential. So the result has some good in it -- not to denigrate my performance – just as a reminder to keep going.
Its also a reminder of the fact that people do get screwed by bad mathematical reasoning working its way into the systems that we use to make group decisions – in bigger and more systematic ways than in this contest. And I might be someone who can help do something about that.
I've moved on at this point. I had figured that after this competition I'd return to some sort of office setting, but I've already gotten two extensions on the carillon adventure – a gig at a renaissance fair in Texas (www.castinbronze.com), and then a Christmas gig on the west coast. So music is going to keep me fully occupied for at least a bit more.
Here's a news clip of the event. You'll hear my name mentioned as the favorite to win (Jonathan Lehrer, whose subtle playing has won the hearts of the crowd), and you'll hear a short snippet of me on the tower. If you look closely when the winner is announed, you can see it all in the expressions of the crowd. A strange mood hung over the room as the crowd cleared out. Almost everybody came over to express their surprise and disappointment. The winner is a nice guy and he certainly didn't ask for these results -- he just showed up and played like the rest of us. So ended the 6th International Fabiola Contest.
While it would be a nice touch on the resume to have the official recognition,
-I, and my playing, have benefited loads from my participation.
-I've established my name on the European continent.
-I got an auxilliary prize for best interpretation of a Flemish work. Not bad for a foreigner.
-Because I got 2 prizes, I ended up taking home almost as much cash as I would've with just 1st place.
-Most importantly, I enjoyed this more than I ever thought I could. The waiting and the preparation were sometimes quite grueling, but overall I found myself very happy during this past week. I feel incredibly lucky to have been able to participate in this event. The awards ceremony was an unfortunate ending, but it wasn't the important part.
Many thanks to all who have supported me and been there via phone, skype, or in spirit to help me through this contest.
I hope you've enjoyed following my time out here, and I look forward to seeing you soon!
PS Here's one of my pieces from the finals, recorded from within the tower.
No comments:
Post a Comment