RoRoRo's predictions and pronouncements - how many "upsets"

I've just noticed that since RoRoRo started recording how often he asserts that results were "upsets", "major upsets", or "astonishing", two things have happened:

1) a minor typo meant that he never actually said astonishing results were astonishing, sorry to those victors who missed their glory moments.

2) he's tallied over 5000 such game results, so time to decide if the cut-offs are in the right place is long overdue (I'd forgotten about it to be honest):

nothing of note: 4867 = 96.78%

simple upset: 109 = 2.17%

major upset: 45 = 0.89%

astonishing: 8 = 0.16%

I know people were saying that RoRoRo was getting a bit over-excited by upsets, but, to be honest, it doesn't look that way at all if he's only drawing attention to less than one game in 30. So I don't think much tweaking is in order.

I see room for a minor tweak, but won't say what it is until later as I don't want to plant ideas in your head, so feel free to just say what comes to mind.

Apart from the tweak to get the astonishment message printed again - that's just been fixed!

IIRC, the rating differences you're using should, in theory, yield very different results than those you've actually recorded.

Assuming that the vast majority of games are played to completion rather than timing out, I think there should be substantially more results classified as simple upsets rather than as nothing of note.

Regarding your first comment - please, do go on...

Regarding your second comment, all games that end after the 4rd move are counted. That was a simple filter to remove (a) immediate "don't want to play" resignations by either player, and (b) immediate "now the ratings points count so I can transfer points to my sock-puppet" resignations by either player.

And holistically - you're overlooking something...

Outside of Chess, and therefore as a total of all games, I think MiM has never accounted for much of the traffic on LG, if that's what you think I'm overlooking. Is that not true?

To go on about my first comment, you'll have to refresh my memory regarding the rating point differences required to earn your various appellations.

I think what you're overlooking is the fact that most games are played between opponents whose ratings are similar, which puts these games in the "nothing unusual" category. You need to know the distribution of the rating difference of games recorded in order to say anything about what the numbers should be "in theory". If ALL games were played between players of rather similar rating, then, obviously, ALL results would be "nothing unusual" both in theory and practice (barring any hickups in RoRoRo).

I'm not forgetting that, but having only 3% as being notable at all would imply some combination of fewer games with significant rating differences than I think likely and fewer upsets than I would think likely, especially since some upsets would occur when new players join and their initial rating doesn't reflect their ability.

OTOH, RoRoRo probably isn't tracking many games outside of the top tournaments, so maybe there really are very few games in the potential upset range. I know I used to regularly get message-reported predictions in my lower championship sections, But I suppose I'm unusual in that regard.

DVD - MitM is indeed relatively unimportant, I never[*] attempted to cater for that.

Marius - you're a damn smart guy! It wasn't an insight I gained until I started gathering the statistics X months ago (but once I'd seen it it clicked, make immediate sense in retrospect)

[\* RoRoRo does have a list of players which he supsects of cheating (because of reports by a range of people on these fora), and will attempt to detect MitM games involving them. At the moment, he does nothing with that information.]

DVD - I can see why you think that, as it makes perfect sense. However, RoRoRo has been set to only post predictions for the absolute top echelons to the fora. Currently it looks like 4 fall into that class, but he's supplying 38 other leagues' predictions by private messages too.

So what percentage of games fall into the possible "upset", possible "major upset", and "possible astonishing" categories?

Assuming I interpret that question correctly - what proportions of games are between players of sufficiently different rating such that an upset would trigger the three different comments - I don't have that data. I only have a tally of how much he's said each comment.

I can gather such data. It would be different in each game. Deeper games would have more possibility, simply because the standard deviation is so much wider. One has to remember that the predictions RoRoRo makes are only for a small subset of the activity on the site, and therefore only a small proportion of the players. Even if I gathered such data it might contain skews that I could know. I'm prepared to, however, so I'll add that soon.

I'm wondering if I want to effectively gather a scatter-graph of p1 vs. p2 ratings, as it might be that the middling masses flood the data with pools of like-rated opponents, but some small number of experts put their ratings at risk quite frequently. So different people are differently-susceptible.

To keep the data I have to store small, I guess I can tally the following for each game type:

{ number of games, sum(rating1), sum(rating1^2), sum(rating2), sum(rating2^2), sum(rating1*rating2) }

And from that I'll be able to work out the mean & standard deviation of the ratings difference, as well as a lot of other stats should I want them. (I'm hoping there's no skew, but I wonder if there's a kurtosis, hmmm...)

Of course, Richard could probably answer this with a simple query!

What Richard could answer would be for all games (of a given type) on the site. What I'm asking is for games that RoRoRo tracks; the ones that form the denominator in your percentages at the top of this thread.

No matter what, I don't think it makes sense to call something an upset if the favored player could not be expected to win some minimum threshold of the time, say 2 out of 3. So to know whether tweaks are in order, we need more data.

Ooh, blimey, I've just looked at his code, and I no longer understand what it's trying to do, it's not just a simple "this was likely less than 1/5 of the time" or anything like that (as it has to cope with draws, and such probabilities are hard or impossible to know, and game dependent). I'll see if I can pluck out a few actual numerical examples of corner cases that show where the regions are.

However, in reality what matters most is whether people like the (implementation of) the feature. I like the idea of an encouraging attaboy, and I hope the recipients do too. If it is noticed that too many or too few are being handed out, then the cutoffs are easy to tweak.