Review of Ratings System - submission from Victoria
Ratings A CASPA Submission
"I think the ranking is just an algorithm. I would much rather win the tournament than get back to No 1 in the world. "My focus is going out and having a solid round of golf and the rankings will be the rankings”
Scottie Scheffler, Golfer
In many ways, Scheffler is correct here; the big “however” here is that that statement, whilst true, does not apply necessarily to Scrabble. Ratings, as a topic in Scrabble, is something very close to many players’ hearts. Ratings go on to determine the rankings.
From a Scrabble perspective, ratings are qualifiers for events around the country and globe. Ratings are the way a player can evaluate their day and, in some ways, can motivate or demotivate players to keep playing.
It is therefore proper to ask questions of our ratings system continuously, to ensure that it remains relevant to how the game is played.
The ratings formula was last changed in 2010.
Scrabble has changed a lot since then and there is a lack of clarity that the ratings system introduced then does not stack up to what events go into the ratings themselves.
This submission looks at this.
What is the background and current situation?
This is from the Scrabble Australia website.
The rating system is essentially the Elo rating system. At the heart is a rule which models the probability that a higher rated player will beat a lower rated player. The curve is a function of the difference in ratings. Prior to Dec 1, 2006, Australia used a logistic curve with a slope parameter of 172. The formula (expressed in spreadsheet language) is 1/(1 + EXP(-x/172)). However checking the actual win rates using many games shows that the curve is wrong.
For instance when the ratings difference is 300, the logistic formula gives the higher rated player an 85% chance of winning, but the data show that they win only 74% of such games.
In 2010 we introduced a straight line rule which better accords with the prior data. Your percent chance of winning is fifty plus one twelfth of the rating difference (but capped at 95%).
The practical consequences are that the higher rated players in a section will find it fairer in maintaining their rating, or being able to progress to the next higher sectionif they can prove their worth.
It was expected that ratings would slowly change as the result of the change, and it is possible that we may again get a mismatch between the observed win proportions and the modelled probability. Monitoring has occurred from time to time, but no changes have been recommended.
Here is an example of how you can calculate your rating change.
Example: You are rated 1453. You win 5 out of 7 games against opponents whose average rating is 1393.
You are 1453- 1393 = 60 points higher on average.
Your calculated probability of winning a game is on average = 50 + 1/12 of 60 = 55%.
You would be expected to win 55% of the 7 games, ie 3.85 games.
You actually won 5 games, which is +1.15 games better than expected.
A multiplier of 20 usually applies1. Your rating gain is 1.15 x 20 = 23 points
A multiplier will effectively be larger than 20 for a provisional player. Players new to our rating system are deemed provisional until they have 50 rated games. Their first few tournaments are in effect treated as one big tournament. This smooths out fluctuations and will allow them to rise more rapidly if they improve during the provisional period.
There is no further clarification on when the monitoring was done, if it continues, or even whether the 74% figure quoted is still accurate.
What is the situation now?
Here are the top 20 players, as of 18 March. The bottom line outlines average rating, total games played, total wins, and overall success rate.
What was the tournament make up in 2010 compared to now?
Going to the archives: In 2010, NSW had over 30 tournaments occurring. In 2022, 24.
At the 2010 Janboree, there were 77 players. At the 2023, 47.
As a trend, the number of players taking part has dropped.
This situation would not be an uncommon one across most of Australia. This matches the membership decline in CASPA over that period.
With less people playing, and less events overall, a system brought in to reflect 2010 Scrabble situations needs to be re-evaluated to fit current conditions.
What are some of the issues that arise in the current rating system?
Impacts of a “bad” day
A single bad day has a dramatic effect on a player’s rating. It is not uncommon for a player to lose anywhere between 30-60 points in a tournament if the results do not go that way. This makes a lot of sense viewed at the micro level.
However, one bad day does not suddenly mean your actual playing standard drops.
How easy is it to regain those points in the current system?
The reason this is important is that ratings define rankings. At the top of the rankings, given the make up, one bad day can still mean you retain a near same position in the rankings.
Eg Naween Fernando at the 2022 Trans-Tasman. Naween comes 2nd, loses 52 ratings points and one position in the national rankings.
Losing 52 ratings points in an environment where the average player rating is 1693 for the top 20 in the state means the likelihood of making that back up again quickly is difficult.
This is one example only. It is not isolated and I am confident more will pop up as more analysis in the stats occurs.
Division Discrepancies
At a recent 5 round tournament in Victoria, a player with a 1300 rating was placed bottom of B. That division had only one player of similar rating whilst everyone else had a rating over 100 points higher at a minimum.
The ratings algorithm meant that they had to win 3 games to get positive ratings on the day (because break even was between 2 and 3).
At the same tournament, if they had been placed top of C, they would have also had to win 3 games, yet they were top seed.
Previously, 5 round events were not really around when the ratings formula was set up. Victoria and SA now have them on the calendar.
The minimum for rating is 4.
In summary
The inferences throughout are as follows:
The system was introduced to solve a problem in 2010
Tournament numbers, structures, and frequencies have changed
There are some flaws in the system that are exacerbated by the changes.
Where to from here?
All I ask is that we set up a group to investigate the rating structure to see that it is fit for purpose for 2023 and beyond.
The straight line method outlined, on its face, has faults.
Do other systems have a more reflective outcome eg would moving to rolling performance averages which takes away one bad event after a period and rewards consistency be effective?
Would it keep people interested in playing, especially those who are ratings conscious?
Do we move to a system which looks at rating based on a division profile (eg average rating) in a way like golf does for ratings points based on event performance.
Or is the current system, when stress tested, provide a near enough equivalency which, despite its faults, is still relevant? Or is it still good, but we need to tweak a bit of the underlying formula and, if so, which bit(s)?
Motion for submission
That CASPA set up a working group with terms of reference to investigate the current viability of the ratings algorithm and to compare it with other ratings algorithms to determine whether to continue with the existing ratings system or to move to another formula. This group is to report back to the 2024 CASPA AGM with their findings.
Moved: Nick Ivanovski.
Nick explained that the situation had arisen where some players are so under-rated that if they did badly in one tournament, they can’t make it up in the same time span. A player had recently spent improving rating up 100 points, having one bad tournament and losing 30 - play hasn’t changed really. It might take 2 or 3 tournaments to get them back. It is an aspect of discouragement to enter tournaments. We want them to keep coming back. Various international systems might use also margins. Last time was 2006 for review. We are playing different style tournaments since 2006.
It was agreed that the system might need to be reviewed and Barry explained that Australia effectively takes performance rating on day plus average on past say 10 tournaments - in Aust system exponential, in UK linear. UK would make established players keep ratings better, but up and coming players would be harder to get up. One way rewards established players and the other punishes upcoming players.
Paul Richards would like feedback points to be reinstated into the rating system. Or to round up on all rating point calculations.
MOTION: THAT THE COORDINATOR SET-UP A COMMITTEE TO REVIEW RATINGS WITH A REPORT BEING PRESENTED TO CASPA BY 30 SEPTEMBER, 2023.
Moved: Nick Ivanovski Seconded: Paul Richards Carried
(Interested people?? - Barry Harridge, Trevor Halsall, Benjamin Lee)
CASPA Promotions - Magnets, Bookmarks, etc. -
Paul Richards said that CASPA needed products to hand out at events. As CASPA did not have a generic supply, do States produce their own? The States should be sharing what they produce - e.g. bookmarks. Nick Ivanovski was of the opinion that word-centric resources did not lend to people “enjoying” the game. E.g. only produce 2-letter word list. Welcome packs were a good idea.
Scrabble 75th anniversary - Mattel do not seem to have done any logo, etc. no big broo-ha. Victoria would be approaching Mattel for the Victorian Championships in June as it was a CSW21 tournament.
CASPA levy - no discussion.
THAT DAVID ELDAR AND THE REPRESENTATIVES IN THE AUSTRALIAN QUOTA OF 13 FOR WESPAC 2023 BE SUBSIDISED $A100 EACH UPON REQUEST.
Moved: Nick Ivanovski Seconded: Jane Taylor Carried.
THAT A LETTER AND A MOTION FOR THE WESPA BGM BE SENT TO WESPA HIGHLIGHTING THE ANGST THE EXPURGATION HAS CAUSED AND ASK THE NEXT WORD LIST UPDATE BE DONE BY COLLINS WITH NO INTERFERENCE FROM MATTEL.
Moved: Paul Richards Seconded: Jane Taylor Carried WA against
And that an agreed process be established that any modifications is established.
Moved Jane Taylor: No seconder.
Paul - We can refer to the uproar proposals to modify other things such as Roald Dahl books has caused.) Qld committee asked for this and doesn’t think Mattel know how much angst was caused. Maybe nothing will come of it, but don’t know if Mattel locally know about it.
Bob - WESPA caused the angst. Mattel said what they had to say. John - WESPA forced ultimatum which was not necessary. Mattel said that the word “Scrabble” would need to be removed if the expurgation was not done.
Nick - Suggested that a better way would be to write to WESPA regarding angst and we want make aware and is continuing and that WESPA communicates it with Mattel and Collins. We need to be cautious is that around time of loss of licence at time of Sports Illustrated article which hit NASPA where it hurt. Whether that was straw that broke the camel’s back.
John - Mattel holds the trademark - it is held over our heads.
Women’s Tournament - Jane Taylor - inaugural women’s tournament - reasonably moderate attendance - maybe annual and from state to state and tacked on to a 2-day tournament. NSW might do it with the CSIM in 2024. Meeting suggested that SA & NSW can work out and maybe do it alternatively. Not a matter for CASPA to need to consider.
Nationals - Format for 2023/24 - (divisions)
2023 - maximum of pairings will be 3 or 2. For 2-day event 2 is right, then 3 is right for a 3-day event.
2024 - wait for end of this tournament. Survey players and take back to Formats committee.
Tightau programme - submission from S.A. -
Complaint from Carmel Dodd On Tue, 14 Mar 2023, 11:16 pm Carmel Dodd, <carmel@triskills.com.au>
wrote: Hi Barry and Paul Barry, please forward to Paul as I don’t have his
email address After a night of consideration I am lodging an official complaint
regarding the Draw for the final game of the tournament. I believe I
was unfairly treated in that my chance of being placed and the
opportunity for prizemoney was diminished by 'the program’. I asked
Paul before the last round did he think there would be repeats he said
'let’s see what the program does’. The 'program’ appears flawed if it
is unable to identify that there was already a winner, and as there was
a BYE for consideration, that winner should have been allocated the
BYE and then the other players can fairly play for the placings. I understand that the new 'Tight Australian Draw’ was adopted and the
only repeat that I am aware of was my repeat with Joanne in the final
game. Regardless of the BYE, it still seems unfair that one player (not
the second highest rated player in the tourney) repeats against the
highest rated player. All the other placed players played 4 of their
games against players rated below 1305. Only THREE of my opponents
were rated below 1305. I would like to have an explanation of who was
advantaged/ disadvantaged by having just the ONE repeat in the last
game. The same Draw was used in the Vic Champs last year I have attended
20 odd Vic Champs and that was the only tourney I can remember where I
came home feeling very disillusioned with Scrabble, I felt the same
yesterday. Yours sincerely Carmel Dodd (attached)
Following completion of the 2023 Australian Scrabble Championship, a survey would be undertaken to ascertain the satisfaction with the programme.
WESPA Executive - Carol Johnsen is happy to continue as CASPA representative - Any other nominations?
MOTION: THAT CAROL JOHNSEN CONTINUES AS CASPA REPRESENTATIVE ON THE WESPA COMMITTEE.
Moved: Nick Ivanovski Seconded: Paul Richards Carried
Funding for:
-
Australian Championship (currently $3000)
-
Masters /State Challenge (currently $3000)