VRC leagues rank by win%. Is this an issue?

This thread is addressed to anyone who has organised or participated in a VRC league.

Here is the document that describes how leagues should be run:
League Play 2013-2014

The document says that teams are to be ranked by win%, and that teams must compete in 75% of the available matches in order to be considered for the finals.

Win% ranking is not ideal in a competition where different teams play different numbers of matches. For one thing, teams who play fewer matches will have greater variance in outcomes. More importantly though, teams are advantaged by choosing not to compete if after playing the minimum number of matches they have a higher than expected win%. These teams could choose to compete under a different number, but not every team has that option and some might choose to miss events if they think that competing might drag them down in the rankings (potentially pulling them below the threshold that would qualify them for regionals/states/nationals). Having teams sit out because of a quirk in the ranking system is clearly not good for the competition.

My question to you is: does this happen? Are you aware of any teams who have chosen not to compete at the end of the season because they were worried their ranking might slip? And if this hasn’t been an issue in your competitions with the minimum number of matches at 75%, how much of an issue do you think it would be if that limit were 50%? or 25%?

There are ways of ranking teams that would make this less of an issue. One way is described here and is based on compensating for the greater uncertainty associated with teams who have played fewer matches:
http://www.evanmiller.org/how-not-to-sort-by-average-rating.html
This rewards teams who have maintained the same win% over more matches, because it’s more likely that their score accurately represents their ability. By changing the confidence parameter you change the relative importance of matches played and win%.

You can’t change the fact that teams whose current rating is higher than their expected full season rating are advantaged by dropping out. What you can do is change the way those ratings are calculated so that more matches played increases the expected rating, making that situation less likely.

Since VRC in New Zealand has become too big for every team who wants to to attend Nationals, Kiwibots is planning to begin running a league in each region to decide which teams qualify.

Kiwibots have tentatively scheduled 12 ranking scrimmages in Auckland between June this year and February next year. Since scrimmages in Auckland are frequent, the number of teams at an average scrimmage has historically been less than 75% of the number of available spots allocated to Auckland at Nationals. This will mean that the required number of matches played in order to be ranked may have to be reduced from 75% of the maximum to something lower. This would increase the probability of teams completing their minimum number of matches with much higher than expected win% scores, which is why I’m asking what your experience has been with what these teams choose to do in that situation.

Yes, this does happen. We had a league where we had to attend 4 out of the 6 meetings, but could attend more. One of our teams was 16-2 and had played their 4 weeks in the first 5 weeks, so they did not play in the last week because they did not need to. My team did bad in the first 4 meetings, so we decided to play a 5th week to possibly raise our ranking.

The rank by win % is needed because if the WP system was used, teams that would play more matches would be ranked higher. At our leage we played between 4 and 6 matches a night, so your total matches depended on what nights you sent to, and if you decided to play more weeks. An example of this is that we were 11-11 and if WP’s were used, we would have been ranked over a 11-6 team because we played an extra week. This is why the win % is needed.

Here is a link to the rankings of our league so that you can see how WP differ from win %: http://www.robotevents.com/fox-valley-vrc-league.html

This is what I was concerned about. Thanks.

I’m not suggesting using WP. That would be an awful way to do it. I’m suggesting using a system that is based off win%, but that accounts for the different number of matches played by different teams.

In general, my opinion is that an ideal ranking system would reward teams both for playing more matches and for winning a greater proportion of them. The exact mathematical formulation could vary, but the page I linked describes one possibility.

It may be that your team’s experience is rare, or that the benefits of an easy to understand system outweigh its flaws, but that’s what I created this thread to find out.

Orginally at ouor league it was that you can ONLY attend 4 of the 6 dates but this got changed as the league progressed. You could make it so that you only play X times and only have Y matches per night so that all teams would have the same number of matches.

Then the problem then becomes that a team may try to avoid some nights when a good team is there so they would have a better winning chance.

Hi Oliver

This participation weighting seems to me to be a pretty good solution.

Another thing to consider, and something I have seen, is that some teams, particularly newer teams, start the season with a robot that is not very “flash”, so could be doing very poorly in terms of win % for the majority of the season. Through dedicated development, a team can improve to the point of being very competitive by Nationals, but would be penalised for early season performance. A possible solution would be to give progressively greater weighting to event results throughout the season, so that the last event before Nationals would carry much higher weighting than the first event.

I guess it also depends on what your philosophy is for deciding which teams should qualify for Nationals. Should it be the best performing teams “on the field” that qualify? My observation is that Kiwibots do not value competition performance very highly compared to other educational aspects of VEX robotics.

I checked back through Robot Events on team numbers attending New Zealand Nationals.

2010 48 teams
2011 43 teams
2012 50 teams
2013 61 teams
2014 52 teams

You would be a brave person to claim that the numbers are trending up. Perhaps the claims of growth are based on some other data?

Paul

Oliver,

Interesting thread. Drop me a line sometime and I will explain how we are intending to get around the issues you raised. The reason why we are calling what we will be running in NZ a ladder and not a league is to make a distinction from the league system you quoted.

We have avoided publishing anything yet as there are a few things to be ironed out before going public with it. First step is to get venues scheduled for the events and then work out a participation schedule for teams.

@Paul

This would not work as we will not be able to cater for every team to attend every event. Weighting would only benefit teams that were allocated those later events. We also want to encourage teams to participate fully from the start where possible.

@Paul

This statement is very negative. We value the educational aspects of VEX more highly than competition performance as we want to develop well rounded team players to support the future knowledge economy of New Zealand. We still value competition performance very highly just not as highly as all the other educational aspectes escpecially when it is a team of one.

@Paul

I am that brave man! I am very confused as to why you felt it was even necessary to put this into your reply :confused: Fact: The number of robots and teams in New Zealand is growing faster and faster. Fact: Teams outside Auckland are growing stronger. Fact: We can cater for a Maximum of 60 teams at the Nationals. Fact: We had 65 teams registered for the nationals in 2014, 13 of which dropped out less than one week before the event. Fact: Good managers plan for the future before reaching a crisis point. We are good managers and refuse to leave things to chance. Fact: We need to have a fair way of selecting who can attend the nationals. Fact: It is better to introduce something when you have room and time to make adjustments rather thatn when circumstances are totally out of control.

I’m probably going to end up dabbling into something I probably shouldn’t…
but I feel it’s unfair to specifically target out teams of one, seemingly to suggest they aren’t being “educated” as well as teams of multiple people. Seeing that they are doing everything from mechanical design to software, saying they aren’t well rounded seems unfair. More specifically, Vex as a whole is a competition, so the awards (especially world-qualifying awards) should be given to those who compete and win. Awards such as a Educate, Inspire, and Mentor of the Year award are examples of those who should be given to those with educational programs that are deemed notable. It should also be noted that these “teams of one” (2941, 2915, 1103, to list a few) have effected and inspired likely hundreds of teams to push to a higher level and reach for a deeper understanding of robotics. That’s my two cents on that topic.

On topic, I would be interested if you averaged your win% out of say 6 tournaments, but say take only the top 4 percentages. Something of that degree would be nice, as it would let teams with poor early seasons to recover, and encourage those who are ahead to try to perform better in the last few tournaments? Just an idea.

I have also wondered why New Zealand discourages one man teams. 2915A and 2941A did a huge amount of good for the Vex community, I know they inspired me. And I think that a large percentage of successful vex teams, while not officially “one man teams,” are successful because of one person.

On topic:

Using data from 3 of the pre-nationals scrimmages, I was able to compile what an early season rankings might look like. As much as possible I tried to ignore that the better ranked teams attended more scrimmages.

My Formula is based of WP% rather than win% purely because it takes ties into account, instead of just wins- making it slightly fairer in my opinion. It uses a weighting based system, like what is used in most University papers. It takes into account WP% as well as Seasonal Match Attendance%. Seeing as WP’s are deemed more important than match attendance I weighted it 65:35, however this can obviously be changed.

Formula: (WP% x 0.65) + (0.35 x (% matches attended throughout season))

On this type of system it encourages teams to simply attend more scrimmages by enticing them with ‘free’ percentage points, even if they don’t win any matches. Teams that do not attend all of the scrimmages start to miss out on a fair amount of ranking points towards the final percentage. For teams that do attend every scrimmage, they can choose to use their best scrimmages towards the calculation of their final WP%, while getting all 35 points available from attendance. If every scrimmage was organised to have 8 games per team, then throughout the 12 scrimmages there will be 96 possible matches per team. If 7 of these scrimmages ‘count’ towards the final ranking, then 56 matches from 7 scrimmages will count towards the WP%. Essentially match attendance is calculated throughout the season, while WP% is calculated only from the team’s top scrimmages.

For example:

A team attends 10 scrimmages throughout the season. Results are as follows (win-loss-tie):

  1. 6-2-0
  2. 5-2-1
  3. 7-0-1
  4. 4-4-0
  5. 5-3-0
  6. 6-2-0
  7. 8-0-0
  8. 8-0-0
  9. 5-3-0
  10. 7-0-1

From these results the team decides they do not want to use results from scrimmage 4, 5 and 9. This means their standings throughout the season is 47-6-3. This gives them 97 out of 112 possible WP equalling 84%.

Since they attended 10 scrimmages however, they will get 80 attendance points out of the possible 96, or 83% of matches.

So: (84%0.65) + (0.3583%) = 83.65/100

Although it does not account for all the potential problems, it is quite simple to understand and could be made public through something as simple as Google Docs. It does have a flaw that it favours teams that can actually attend all the scrimmages, but as I stated before, the teams that attend the most scrimmage were better placed in rankings anyway.
Food for thought.

Off Topic:

There is data to support that the number of teams in New Zealand is increasing. As Chris said, 13 teams pulled out of Nationals that were originally registered. There is actually a reason it was capped :stuck_out_tongue:

You really, really are…

The latest Kiwibots blog post mentioned ranking all teams based on the same number of scrimmages, and taking only a team’s best scrimmages if they have attended more than the minimum. To be perfectly honest, I read the email pretty quickly when it first arrived and I didn’t notice that you had said this until I read over it again earlier today. This solution seems like a very practical one. If there’s more to it though, I’d love to hear it. I’ve dropped you a line, like you said.

I think this thread is still relevant to league play in general though, even if the issue has already been solved for the NZ Nationals ladders.

I hope you’ll pardon my intrusion here. I really don’t think Paul was trying to have a go at Kiwibots. Yes Otumoetai would have benefited over the years from Kiwibots having a more competition-oriented approach, but I think Paul’s statement is pretty neutral under the circumstances and I think it actually reads pretty similarly to your own.

To be fair, it’s not that easy for teams to know how many teams compete in each region. Precise information on things like the long term trends in the number of active NZ VEX teams is hard to come by.

You’re in Massachusetts, correct? I don’t think you have to be worried that anyone from New Zealand will hold a grudge against you because of anything you say here ;).

The RECF doesn’t really take a position on how awards should be given out. They do say that the Excellence Award is (at least partially) a “Robot Performance” award, but even that could be seen as more of a guideline.

I don’t think you need to allow teams to choose their best scrimmages AND reward teams with ranking points for attending. Just having one or the other would be sufficient, and I think letting teams choose the best scrimmages is the better option. It makes it impossible for a team to lose points by attending a voluntary scrimmage (which attendance points don’t do) and it feels like less of a penalty for teams who can’t make it to a scrimmage. Even if the expected points gain by attending a scrimmage is greater under a pick-the-best system than under an attendance points system, the attendance points system would still make it *feel *more like the team is slipping behind if they miss a scrimmage.

I’m not super keen on these sorts of attempts to preempt discussions that deserve to be had, even if they are a bit off-topic.

Our area’s dabbling with a league in Gateway had this exact issue. I don’t know of a good answer though.

It’s tough in the initial stages of a league but as you get more years you could enforce the miniumum number of events rule.

If we had enforced a minimum number of events, the league numbers would have collapsed without enough teams to hold a 24 team elimination. So scouting becomes important.

(sorry this American football reference may be lost on the Kiwi folks)
Downside to win% is kind of like in the NFL, where week 17 - right before playoffs - is a mess with teams putting 2nd string players out there to preserve them for the playoffs or if previously eliminated, send in 2nd stringers since there’s no hope of making the playoffs. That’s why most fantasy football leagues end at week 16 when there is less chance of stars sitting. Maybe a hypothetical analogy for New Zealand is for cricket world cup qualifying or group stages, say a final match of NZ vs Australia where you might sit your stars a day since you have it wrapped the group #1 spot already.

In a Vex league, if you have the minimum number of events under your belt and your win % is good because say becasue you got done your robot before everyone else even got building early in the season, you elect to sit out the last 1-2 tournaments. Similarly those that are mathematically eliminated might might stay home too. So how fun would those last events be?

(In the US, baseball is even worse with poor quality for eliminated teams in September. NBA basketball seems to have entire seasons now with intentional poor play like the Philadelphia 76ers for a possible #1 draft pick)