We have all had times were we felt as if we deserved a higher rank on the leaderboard than we received because the schedule was not in our favor. I propose that vex should implement a new system for ranking robots through the qualifying rounds. A new system such as CCWM that truly judges a robot based on what they have contributed could help us to ensure that the robot that is truly deserving of the top rank gets it. I would like to know everyone’s thoughts on overhauling the current system and what you think should replace it.
Well, “the robot that is truly deserving” is the robot that does not lose. Rankings is minor compared to scouting and developing relationships with potential alliance partner.
6 Likes
in my experience, the rankings this year have been a bit screwy compared to previous years. I attribute this mostly to two things, how crucial the goal rush is to winning a match, and how you don’t just need a good alliance partner to win a difficult match, you need a really good alliance who can rush faster than your opponents. And also the AWP task does not align with winning matches at all, because in close matches, if a team were to prioritize the awp, they would most likely lose the match because they neglected the goal rush during auton. But teams that have more fortunate schedules have better odds of being able to both win matches, and get the awp. So while early in the season when goal rush wasn’t as important, awp did provide an excellent way for good teams to rank higher, at this stage in the game all it appears to be doing is amplifying the luck aspect of qualification schedules.
I don’t think there is any fundamental flaw in the ranking system, even though luck is undeniably a significant part of it, but I think in the future the games should be less requiring of a good alliance partner to win, and the awp should be crafted in a way that makes it align well with the task of winning.
21 Likes
This year was also especially annoying in the ranking system early on when many teams got disqualified for rule infractions we didn’t even know existed. This happened to us twice in just the first tournament.
Best way to reduce the influence of luck is more matches. Best way to get more matches is for competitors to help keep cycle times fast.
I like that the AWP is mis-aligned with putting an alliance in position to win the match. Makes for interesting strategic decisions and encourages teams to have multiple auton routines (at least 1 for quals and 1 for elims).
For as “easy” as it seems to accomplish, the AWP success rate for the season was well below 10%. I know when it came out, I thought that it would be too easy and am happy that it was not as common as I thought it would be.
Compared to Change Up, I think Tipping Point allows really good teams to win what are effectively 1v2 matches.
Of all the things in VRC to change, I think the ranking system is currently in least need of change. In nearly all cases, the “most capable” robot alliances are winning tournaments
8 Likes
I guess you’re right. Even though sometimes undeserving teams do end up with high ranks it is usually the best alliance that ends up winning the competition.
I did too at the start of the season, when you could solo the awp yourself and still win matches. But now, whether or not you can get the AWP doesn’t really come down to a strategic decision anymore, it’s mostly based on, is my schedule good enough where I can either solo the awp and still win the match, or can my partner do their half while I do mine, and still win the match? and in most cases, I’ve found that the answer is no. Because if you do the awp before rushing neutral goals, you will almost certainly lose the match. And if you rush the neutrals first, and then try to do the half awp afterwards, you rely on a partner to succeed in their half, or no awp.
maybe the strategic choice thing would have worked out a lot better in a game where the most important time wasn’t the first 5 seconds.
4 Likes
In my region of South Dakota I can honestly say that there are a lot of not very good teams here. We obviously have some very good ones like 77000X but also a lot of teams who have very simple robots, some without an auton at all. Here I can say it is very feasible to win the AWP and still go on to win the match.
I have a lot of problems with CCWM. I do not think it should be used for rankings as for many games it is wildly inaccurate.
The last game that CCWM was good was turning point. In turning point, you wanted to score as many points as possible and reduce the other teams score as much as possible. When given the opportunity to score points, it was always worth it. Turning point was a linear scoring game too and OPR (which helps calculate CCWM) only works with linear scoring games.
In tower takeover, CCWM didn’t work, because it was a smart decision to score points for the other team. If you had 4 orange cubes and the other side had 1 orange cube, it was a good idea to score an orange cube in a tower as it give you a net gain of 3 points. However, this action gave your opponents 1 point. This messes up DPR (the other factor used to calculate CCWM).
In change up, CCWM didn’t work because the scoring was no where near linear. There would be extremely close matches in which 1 side had double the score of the other side. Most match scores were in the 20 point range, but if you go an easy match, you could get a score in the 60s or 70s, wildly screwing OPR.
In tipping point, there are 2 big issues. First, you have no control over how many points your opponent scores in the last 30 points. If I leave the other team with alliance 2 goals, they could score 40 points, or 200 points. The second issue, is that it isn’t always the best decision to score more points. If I have a balanced platform, and I am guaranteed to win the match, I won’t elevate my extra goal just in case I mess up. There is nothing to gain and everything to loose. There have been many matches in which I could score more points, but I don’t.
By making rankings based on CCWM, you introduce more luck and shift the focus away from winning a match.
11 Likes
this is probably true for a lot of regions, not so much in southern california. Here, in a fairly balanced high level match, prioritizing awp and winning are almost always mutually exclusive. Which means the usually the only teams that benefit from awp are ones who have a schedule fortunate enough to allow them to do so safely.
4 Likes
Perhaps we differ in what we consider “strategic”, but the logic you walk through above seems like a “strategic risk assessment” to me. Perhaps factor in - if my alliance is unlikely to win this qualification match, can we at least come away with 1 WP rather than 0 by successfully completing the AWP, and teams are able to consider a number of cost-benefit choices.
4 Likes
Just throwing something out there, perhaps one of these two systems would serve as a better alternative. These systems I am throwing out there are true skill http://vrc-data-analysis.com/
and elo https://vexrankings.wixsite.com/home. Here are examples of both of them.
in this case there would be a strategic decision, I agree.
but there is often the situation where you are likely to win the match, but only if you forgo awp in favor of rushing. Your schedule in this case isn’t a bad one, it would be a pretty fair schedule. But it’s not so much in your favor that you can repeatedly prioritize awp, because then you would be sacrificing the win, which isn’t worth it.
3 Likes
I would not recommend using TrueSkill or CCWM as metrics to determine rankings. They are far too opaque. Teams can easily look at their record and match results and make sure that their WP/match, AP, and SP are calculated correctly by hand. Not so with CCWM or TrueSkill. Both CCWM and TrueSkill also need a fair number of matches in order to be meaningful.
Best to focus on how to enable more qualification matches at tournaments than messing with the method of ranking. Under any system, more matches reduces unfavorable schedules and/or luck.
10 Likes
OK, so lets say team ABC and XYZ are equal in skill and quality. ABC has a “favorable” schedule that allows them to prioritize AWP each match, while XYZ has a tougher schedule. So ABC ranks higher than XYZ, assuming they win the same number of matches. Has “luck” played a factor? Of course. Best way to fix that is increase the number of matches ABC and XYZ play which the “law of large numbers” would converge to an overall schedule of similar difficulties for each team.
4 Likes
One thing I think needs to be changed is SP. I feel like there are more favorable ways of measuring strength of schedule than SP, for example an average of your opponents win percentage. Perhaps a metric that also incorperates your alliance partners ability as well?
2 Likes
Not sure how many times rankings are determined by the 3rd component (SP). You’re probably onto something here, maybe take “Sum of Opponent Wins minus Sum of Alliance Partner Wins”. But again, I come back to preferring to play more qualification matches, which would further reduce the number of times the 3rd ranking factor would come into play.
Maybe it would mean allowing multiple head-refs (e.g. one per-field for tournaments that run on 2 or more fields). Making games quick to reset.
9 Likes
It would be nice to just have more matches but we just don’t have that many fields here and even fewer referees. It just wouldn’t be feasible to have more matches here in South Dakota.
Definitely agree about having more matches.
Tbh I feel like that hypothetical metric should be before AP. (So it would be the tiebreaker right after WP. Rankings would go wp, strength of schedule, then AP). To me, AWP has kind of become a substitute for AP in a way.
3 Likes
The Alliance selection process allows teams to make the case to their peers that they are “better” than their ranking would indicate.
2 Likes