Brilliant Ratings
I think the Ratings on brilliant go down too much when you get a problem wrong and doesn't go up enough when you get a problem correct. Sometimes when I get a problem wrong the first time and then get it correct the second time I end up with less rating than i started with
Note by
Yan Yau Cheng
7 years, 4 months ago
No vote yet
1 vote
Easy Math Editor
This discussion board is a place to discuss our Daily Challenges and the math and science related to those challenges. Explanations are more than just a solution — they should explain the steps and thinking strategies that you used to obtain the solution. Comments should further the discussion of math and science.
When posting on Brilliant:
*italics*or_italics_**bold**or__bold__paragraph 1
paragraph 2
[example link](https://brilliant.org)> This is a quote# I indented these lines # 4 spaces, and now they show # up as a code block. print "hello world"\(...\)or\[...\]to ensure proper formatting.2 \times 32^{34}a_{i-1}\frac{2}{3}\sqrt{2}\sum_{i=1}^3\sin \theta\boxed{123}Comments
Yes! Thank you for voicing something that almost everyone else, I think, has been wondering about. I contacted Silas, and he claims that you never lose ratings for a correct answer, even on the the third try, but nonetheless it only goes up by 1 or 2 points. It's obnoxious because it doesn't encourage people to try new things, and experiment with a new area of math because they don't want to lose an obscene amount of points.
Log in to reply
There is the possibility that this is a front-end issue, where we do not update your displayed rating aggressively, especially for old pages / problems. and I'd have to look further into that. In your specific case, see other reply below.
For 'older' users who have done 5 problems in the topic, I do not see any evidence in the backend for them to lose 50+ points on a problem, unless their rating is significantly higher than the problem.
For new members, their rating change is more aggressive as we do not have much information about them as yet, other than the diagnostic problem which they did.
For 'older' users, if you answered the problem correctly in your second time, your net rating change would be positive. If you answered the problem correctly in your third time, your net rating change should be negligible (well within ±5 ).
For new members, this may not be the case, as the aggressive rating changes could penalize you at first, before you finally get the problem correct. However, this edge case is extremely rare.
Log in to reply
And when I get 3 wrong on one problem, that can be as much as 40-100 points off! And I get all 3 wrong a lot!
I think we should redo the whole determining your level thing for each topic because the point system is kind of screwed up.
Log in to reply
Agree
I partially agree with this, but you need to be careful because (in theory) a person who is supposed to be Level 3 could be able to "farm" Level 1-3 problems to reach Level 5.
Your rating should never decrease upon solving a problem correctly on the second try (unless the rating of the problem has drastically changed). The next time it happens, can you send me an email listing the problem that you worked on? If your rating in the backend went up, then this could arise from information not updating aggressively enough (or a display issue). Try refreshing the page and see if your rating changes.
For (long term) users whose ratings have stabilized, then the reward/loss of ratings indicate the relative chance that you will be getting the problem correct. As such, for a person who works only on very easy (relative to his level) problems where he "should be" getting them right 90% of the time, he might see a -45 if wrong and a +5 if correct, which leads to a net of 0 on expectation. Conversely, if you wok on a problem which you should only be getting right 10% of the time, then you should see something like +45 if right and -5 if wrong. (Rating change values not necessarily accurate)
Log in to reply
Actually, I have lost 50-200 points for a wrong answer. It's not fair. I mean, people don't actually get stupider. How can you go and try new things, and try your hand at something you're not familiar with if you're afraid of losing huge numbers of points. Please have Silas forward my emails that I sent him to you discussing this problem, and my solution. Thanks!
Log in to reply
I have looked into your ratings of Combinatorics in the past 30 days, and at no point in time did your rating decrease by 50 or more in the past month. You do have a significant amount of wrong attempts, though there are very few in which you lost 30-50 points. Even in those situations, the problem was extremely easy and when you next got it correct you gained the points back.
Over the period of Jan 30 to Feb 2 (midday), your rating dropped from 1500 to 1200. During that period, you viewed 48 problems, had 21 eventually correct, and submitted 62 wrong attempts (3 wrong attempts to several problems). The significant number of wrong attempts was responsible for your rating drop. It did not happen during 1 problem, but over a period of time. As mentioned, it could appear to happen after 1 problem due to your displayed rating not updating.
Looking further into it, on Feb 2, over the span of 30 minutes, you had viewed 60+ problems in Combinatorics and given 3 incorrect answers to each of them. This resulted in your rating dropping by 1000 points in total. (You initially lost 30-50 points per problem from 3 wrong attempts, and that tapered off as your rating dropped very low.)
I see that we did a manual reset to your Combinatorics rating after that. Since then, I see 6 correct answers (each of which gave you 30-50 rating) and 60+ wrong attempts. Over this entire period, your net rating change is negligible. However, you lost up to 100 rating, before regaining it back towards the end.
If this is not how you have used the site, you should change your password / check if someone else had access to your account.
A possible explanation as to why you see the rating drop in 1 question, is that we might not be aggressively updating your rating changes as you work on different problems. For example, if you open Combinatorics problem #1 with a rating of 2000, and then get 20 problems wrong so that your rating is now 1800. When you answer Combinatorics problem #1 wrongly, it would appear that your rating dropped from 2000 to 1800. Given your pattern of answering, I believe that this is the most reasonable explanation.
We do not make ratings update constantly as that will be a big strain on our servers. However, we could make ratings update upon refreshing of the page, and ensuring that the cached value is not too old. I will look further into this.
Note: initial rating changes when you first joined are much more aggressive, since we only had a diagnostic problem to base it on. At that stage, answering a simple problem wrongly can change your score by 50-100, which allows us to better sort people into the correct ability group.
Log in to reply
About the 1,000 rating loss... I was proving a point to Silas that just because somebody gets a couple of problems wrong doesn't mean that they're any dumber. Which is completely true. Nobody really does get any dumber, (excepting NFL players), so when I lose points for trying things that I haven't had a lot of experience with, I get frustrated. I really don't like losing points at all, and I am the type of person who likes to try new things, so I think it's really bad that Brilliant, a site for broadening math abilities, apparently doesn't want users to try things they aren't great at. What I suggested to Silas is that instead of tweaking the ratings system to lose less points, that the whole entire system is revamped, and the diagnostic test consists of 10 or so questions, and from whatever rating you get on that, you can only improve, but usually no more than 5 points, and you only get points for problems that are higher than your rating. I think this would be great, because it would be a more accurate diagnostic test (not just 1 problem), and you could try new things without worrying about ratings.
See my comment below, please.
Hi everybody,
Thanks for pointing this out (again) and being patient.
We constantly making adjustments in the background and tuning the rating system. There will be bigger changes this coming week. As Trevor and others pointed out, there isn't a "simple" fix to ratings—it is a dynamic system, so it needs to be adjusted iteratively.
I'm used to it because we can have another chance but for multiple choices it's crazy!
Log in to reply
I agree. I really don't like the format of Multiple choice problems. These type of problems should appear only on an entrance exam.
People just seem to guess the answer sometimes. Upon that, we just have ONE chance!
Log in to reply
I agree... 4 chances will be more than enough.
i totally agree.... if they have to post Multiple choice there shouldn't be any points deducted or given for the problem
Agree!!
Log in to reply
ya wright bro
I agree with you. Thanks for mentioning it.
I partially agree with the above statement; however, I think everyone needs to realize how much Brilliant has tried to resolve this issue in the past several months (it was much worse when I joined Brilliant in October). Basically, what I'm saying is give it a little bit of time, and if nothing has happened, then you should bring this issue up again.
totally agree with u. i had rating 1850, then i got a 2380 problem wrong and then my rating went1300
Log in to reply
The situation that you described should never be possible. Can you give me the question in which you saw this happen?
Even if a person with rating 3000 got a problem with rating 1000 wrong, I would be surprised if his rating dropped by more than 150.
A slight possibility is that the topic of the problem was updated (but this happens very rarely). For example, if the initial topic was Number Theory (where you are level 4), and it gets updated to Computer Science (where you are level 1), then you might see your displayed rating change from 1800 to 1200.
Log in to reply
Wrong. I lost 300 points in my combinatorics. Silas put me at 1500, and it dropped to 1200 after one problem.
Log in to reply
And is that 3000 level person actually 150 points stupider because of one problem? People don't get stupider (except for NFL players). People only get smarter. Ask Silas about those emails!
Log in to reply
Edit: I consolidated my response above.
I'm not actually remembering the problem. But as far I do remember, I had put the answer in fractions (in the form a/b) instead of decimals. Can this result in a drastic change in the rating?
Log in to reply
I have looked into your ratings. At no point in time do I see that answering a problem decreased your rating in the topic by more than 100. (Not that as you are a new member, rating changes can large initially, like 60-100, to get you to a suitable level, after which it tapers off and should be closer to 30)
I believe that if you tried entering 3/5 (instead of 0.6) our system would not accept it. We only allow for decimal / integer answers. Can you clarify what you mean?
No. All wrong answers are treated equally. It doesn't matter whether you made a calculation error, a logical error, or a typo.
Log in to reply
I don't know man! You are saying it so surely that now I am starting to doubt my own memory. I think I'll just let it be and try to gain back my rating. Sorry for any inconvenience.
I also agree with this.
Well i request all of you to check out those Mechanics level-I questions. Those questions are like more of your guesses than physics but if you get 1 right, your rating goes up by a point(many times it doesn't) but if you get one wrong, atleast 30 points are deducted.
Log in to reply
Yeah, I hate that.
It doesn't make much sense some of the time, because you kind of get scared to answer a problem. I personally don't have my score go to far down all of the time, but the amount that I've heard people lose on this post seems insane. Also, when somebody loses enough points to level down, it says that they have leveled up.
Log in to reply
Really?
I have written many posts to express how mistakes/errors etc have taken place when I gave the correct answer but have been ignored. The scoring for correct answers has been ... and no-one has tried to address the issues.
Log in to reply
We're spending a lot of time on this, primarily in our courses. When you submit a request like this on problems that a community member creates, it is up to them to correct it – we don't have any influence over them!
Log in to reply
The scoring for such questions is not facilitated by STAFF, only by those who created the problem? Is that so? Then what about the errors in scoring and others?
Log in to reply
I'm sorry – I don't think I can parse your question. Could you restate it?
Log in to reply
Dear Silas Hundt: How is scoring done?by STAFF or ...? How about errors in scoring ie getting a question correct but marked as incorrect? My experience is there is no one to correct this. I have sent many messages and notes which have not been addressed or noted, who is in charge of answering/handling these matters? Thanks.
Log in to reply
This depends on which part of the site the question is in – if it is in the courses, daily challenges, and practice concept questions, 100% of messages are at least read. If the question was actually wrong, we fix the answer, and your "score" (whether or not you get it right or wrong) is updated and the system sends an email.
Log in to reply
Till today the errors in scoring, updating etc have not been amended/addressed/acted on etc despite all the emails I have sent since ... and yet the errors still persist.
Log in to reply
In which section of the site?
Log in to reply
There are questions which were incorrectly marked. I have sent emails about this.
Log in to reply
I've reviewed the emails that you've sent, and they have all been responded to or refer to community-generated parts of the site (which we don't have the people power to respond to).
Log in to reply
Can you parse that? Can't respond to community, then when errors occur and keep on happening and continue to carry on, people have no redress or cannot have any means of finding a way of solving the problem, right? Who decides whether answers are correct?
Log in to reply
If a community member made a problem – they are the people who are responsible for the problem's correctness. That's why it says their name on the problem.
I would like to type here that brilliant is not giving prizes now i suggest brilliant to be back with there prizes