New ratings system for problem solving on Brilliant
So it turns out that Christopher J has predicted the future! We liked his idea so much we have been developing it for a few months now.
We have indeed just launched an adaptive ratings system on Brilliant. Levels still exist, except now instead of being determined by your performance over the past two weeks, your level is determined by your rating, which changes each time you solve or fail to solve a problem.
Read about levels and the new rating system here. You will note that the colors designating each level have changed as well. This was done to better capture the fact that levels on Brilliant are no longer sharply distinct things, but are more related along a continuum.
You can also link to an explanation of ratings, by clicking the "what's this?" link next to your rating on the problem solving page.
Remember while your levels are visible to everyone your rating in any given topic is visible only to you.
We hope this will help you gain more specific insight into your progress and how hard a particular problem is relative to your experience level. It will also help us better gauge the difficulty of the problems we distribute to you.
Let us know if you have any questions in this thread.
Note by
Peter Taylor
7 years, 7 months ago
Easy Math Editor
This discussion board is a place to discuss our Daily Challenges and the math and science related to those challenges. Explanations are more than just a solution — they should explain the steps and thinking strategies that you used to obtain the solution. Comments should further the discussion of math and science.
When posting on Brilliant:
*italics*or_italics_**bold**or__bold__paragraph 1
paragraph 2
[example link](https://brilliant.org)> This is a quote# I indented these lines # 4 spaces, and now they show # up as a code block. print "hello world"\(...\)or\[...\]to ensure proper formatting.2 \times 32^{34}a_{i-1}\frac{2}{3}\sqrt{2}\sum_{i=1}^3\sin \theta\boxed{123}Comments
It's always nice when reality conforms to our wishes. :)
Why do I see a dim star beside my rating for each of Maths and Physics ?? How do I enLIGHTen it ...
I think the new rating system is great but I think that the colours are not ideal. Now it is less easy to easily distinguish one level from another. Also, as previously voiced by other Brilliant members, the members who have ordered their coloured lanyards to indicate that they have reached a certain level have now been 'outdated' and no long correspond to the correct colour. I know that you mentioned that you did this to make the levels more like a 'continuum' which is understandable but I believe that the sharp and distinct colours such as blue, green, red and yellow make it clear which level you are in and cause for less confusion. Maybe having distinct colours will also make people more motivated to reach the next tier of levels. Someone would probably be more excited to go from yellow to green than to go from purple to a slightly darker shade of purple. I guess it doesn't really matter too much but I have seen many people such as myself on the discussion board that dislike the new colour scheme and it seems to be a common concern. Thanks for the update nonetheless as I really liked everything else that came with it.
Log in to reply
I think that, rather, the colors should be a direct function of your rating, not your level, maybe
Log in to reply
It would be like a rainbow! yay, cool idea.
hi， cole
Any estimates for rating comparisons? For example, what rating would roughly correspond to USAJMO? IMO? Is the scale similar to chess ELO?
Will we level down if we don't do the problems?
Log in to reply
I checked this today by activating Algebra, and then I viewed all four problems one after the other. My rating changed each time. (It started off mid-2100s for the first problem, and was down to 2094 by the last one).
Actually came here to query why the rating was changing so quickly, but realized after reading these posts that it's leveling me down just for viewing them.
Maybe one solution to the problem Calvin mentioned of exploitation is that if you solve at least one problem in the set for the week, then you do not get any rating loss for other problems in the same set that you viewed but didn't attempt?
Inactivity will not level you down. If you view lots of problems without solving any of them, you will indeed level down, however you can always regain your rating and your level by solving the problems you viewed.
Log in to reply
I do not think that viewing problems should make your rating go down. I often look at problems, knowing I won't have time to do them, just because they are interesting. Viewing problems and not solving them does not imply you can't solve them.
Also, I think you should be able to see the rating for problems that aren't in your own set.
Other than that, I like the new system, and how you can see the difficulty of problems. One question though: How are ratings calculated?
Log in to reply
Yes.. I request Brilliant to change this.. Otherwise I will be afraid of viewing the problems in the first place. :(
I agree completely, the past month I just didn't look at the sets even though I know I would be missing out since I don't want to level down just because I don't have time really to do the problems. If someone can't do the problem, then it'd be obvious when they put in an answer and get it wrong. The problem with this is that it does not account for the fact when somebody doesn't get a problem so much that they don't even put in an answer. However, I think this is a lesser externality than having people not want to even look at problems in fear of leveling down.
Thanks for the feedback, Daniel. We are still thinking about the right way to handle "views" with no tries. Remember, though, that your rating is private. There is no shame in your rating going down a little.
Also, ratings is off in "Practice"—try problems with no fear!
Log in to reply
I also think we shouldn't lose points for viewing problems. And there's a difference between Practice and Challenges.
You can always wait until the next week and look at them. :) Just stop for one week, next week view them, won't decrease anything.
Log in to reply
This is not a true statement.
Can you let me know why you thought this is true? If we stated it somewhere, I'd have to correct it.
Log in to reply
Well, after a week, you can see the answers and solutions, so it makes no sense for your rating to go down.
Log in to reply
You only get to see answers and solutions to problems that you have already viewed the previous week, which already had the initial deduction (just once). Otherwise, we would not list any problems for the week under past solutions.
If you click on a public link (though a profile, or otherwise) to a problem that you have not seen before, then that would still affect your rating.
Log in to reply
Oh okay I did not know that. I stand corrected.
Peter, either you've forgotten to change the FAQ, or what you said above isn't true.
Quote from FAQ: "Don't worry; viewing and not answering problems will not cause you to level down, nor will taking a break from Brilliant."
Log in to reply
Aha! Good catch. We forgot to update the FAQ.
That sentence is now half-true. Taking a break from Brilliant still won't level you down, but viewing enough problems, without solving them could level you down.
Till today I was thinking that we are only levelled down on viewing sufficient problems and attempting sufficient number of them wrongly. Calvin, himself addressed this in a few discussions! So, is this part of the new system?
Log in to reply
As we change the system, we have to update some of the rules, and resolve various issues that arise. Our mentality remains the same, and ratings allows us to be more flexible in responding to various scenarios.
A strong student in his level will not drop a level by glancing at 5-8 problems over 2 weeks. However, if someone viewed 10+ problems without being able to answer any of them, we believe it would be better to work on the fundamentals first. The system that you mentioned was abused by members who unable to answer any of the questions, but artificially maintained their level due to not attempting sufficient number of them wrongly each week.
Log in to reply
As I have already mentioned somewhere else, I think if users are allowed to level down own their own (but not level up of course!) then this issue will be solved quite easily. Why not to do that?
Log in to reply
I would like that!
With the current ratings system, it is clear that anyone can easily (manually) level down by viewing enough problems / getting them wrong (as opposed to just stating it in the FAQ). They do not need to wait 2 or more weeks.
Log in to reply
I think users should have the option to manually level down if they want to, instead of arranging it by getting hard questions wrong on purpose.
Hmm.. I have one more question.. if I view the same problem more than once.. let us say I close my browser and reopen, then will my rating reduce every time?
Log in to reply
Nope. It only decreases once (during the initial view).
That's fair enough for me. I was expecting a much more stringent rule.
But, I request you to create a page listing all such rules and guidelines of the site and updating them whenever necessary.
Woww.. this is really cool!! This is very similar to rating system for chess which I always love.. :)
One thing I noticed just now.. Some of the problems have higher ratings but lower points. How is this possible? One should get more points by solving harder problems.. right? Or is it that ratings of the problems are also determined by how many people and of what rating could solve these? If that is the case, then it is fine I think.
Log in to reply
Points and ratings are not currently computed in the same way. This will be changing soon as well.
How can I know my rating?
Log in to reply
Right now, the place to see your current rating in a topic is next to any weekly problem.
Log in to reply
What about in your profile next to your level.
Log in to reply
Soon!
When is your level updated to match your rating? From my ratings, I should be a Level 4 in Physics and a Level 3 in Combinatorics, but instead I'm at Levels 3 and 2 respectively.
And why do the problem ratings typically fall outside the band for the level? For example, this week's Level 3 Algebra problem ratings are (at the time of writing) 1934 (Level 4), 1696 (Level 3), 2151 (Level 5!) and 2225 (also Level 5!). Only one of them falls within the band for Level 3!
Log in to reply
Hi John,
Sounds like a bug! We'll check it out, thanks for writing.
We are still tuning the ratings system. Note that a problem's rating is not necessarily supposed to line up with the user levels.
The important question for you is: do those problems feel too hard for you?
Log in to reply
I found another BUG here which I exploited once ...
In a physics problem I had used up 2 tries guessing the answers as A and B and I knew that my answer could be either of X or Y but because of only one try left I thought of the following...
I opened the same problem page in 2 tabs and gave the answer as X in one and as Y in the other and pressed enter in both almost at the same time .. Both pages showed that all my answers were wrong but also something else one showed that I had entered A,B & X the other showed that I had entered A,B & Y ... So I actually had used 4 tries and couldn't get to the answer.. I hope you fix this ;-)
Log in to reply
The "Bug" is that the server can handle more than one request at the same time. While this may make it seem like you get 2 tries, whichever came in slightly after the other (from the perspective of the servers) would end up over-writing the other so it would basically be like tossing a coin between the two.
It is always a better idea to enter just the one answer that you think is more likely to be correct.
Well, they were OK for me. But my main problem was really with the disparity between rating and level. But since you lot will be tuning the ratings, I'll just wait and see what happens. Thanks for responding.
Hmm...all my levels were upgraded accordingly when the new week started, except Physics. Rating-wise, I should be Level 4, but instead I am still Level 3.
Log in to reply
According to your profile, you are now Level 4 in Physics!
Log in to reply
But the Challenge page still says I'm Level 3, and I got the Level 3 problems this week...
Log in to reply
Yep. Your problems for the week are decided by your level at the beginning of the week. Next week, you will receive the Level 4 problem set.
Log in to reply
Ah, I see. Thanks for clarifying.
I don't think there is a level 6 or 120...
Great improvement. I was wondering, does my rating increase more if I solve a problem with a higher rating? I suppose yes, and if so, does my rating increase even more if a problem turns out to be harder than at the time I solved it?
Log in to reply
Yes. Solving a higher rated problem increase your rating by a larger extent, as stated in "what's this".
No. Your rating is only affected at the point when you (initially) viewed or answered the problem (both correctly and incorrectly). Viewing the question repeatedly (whether you solved it or not) will not affect your rating.
Its really amazing! I dont know in which game it is used but I know that brilliant.org would be the first site to do this. HATS OFF TO BRILLIANT!!!!!!
A suggestion: shouldn't our rating increase when we publish a solution to a problem? Publishing, or even writing out a solution verifies that you were able to solve the problem without relying on things like computers for symbolic manipulation or anything of that sort, and are able to communicate your method clearly. (Yes, I know there are a couple of solutions out there that list programs to solve a problem, but those are in a very small minority.)
Log in to reply
The problem is that then a lot of people would post poorly-written and erroneous solutions. Alternatively, upvotes could increase rating.
Log in to reply
That works too.
Love the new rating system. I think it is great!
This question is a bit off topic, but I was wondering why there are no more geometry questions like there used to be? And are they going to be brought back?
Log in to reply
Also I can't access the Data Structures and Algorithms questions. Is this the same for everyone or is it a bug?
Log in to reply
Same for everybody.. it will come back soon..
Log in to reply
Ok. Thank you :)
Why have I gotten so many vote downs? :(
Log in to reply
Probably because your question was not on topic, and they have been asked many times before.
The rating system is simply awesome! Nice work brilliant team. A question i have in my mind- how much my rating increase whenever I solve a problem?
Just had an idea: what if, instead of having problems by level, you could have problems by rating? I mean this in the sense that you wouldn't get "Level 4 algebra problems" but "Rating 1600-2100" problems, or something like that.
A question and clarification. Do the ratings fall when you do not attempt any question? As it turns out that i had scaled the barrier from one level to another but when the new week came my ratings had fallen.
Log in to reply
If you view the problem i.e. open it's link, your rating would fall, even if you didn't try to answer. If you get it wrong, your rating would fall even more. If you get it right your ratings would increase significantly enough to compensate for both reductions.
If you do not open problems link and thus don't see the problem, your ratings would not be affected.
My Algebra has just been upgraded to Level 5. However, my rating is [removed - Calvin]
Shouldn't my Algebra be at Level 4? I mean, I like having my algebra rated highly, and personally I think I'm good at it (my favourite), but Algebra Level 5 is 2000 and above, right?
Just pointing out a possible error in the system
Log in to reply
I see that your rating did cross over 2000 when you solved a probelm, then came down slightly when you viewed a new problem.
There is slight leeway in lowering a level, just so that you do not aggressively toggle back and forth between levels when you are right at the threshold.
Log in to reply
There is another issue with the "viewing penalty" namely viewing problems that are in other problem sets, like a level 5 combinatorics set for instances. When I was looking at a level 5 Number Theory problem the systems took off rating points.
Log in to reply
How can you see level 5 problem if you are on level 4 (or that matter any lower level) ?
Log in to reply
Go on the profile's of those in the respective level you want to observe, check their activity and you'll see the problems they have solved, e.g. the ones in that level.
Can I get degrade immediately in Number Theory?
I have a complaint. That is every time the server updates, my rating goes down. And it has happened twice now. First I had 1731 for number theory, then went down to 1681 and now 1641. It seems it won't be long before i level down to level two. This has happened with all the other topics too. I was meant to be on level three number theory, algebra and number theory. It will be good if you could help me with it.
Log in to reply
If you read the "What's this" explanation, your rating goes down when you initially view the problem, or answer it wrongly. It goes up when you answer the problem correctly. I've reviewed your rating changes, and they are occurring as intended by our algorithm.
Log in to reply
Ok, I understand, thanks for your help.
If we have the rating 1750-2000, doesn't it imply that we are in the 5th level?
Log in to reply
We have shown your rating on the challenges page, along with the next level up target.
If your rating is in the range 1750-2000, then you would be in level 4. There is a slight allowance for level 5 users who had a small drop in rating, due to getting a problem wrong or viewing more problems. So it's possible to have a rating just under 2000, while still be in level 5.
If you look at the levels page, we stated level up targets, which are nicely rounded off to 1250, 1500, 1750, 2000.
Log in to reply
Yesterday, it showed "level up at 1750" when I was in level 4 in number theory. Hence my question.
It is resolved now.
Why is that the ratings of problems change so fast ?
I've noticed that over the past few days my physics rating has been fluctuating even though I haven't looked at or attempted a problem since the beginning of the week. The last time I finished with physics for the week, my rating was slightly above 1500, about 1503. Then by yesterday it had gone down to 1497 and today it was gone back up to 1499. Why would it be changing like this?
I think you could add Selling Tips because when people do not know where to start, Only with Shopping Tip the Person has a Greater Sense about What Problem Want !
Due to this new ratings system, I've been leveled down to level 4 in algebra. Now, I'm pretty darn good at algebra. In fact, I just finished all 4 algebra level 4 problems this week in ≤ 15 minutes. Please level me back up to level 5 in algebra.
Log in to reply
I see that you are now level 5 in Algebra. If you believe that this is still an issue, send me an email.
Note that your ratings will move around some during the week, as you work to solve problems.
Hi. I just saw some comments about the "views" and I got an idea, so I want to share it. Maybe, if the problems are attached in a PDF document, the users will get able to see them without starting the weekly challenge. That could be great, because then the rating will be evaluated correctly. If a user opens the PDF just because he/she wants to see the problems, he or she will not be penalized. But if he/she starts the challenge, it will mean that he/she knows some of the problems, so in case of missing them, the user can be penalized justly. How does it looks?
Log in to reply
What's the difference between this and just taking away the viewing penalty?
Log in to reply
Well.. you'll be able to decide if whether you are prepared or not. I think the penalty is justified after you miss a problem, and it is possible only if you see it in other way, different than starting the challenge. It's just an idea...
Log in to reply
If one is not prepared one should be penalized and later sent in down level. You can see problems one by one while solving each before viewing the next. That way you only lose for actually not doing the problems instead of just not having time.
Anyhow Brilliant has a pressure of maintaining sufficient traffic of students actually solving problems, so this makes lot of sense from their perspective.
Log in to reply
With prepared i meant like being prepared to start working with the problem on that specific moment.
Log in to reply
Well you have entire week :) Your ratings would be deducted only once for viewing. You may few it thousand times (or more) after that without any loss.