A problem for Level 5 users in solving low Level problems
Does it seem fair that if you're at Level 5 in some category, and you answer "wrongly" some problem in Level 1, your rating can drop as much as 34 points, and yet if you had answered "correctly", your rating won't go up at all? Doesn't that strongly discourage anyone from trying any problems at all that are below their Level?
A recent algebra problem posing a simple linear equation has the so-called "correct" answer of x = 0, when in fact any value of x would be a solution. An Algebra Level 5 user can lose as much as 34 points in tangling with this bogus question.
In game theory, one looks at expected returns versus expected potential losses. Since low Level problems return nothing in rating at risk of a considerable drop in rating, as compared to higher Level problems, doesn't that mean that the best strategy is to avoid the lowest Level problems whenever possible? Especially when a significant percentage of them are poorly worded or even have wrong "correct" answers?
I would suggest that deductions in ratings for wrong answers should be the same or about the same as increases in ratings for right answers.
Note by
Michael Mendrin
7 years, 2 months ago
Easy Math Editor
This discussion board is a place to discuss our Daily Challenges and the math and science related to those challenges. Explanations are more than just a solution — they should explain the steps and thinking strategies that you used to obtain the solution. Comments should further the discussion of math and science.
When posting on Brilliant:
*italics*or_italics_**bold**or__bold__paragraph 1
paragraph 2
[example link](https://brilliant.org)> This is a quote# I indented these lines # 4 spaces, and now they show # up as a code block. print "hello world"\(...\)or\[...\]to ensure proper formatting.2 \times 32^{34}a_{i-1}\frac{2}{3}\sqrt{2}\sum_{i=1}^3\sin \theta\boxed{123}Comments
I agree, I think the rating system is way too harsh. I find it's difficult to maintain my level because even though I may answer 3 level 5 problems correctly, getting one level 3 problem wrong seems to negate all of the gains that I made, even if it was due to a simple reading error. I think the rating system needs to be modified so that either it is less harsh on getting lower level problems wrong or more generous when you get higher level problems right. The latter would probably be more encouraging for users of the website but maybe a bit of both would work.
Log in to reply
If all the problems were perfect and thoroughly vetted, as if taken from Olympiad problems, then it could be tolerable, because then at least I can trust to know exactly what the problem is, and that it has an unique, correct answer. Unfortunately, that's far from the case in Brilliant, where even a lot of the rated problems are defective and you just don't know you're getting into to. We shouldn't be punished for having tackled defective problems. No, the Brilliant staff doesn't have the time to address each and every complaint about such bad problems, there has to be a better way. One way would be to make the rating system a little more realistic and less harsh.
Log in to reply
That's what I meant by my comment.
Log in to reply
Well, I'm trying to completely concur, Cole. By the way, thanks for the interesting problems you've been posting. They've been fun.
Log in to reply
I'm glad you made this post, this issue has been on my mind for a while. It was good to see that you made a solution to my center of mass question, you got my upvote.
Log in to reply
Brilliant has updated the algorithm it seems, all thanks to @Michael Mendrin ! and all who reshared and liked the note..
hey there I am Salman Zafar I am new here can you just help me out ? I will be verrrrrrrry thankful
hey there I am Salman Zafar I am new here can you just help me out ? I will be verrrrrrrry thankful
Thanks for your feedback.
The system is set up such that the expected gain from solving a problem is 0. For example, if we're 99% sure that a level 5 can solve a level 1 problem, then if they get -34 for a wrong answer, they should get +0.345 for a right answer. While we round your rating off when displaying, fractional changes are still recorded.
I agree that bogus problems are a concern. Your rating will update when the answer is corrected. This often rewards you, due to the fake high rating of the problem.
We are also working on providing more clarity on the validity of problems, and will provide more tools to the community to moderate problems.
Log in to reply
This policy of "0 expected gain" makes sense only if all of the problems are perfect (i.e. clearly and unambiguously worded, with correct answers). But since we know that a good percentage of them are not, this needs to be factored in. Let's imagine that I'm at some casino playing the roulette table, the odds of, say, 00 winning is 1/37, but the payout is 35 to 1, slightly less than 0 expected gain (because the casino has to make money). It's a risk I'm willing to take, because then I can expect to play for a while and have fun before losing all of my money. However, if the odds of my winning is, say, 1/40 (which is about what happens when I tackle Level 1 problems, because of problems with poor wording or incorrect answers), then there's going to be a lot less than 0 expected gain, and I'm going to find out that my money is going to run out in a hurry. And so I'll leave for a more fair casino.
The Brilliant staff has resources to understand and use statistics, and my advice would be that Brilliant uses the same quality control philosophies that manufacturers have, which is that they don't start out assuming that they are delivering perfect products every time. They have to factor in the reality that a certain percentage of their products are doing to have defects, otherwise they are going to lose customers. If I were on the Brilliant staff, I'd be focused on making my customers happy, instead of ignoring what it is like for consumers to use Brilliant. I am one consumer, and I would have to say this is probably the most aggravating characteristic of Brilliant, which is that I cannot just go and tackle problems "for fun", because I know how dangerous it can be if I happen to run into a bogus problem. Stepping on a land mine once in a while is okay, but after it's blown up in your face too many times, well, you start avoiding the minefields, and look elsewhere. You might already be losing consumers after they've tried out Brilliant for a while.
My strong suggestion to the Brilliant staff is that they pay attention to what makes their consumers happy, not what "ideals should be upheld". There are ways to do both. Be creative.
How about marking a user posted problems after they passed a quality check with some icon indicating that the answer is correct and that the question contains no ambiguities.
I think the multiple choice problems should all be unrated. I remember going down like 300-400 in calculus because I got a MC problem wrong.
Log in to reply
Haha but you are back at 5.
I think that questions that answer as explanation should be a discussion instead, not a multiple choice. People with most votes gets a ratings and stuffs. I made a very stupid mistakes on easy problems and the score dropped down like 250 by just a single problem. Honestly, I remembered of having a nightmare after that problem on that night lol.
Yup I agree. I also experienced this drastic rating decline in algebra when I by mistake clicked on the wrong option of an MCQ while attempting a level 2 ques. (I was level 4 on algebra then..) Unfortunately fell to Level 3 now...
Log in to reply
hey Vishal I am Salman Zafar I am new here can you just help me out ? I will be verrrrrrrry thankful
Same for me, I was at level 4 the actual answer of a question was 512 but the answer given was wrong and stated that the answer is 256 which only a mad person would do. Unfortunately now I am in level 3 in algebra
Seriously. I agree SO much.
Log in to reply
Same here
Indeed, it seems like there is more penalty for incorrectly answering a problem than there is reward for correct responses. In my opinion, the best solution to this would be to increase the number of points gained by answering harder problems and change the system so that points are not lost by answering incorrectly until all three tries have been used.
Log in to reply
Excellent idea Lee! @Peter Taylor have you ever considered this?
Can't say anything, but I agree with this.
You won't believe me, but this very feature of Brilliant has cured me of my really bad habit of answering the questions whose answers are doubtful to me. It helped me greatly in the exams which have negative markings. Some of my great downfalls are -
Combinatorics - Level 3 to Level 1
Mechanics - Level 4 to Level 3
Electricity & Magnetism - Level 2 - Level 1
How I am thankful to Brilliant, I can't tell you. (I am not being in the least sarcastic here, seriously!).
I know its really a very big problem these days, that's why I never even attempt level 4(or below) questions these days(only 5's) as they're much more profitable.