Take any 3 balls

First show that by rational root theorem that if $R=G$ is a solution, then the cubic equation yields no solution, so $R\ne G$ .

WLOG $R > G> 0$ , then by AM-GM inequality , we have $\sqrt{RG} < \tfrac{R+G}2 \quad \Leftrightarrow \quad RG < \tfrac{(R+G)^2}4 \quad \Leftrightarrow \quad 200RG < 50 (R+G)^2 \; .$

Let $Y = R+G\geq 1+1=2$ , then we want to find the integer solution for $50Y^2 > Y^3- Y$ , factoring out the linear factor and solving the quadratic inequality by quadratic formula gives

$2 \leq Y < 25 + \sqrt{626} \approx 50.02 \tag{1}$

Now consider a quadratic equation with roots $R$ and $G$ , by Vieta's formula - forming quadratics , we have

$X^2 - YX + \tfrac{Y^3-Y}{200} = 0$

Since $Y=R+G$ is an integer, then the quadratic discriminant of the equation above is a perfect square:

$Y^2 - 4 \left( \tfrac{Y^3-Y}{200} \right) = M^2 \: \Leftrightarrow \: Y^3 - Y \equiv 0 \pmod{50} \: \Leftrightarrow \: R+G = Y \equiv 0, \pm1 \pmod{25} \qquad (2)$

Hence, the possible values of $Y=R+G$ boils down to $\{ 24,25,26,49,50 \} .$

Now just plug in these 5 possible solutions and with enough patience, we can show that $R+G = 49$ is the only solution.

Hey thanks, Brian! Yeah, proving that this is the solution gets kinda tedious... Oh well, glad you liked it! :)