Blues must be separated!

Let us temporarily repaint the red and the green balls black and line the black balls up in a row.Arranging $10$ black balls and $5$ blue balls so that no two blue balls are next to each other is the same as placing $5$ blue balls into $11$ "buckets" between the black balls, and at the ends of the row of the black balls, so that no two blue balls are in the same buckets.That is, we choose just $5$ buckets out of $11$ - this can be done in ${11 \choose 5}$ ways. Finally, there are ${10 \choose 5}$ ways to repaint the black balls red and green. Thus the answer is ${11 \choose 5} {10 \choose 5}= \boxed{116424}$

Here's another approach using stars and bars:

We need to place blue balls in such a way that no two blue balls are next to each other. That means between two blue balls, there's at least one red or green ball. Let's think that one red or green ball will come right to each first four blue balls.

So one of the possible way is $\text{\_ | *\_ | *\_ | *\_ | *\_ | \_}$ . Here each vertical bars are blue balls and each stars are any of red or green balls that appear right to each first blue balls. We'll think bars and stars as a unit while selecting the place for them to sit. So we need to place first the bars( blue balls) and then the stars will be placed right to them. We have 11 places from which we need to choose 5 place for the blue balls to place, which can be done in $\binom{11}{5} = 462$ ways.

Now for the the first * we have 10 balls to choose from, for the second * we have 9 balls to choose from, ..., for the last * we have 7 balls to choose from. Similarly for the first _ we have 6 balls to choose from, for the second _ we have 5 balls to choose from and so on. There we can fill rest of the * and _ in $10!$ ways.

We placed the blue balls one by one but we permuted the red and green balls while placing, so they can be rearranged themselves, so to overcome over counting we'll divide by $5!^2$ .

$\therefore$ Total no of ways to place blue , red and green balls so that no two blue balls are together $= \frac{462 \times 10!}{5!^2} = 116424$

Simply,we have to sit a blue ball between two other painted balls or on the two corners.

Thus we have 9+2 available sits for 5 blue ball

In those 11 seat 5 blue balls may sit in 11C5 different ways.

For each of arrangement in 11C5 the 5 red and 5 green may sit in [[10!/5!*5! ]]=R different ways.

Thus answer becomes 11C5*R=116424