Asymmetric replacement

To reach $\{ 0,2,5 \}$ , take $a = 4$ and $b = 3$ . Only one step!

Notice that the sums of squares of elements of the set is invariant under the operation, because $a^2+b^2 = (0.6a-0.8b)^2+(0.8a+0.6b)^2$ . Since $2^2+3^2+4^2=29$ and $1^2+3^2+5^2 \ne 29$ , we cannot reach $\{1,3,5 \}$ .

Every step increases the sum of the set of $0,4a + 0.8b$ which is even. Starting set has $S=2+3+4$ which is odd, so the result set must have an odd value of $S$ . Only $b$ fulfills this requirement

The initial sum is 2+3+4=9

As a and b are replaced by 0.6a-0.8b and 0.8a+0.6b, the new sum will be:9+0.4a-1.2b.

0.4a and 1.2b are always even so, every new sum must be(an odd minus or plus an even number is always odd) odd and not 9.

Only b satisfies this condition