Binomial Expansion

$x^5$ can be achieved by three ways (

(1): $(ax^2)^2 x ^1 1^6$

(2) $(ax^2)^1 x^3 1^5$

(3) $(ax^2)^0 x^5 1^4$

So co-efficient of $x^5$ is $C=\dfrac{9!}{2!1!6!}a^2 + \dfrac{9!}{5!3!1!}a + \dfrac{9!}{5!4!0!}=252 a^2 + 504 a+126$

Just differentiate or make a perfect square method to find minimum of $a=-1$