Using trigonometry

Circle $\small{\mathfrak{C}:x^2+y^2=9\implies x=3\cos p, y=3\sin p}$

Ellipse $\small{\mathfrak E:4a^2+9b^2=16\implies a=2\cos q, y=3/2\sin q}$ .

Coming to given expression, completing the squares to reduce the given expression to:

$\large (2ax-3by)^2$

By substituting $x,y,a,b$ found previously this is:

$=(12(\underbrace{\cos p\cos q-\sin p\sin q}_{\cos(p+q)}))^2$

$\large =144(\cos(p+q))^2$

It's maximum value is $\large\color{#0C6AC7}{144}$ when $\color{#D61F06}{p=-q}$ .

Use Cauchy Schwartz inequality