Can you see trigonometry?

The equation $25x^2 - 20xy + 40y^2 = 36$ is an ellipse with origin as center and semi-axes at an angle with the $x$ - and $y$ -axes. It can therefore be represented by a standard form of ellipse.

$\begin{aligned} 25x^2 - 20xy + 40y^2 & = 36 \\ 16x^2 + 16xy + 4y^2 + 9x^2 - 36xy + 36y^2 & = 36 & \small \color{#3D99F6} \text{Divide both sides by }36 \\ \frac {4x^2 + 4xy + y^2}9 + \frac {x^2 - 4xy + 4y^2}4 & = 1 \\ \left(\frac {2x+y}3 \right)^2 + \left(\frac {x-2y}2 \right)^2 & = 1 & \small \color{#3D99F6} \text{Standard ellipse equation} \end{aligned}$

Then we can assign

$\begin{cases} \dfrac {2x+y}3 = \cos \theta & \implies 2x+y = 3\cos \theta \\ \dfrac {x-2y}2 = \sin \theta & \implies x-2y = 2\sin \theta \end{cases}$

Noting that $2x^2 - 3xy - 2y^2 = (2x+y)(x-2y) = 6 \sin \theta \cos \theta = 3\sin (2\theta)$ , $\implies \max \left(2x^2 - 3xy - 2y^2 \right) = \boxed{3}$ , when $\sin (2\theta)$ has its maximum value of 1.

The trig clue suggests that the constraint can be reformulated as $U^2+V^2=36$ for some linear combinations $U$ and $V$ of $x$ and $y$ .

Noting that $25=3^2+4^2$ and $40=2^2+6^2$ , we can try combinations of these coefficients until we find

$25x^2-20xy+40y^2=(3x-6y)^2+(4x+2y)^2=36$

Now we can write $3x-6y = 6\cos{\theta}$ , $4x+2y = 6\sin{\theta}$ .

We also have $(3x-6y)(4x+2y)=12x^2-18xy-12y^2=6(2x^2-3xy-2y^2)$

Substituting in, this is

$2x^2-3xy-2y^2 = 6\cos{\theta}\sin{\theta} = 3\sin{2\theta}$

which clearly has a maximum value of $\boxed3$ when $\theta=\frac{\pi}{4}$ or $\theta=\frac{5\pi}{4}$ . We can double check this answer by substituting $\theta=\frac{\pi}{4}$ to give

$3x-6y=3 \sqrt2$ , $4x+2y=3\sqrt2$

Solving these gives $x=\frac{4\sqrt{2}}{5}$ and $y=-\frac{\sqrt2}{10}$ , which do indeed attain the maximum value of $2x^2-3xy-2y^2=3$ .