Where could the vertex be ?

The orthogonal matrix $P \; = \; \left(\begin{array}{ccc} \frac{1}{\sqrt{2}} & \frac{1}{\sqrt{6}} & \frac{1}{\sqrt{3}} \\ -\frac{1}{\sqrt{2}} & \frac{1}{\sqrt{6}} & \frac{1}{\sqrt{3}} \\ 0 & -\frac{2}{\sqrt{6}} & \frac{1}{\sqrt{3}} \end{array}\right)$ is such that $P^TAP \; = \; \left(\begin{array}{ccc} 54 & 0 & 0 \\ 0 & 24 & 0 \\ 0 & 0 & 0 \end{array}\right)$ If we define $\left(\begin{array}{c} X \\ Y \\ Z \end{array}\right) \; = \; P^T\mathbf{r} \hspace{2cm} \left(\begin{array}{c} \alpha \\ \beta \\ \gamma \end{array}\right) \; = \; P^T\mathbf{b}$ then the equation becomes $\begin{aligned} 54X^2 + 24Y^2 + \alpha X + \beta Y + \gamma Z + c & = \; 0 \\ 54\left(X + \tfrac{\alpha}{108}\right)^2 + 24\left(Y +\tfrac{\beta}{48}\right)^2 + \gamma Z & = \; \tfrac{\alpha^2}{216} + \tfrac{\beta^2}{96} - c \end{aligned}$ and hence the vertex of the paraboloid occurs when $X \; = \; -\tfrac{\alpha}{108} \; = \; \sqrt{2}\;, \hspace{1cm} Y \; = \; -\tfrac{\beta}{48} = 2\sqrt{6}\;, \hspace{1cm} Z \; = \; \tfrac{1}{\gamma}\left(\tfrac{\alpha^2}{216} + \tfrac{\beta^2}{96} - c \right) \; =\; 2\sqrt{3}$ namely when $\mathbf{r} \; = \; P\left(\begin{array}{c} \sqrt{2} \\ 2\sqrt{6} \\ 2\sqrt{3} \end{array}\right)$ The coordinates of the vertex are $\boxed{(5,3,-2)}$ .

A paraboloid equation is of the form $(r-v)^T\bar{A}(r-v)+\bar{b}^T(r-v)=0$ where $v$ is the vertex.

Comparing the matrix coefficients of the quadratic equation in the question to the one above, it can be easily deduced that

$\bar{A}=A$ , .... (1)

$\bar{b}-2v^T\bar{A}=b^T$ .... (2) and

$v^TAv-b^Tv=c$ ..... (3)

Substituting the value of $\bar{A}$ from (1) into (2) and $\bar{b}$ from (2) into (3), we get the equation

$v^TAv+b^Tv+c=0$ .

[In other words, the vertex satisfies the quadratic equation].

A "low tech" substitution of the choices shows that only $\mbox{(5,3,-2)}$ satisfies the given equation.