Maximum volume of the pyramid

The volume of pyramid is given by $V = \dfrac {Ah}3$ , where $A$ is the base area and $h$ is the height of the pyramid. With fixed slanting side lengths of 6, the base triangle has a largest area when it is equilateral. Therefore, the pyramid with the largest volume is one with a equilateral triangular base.

If the base equilateral triangle has a side length of $a$ , then the base area $A=\dfrac {a^2\sqrt 3}4$ , the height $h = \sqrt{6^2 - \left(\dfrac a{\sqrt 3}\right)^2}$ , $\implies a^2 = 108 - 3h^2$ , and the volume $V = \dfrac {Ah} 3 = \dfrac {a^2 \sqrt 3 \cdot h}{3 \cdot 4} = \dfrac {\sqrt 3(108h -3h^3)} {12} = \dfrac {\sqrt 3(36h-h^3)}4$ . $V$ is maximum, when $\dfrac d{dh} (36h-h^3) = 36-3h^2 = 0$ or $h = 2\sqrt 3$ . Note that $\dfrac {d^2}{dh^2} (36h-h^3) \bigg|_{h=2\sqrt 3} < 0$ . The $V_{\max} = \dfrac {\sqrt 3\left(36(2\sqrt 3)-(2\sqrt 3)^3\right)}4 = \boxed{36}$ .

We can find this without the explicit assumption that the base is equilateral as follows.

Take $P$ as an origin. $A,B,C$ are points on a sphere centred at $P$ with radius $6$ ; the volume $V$ of the tetrahedron is given by

$V=\frac16 \det \begin{pmatrix} A_x & A_y & A_z \\ B_x & B_y & B_z \\ C_x & C_y & C_z \end{pmatrix}$

Without loss of generality, we can take $A$ to be the "north pole" of the sphere and $B$ to be on the "prime meridian", so that the points' coordinates are

$A(0,0,6) \\ B(6\sin \theta_b, 0, 6\cos \theta_b) \\ C(6\sin \theta_c \cos \phi_c, 6\sin \theta_c \sin \phi_c, 6\cos \theta_c)$

(using spherical polar coordinates). Plugging this in to the volume formula we get the surprisingly nice $V=36 \sin \theta_b \sin \theta_c \sin \phi_c$ , which is clearly maximised when $\theta_b=\theta_c=\phi_c=90^{\circ}$ giving $V=\boxed{36}$ .

Geometrically, this is one eighth of an octahedron. Its faces are three isosceles right-angled triangles and (as expected) an equilateral triangle.

Volume of the pyramid will attain an optimum when it's base is an equilateral triangle. Let the length of each side of the base $\triangle {ABC}$ be $a$ , and the height of the pyramid be $h$ . Then $h^2=36-\dfrac{a^2}{3}$ . Volume of the pyramid is $V=\dfrac{1}{3}\times \dfrac{√3a^2}{4}\times h=\dfrac{1}{12}\times \sqrt {108a^4-a^6}$ . This attains a maximum when $a^2=72$ , the maximum volume is $\dfrac{72}{12}\times \sqrt {108-72}=\boxed {36}$

Alternatively

Let $\vec {PA}=\vec a, \vec {PB}=\vec b$ and $\vec {PC}=\vec c$ . Then the volume of the pyramid is $\dfrac{1}{6}\vec a\cdot (\vec b\times \vec c)$ , which attains it's maximum value of $\dfrac{abc}{6}$ when $\vec a, \vec b$ and $\vec c$ are mutually perpendicular. Here $a=b=c=6$ and the maximum volume is $\boxed {36}$