Optimization (edited)

This approach will use coordinate geometry and calculus:

Here is a cross-section that runs along the cone's height. The circle here is 6 units (the radius) above the y-axis, so it has equation $x^2+(y-6)^2=36$ .

We can plug in the point (r,12-h) to find an equation relating r and h:

$r^2+(12-h-6)^2=36$

which simplifies to:

$r^2=36-(h-6)^2$

We can plug this in for $r^2$ in the equation for a cone's volume:

$V=\frac{\pi}{3}r^2h=\frac{\pi}{3}(36-(h-6)^2)h=\frac{\pi}{3}(36h-h(h-6)^2)$

Now we can differentiate with respect to h and find the critical points:

$\frac{dV}{dh}=\frac{\pi}{3}(36-[(h-6)^2+2h(h-6)])=\frac{\pi}{3}(36-(h-6)(h-6+2h))=\frac{\pi}{3}(36-(h-6)(3h-6))=\frac{\pi}{3}(36-3(h-6)(h-2))=\pi(12-(h-6)(h-2))$

So, $\frac{dV}{dh}=0\iff 12-(h-6)(h-2)=0=12-(h^2-8h+12)=-h^2+8h \Rightarrow h=0,8$

$\frac{dV}{dh}$ is a parabola with a negative leading term, so only at the rightmost zero will it transition from positive to negative, meaning $h=8$ is the only relative maximum. Additionally, we are only working on the interval $(0,12)$ , so it is an absolute maximum over this interval.

Plugging $h=8$ back into the equation relating r and h, we get $r^2=36-(8-6)^2=36-2^2=32$

Plugging this in, along with the height, into the volume equation gives us $V=\frac{\pi}{3}(32)(8)=\frac{256\pi}{3} \approx 268.083$