Making Maximum Mess

As in the diagram, we place the cone horizontally with its tip at the origin. Let the radius of the ball be $R$ and the $x$ -coordinate of its centre be L. The volume of displaced water is the volume of the revolution that is obtained by rotating the upper part of the circle from $x=L-R$ to $x=h$ about the $x$ -axis. Since the edge of the cone is tangent to the circle, we have $\dfrac{R}{L} = \sin{\theta}$ , or $R=L\sin{\theta}$ .

The equation of the circle is $(x-L)^2+y^2=R^2$ and the upper half is the graph of the function $y=\sqrt{R^2-(x-L)^2}$

Using the washer method, we find the volume of displaced water to be

$\large\begin{aligned} V & =\displaystyle\int_{L-R}^{h}\pi\left(\sqrt{R^2-(x-L)^2}\right)^2\,dx \\ &=\pi\left(L^2\sin^2{\theta h}-\dfrac{(h-L)^3}{3}-L^3\sin^2\theta(1-\sin{\theta})-\dfrac{L^3\sin^3{\theta}}{3}\right) \\ \dfrac{dV}{dL}&=\pi(L-h-L\sin{\theta})(L-h+L\sin\theta-2L\sin^2\theta) \\ \end{aligned}$

For $\large \dfrac{dV}{dL}=0 \\ \implies L=\dfrac{h}{1-\sin\theta} \text{ or } L=\dfrac{h}{1+\sin\theta-2\sin^2\theta}$ .

It can be shown by double derivative test that maxima ocuurs at

$\large L_{max}=\dfrac{h}{1+\sin\theta-2\sin^2\theta}$

Now,

$\large R_{max}=L_{max}\sin\theta=\dfrac{h\sin\theta}{1+\sin\theta-2\sin^2\theta}$

And $\large\boxed{V_{max}=\dfrac{4\pi h^3\sin^3{\theta}}{3(\sin{3\theta}+1)}}$

Putting $h=1$ and $\theta=\frac{\pi}{6}$ $\implies \boxed{V_{max}=\dfrac{\pi}{12}}$

Given $\displaystyle \theta=\frac{\pi}{6}$ and $h=1$ , than $\displaystyle \tan{\frac{\pi}{6}}=\frac{R_c}{h}$ , where $\displaystyle R_c$ is the radius of the cone. So, $\displaystyle R_c=\frac{\sqrt{3}}{3}$ . Let's place the longitudinal cross-section of the entire shape in the $xy$ plane, where the cone vertex is in $\displaystyle (0,0)$ and the center of the cone base in $(0,1)$ . This point is also the center of the sphere. The two sides of the cone cross-section are segments from $\displaystyle (0,0)$ to $\displaystyle \left( \frac{\sqrt{3}}{3},1\right)$ and from $\displaystyle (0,0)$ to $\displaystyle \left( -\frac{\sqrt{3}}{3},1\right)$ . The segments belong, respectively to $\displaystyle y=\sqrt{3}x$ and $\displaystyle y=-\sqrt{3}x$ . Now, to find the radius of the biggest sphere, so that we can spill out the maximum amount of water, is exactly the distance between $\displaystyle (0,1)$ and one of the two lines. That's because in this way we obtain the tangency between the sphere and the cone. So, if we choose $\displaystyle y=\sqrt{3}x$

$\displaystyle R_s=\frac{|0\sqrt{3}-1|}{\sqrt{1+3}}=\frac{1}{2}$ $\quad \quad \quad \star$

Due to the fact that only half of the sphere is soaked in the water, the volume of the spilled-out water is equal to a half of the sphere volume.

$\displaystyle \frac{V_s}{2}=\frac{1}{2}\cdot\frac{4}{3}\pi\cdot \left(\frac{1}{2}\right)^3=\frac{\pi}{12}$

---

Note $\star$ : I've used the point-line distance formula $\frac{|ax+by+c|}{\sqrt{a^2+b^2}}$