The Shrinking Square

Here are outlines of two possible methods, one with integration and one without ....

(i) To make integration easier, rotate the circle and square counterclockwise about $O$ an angle of $\arctan(2)$ radians so that side $AD$ is horizontal and side $AB$ is vertical. Under this rotation the coordinates of $A$ then become

$A'(0.1304951, 3.7888545) = A'(a,b)$ .

Now look at the area of the region inside the circle that is above $y = b$ and to the right of $x = a$ . Now the line $x = a$ intersects the circle at

$P(a,9.36206) = P(a,c)$ ,

and the line $y = b$ intersects the circle at

$Q(4.79744,b) = Q(d,b)$ .

The area of this region is then

$\displaystyle\int_{a}^{d} (\sqrt{64 - (x + 3)^{2}} + 2 - b) dx = 17.2855$ .

Now note that the maximum of $(c - b)$ and $(d - a)$ is $5.5732255$ , which when squared is $31.060842$ , which is less than half the area of the region calculated above. This indicates that both sides of the desired square must extend beyond the circle. This in turn means that the area calculated above will be half the area of the desired square, and hence the side length of said square will be

$\sqrt{2*17.2855} = \boxed{5.8797}$ to $4$ decimal places.

(Edit: This is the graph of the (rotated) solution square.)

For the second method we won't need to rotate the diagram. By a similar argument as above, we can conclude that the sides of the desired square will extend beyond the circle. Labeling as $R$ and $S$ the respective points of intersection of sides $AB$ and $AD$ with the circle, we then need to subtract the areas of triangles $\Delta OAR$ and $\Delta OAS$ from that of sector $ORS$ to obtain the area of region $ARS$ . Once found, we then proceed as before by doubling this value and taking its square root to find the side length of the desired square. This involves the determination of the coordinates of points $R$ and $S$ , distances $AR$ and $AS$ , angles $\angle AOR$ and $\angle AOS$ , areas of the two triangles using Heron's formula and the area of the sector using the formula $(\frac{1}{2})*8^{2}*(\angle AOR + \angle AOS)$ , This is laborious but straightforward, and seeing that I've been typing for some while I will leave it to the reader to confirm that this will return the same value for the side length of the desired square. (It does, I assure you. :) )

The equation of two lines(the sides of the square) is y = x/2 (AB) and y=-2x (AD) the intersection of line AD to the circle is (2.0871,1.0436) and the intersection of line AB to the circle is (4.9848,2.4924). Integrating line AB - line AD from 0 to 2.0871 and line AB to lower part of circle (2-sqrt(64-(x+3)^2)) from 2.0871 to 4.9848 we get the area to be 17.2756 square units. The side of the square is the square root of 2 times 17.2756, which is 5.878 units.