How Big Is the Red Square?

Suppose the unit circle is centered at the origin and the red square is symmetric about the (negative) $x$ -axis.

A diagonal of the green square has the same measure as the diameter of the circle, so the side of the square has length $\frac{2}{\sqrt{2}} = \sqrt{2}$ . The leftmost side of the green square thus lies along the line $x = -\frac{\sqrt{2}}{2}$ .

Now the equation of the unit circle is $x^{2} + y^{2} = 1$ . So if the $y$ -coordinates of the upper and lower vertices of the red square are $a$ and $-a$ , respectively, (with $a \gt 0$ ), then the height of the red square will be $2a$ and the width will be $\sqrt{1 - a^{2}} - \frac{\sqrt{2}}{2}$ . Since the height and width must be equal, we must have

$\sqrt{1 - a^{2}} - \frac{\sqrt{2}}{2} = 2a$

$\Longrightarrow (1 - a^{2}) - \sqrt{2(1 - a^{2})} + \frac{1}{2} = 4a^{2}$

$\Longrightarrow -\sqrt{2(1 - a^{2})} = 5a^{2} - \frac{3}{2}$

$\Longrightarrow 2 - 2a^{2} = 25a^{4} - 15a^{2} + \frac{9}{4}$

$\Longrightarrow 100a^{4} - 52a^{2} + 1 = 0$

$\Longrightarrow a^{2} = \dfrac{52 \pm \sqrt{52^{2} - 4*100}}{200} = \dfrac{52 \pm 48}{200}$ ,

so either $a^{2} = \frac{1}{2}$ or $a^{2} = \frac{4}{200} = \frac{1}{50}$ . Now the first of these corresponds to the green square, so for the red square we must have that $a^{2} = \frac{1}{50}$ .

The area of the red square will then be $(2a)^{2} = 4a^{2} = \frac{4}{50} = \frac{2}{25} = \boxed{0.08}$ .

Let A be left top corner of small square ABCD, side S, go counter clockwise, O the center to the right.
M midpoint of AB, N of DC.
$2*NO=side~ of ~the~ big~ square=\sqrt2. ~~~~~~\because~circle~Diameter~ = ~Big~square~Diagonal. \\ In ~rt.~\Delta ~AMO:-~~~~~ MO=S+NO=S+\dfrac {\sqrt2} 2.~~~~~~~~~~~~AM=\frac 1 2*S~~~~~~~~~~~~Hyp.~AO=1. \\ \therefore~MO^2+AM^2=AO^2,~~~~\implies~(S+\dfrac {\sqrt2} 2)^2 + (\frac 1 2*S)^2=1.\\ Solving~ quadratic~in~S, ~\frac 5 4 *S^2+\sqrt2*S-\frac 1 2=0,~~~~~~~~ S^2=\dfrac 2 {25}= \Large~~\color{#D61F06}{0.08}$

Side of red square, $a$ , is the difference between $OC=\cos \alpha$ and $OB=\frac{1}{\sqrt{2}}$ .

$a=\cos\alpha-\frac{1}{\sqrt{2}}$

From $\triangle DCC$ we can see that $\frac{a}{2}=\sin\alpha$ or $a=2\sin\alpha$

Comparing the two produces an equation in $\sin\alpha$

$\sqrt{1-\sin^2\alpha}-\frac{1}{\sqrt{2}}=2\sin\alpha$

the solution of which is $\sin\alpha=\frac{1}{2\sqrt{5}}$

From that $a=2\sin\alpha=\frac{\sqrt{2}}{5}$

and the area $A=a^2=\frac{2}{25}=\boxed{0.08}$