Arrrr

Let $r$ be the radius of the smaller circle. Let $A$ be the center of the unit circle, $B$ be the center of the smaller circle, and $C$ be the point where the horizontal line through $A$ and the vertical line through $B$ intersect.

Then $\Delta ABC$ is a right (isosceles) triangle with hypotenuse $AB = 1 + r$ and sides $AC = BC = 1 - r$ . We then use the Pythagorean Theorem to find that

$(1 + r)^{2} = (1 - r)^{2} + (1 - r)^{2}$

$\Longrightarrow 1 + 2r + r^{2} = 2*(1 - 2r + r^{2})$

$\Longrightarrow r^{2} - 6r + 1 = 0$

$\Longrightarrow r = \dfrac{6 \pm \sqrt{36 - 4}}{2} = 3 \pm 2\sqrt{2}$ .

Now we must have $r \lt 1$ , and so $r = 3 - 2\sqrt{2} = \boxed{0.172}$ to $3$ decimal places.

I use geometric progression.

Proof:

Let us draw infinitely circles, which against a right triangle and let the center of each circle is $A_1, A_2, A_3,...$ respectively. Let us trace the horizontal and the vertical line from all centers to the feet of an angle and we get squares. Since all squares from the horizontal and vertical line are similar, we may assume that there is a geometric progression.

Look at the largest square. The diagonal of the largest square are $\sqrt{2}$ . Since we know that the largest radius is 1, then the rest is $\sqrt{2}-1$ .

We have assumed that this is a geometric progression. So, the radius of the smaller circle is $a, a^2, a^3, ...$ respectively, with $a<1$ . It is easy to see that

$\sum_{n=1}^{\infty} 2a^{n} = \sqrt{2}-1$

$\frac{2 a}{1-a} = \sqrt{2}-1$

A little algebraic, we provide $a=\frac{\sqrt(2)-1}{\sqrt(2)+1} = (sqrt(2)-1)^2 = 0.1715728752538 \approx 0.172$

Since we want to find the small one, it's easy to get since $a=0.172$

Let $R =$ Radius of bigger circle and $r =$ Radius of smaller circle

The distance from the centre of the big circle to the origin is calculated using Pythagoras' theorem

$c = \sqrt {1^2 + 1^2}$

or in this case if we put $R$ instead

$c = \sqrt {2R^2}$

So we have the distance from the centre to the origin so let's take $R$ to get a value from the part of the smaller circle which touches the bigger one to the origin.

$\sqrt {2R^2} - R$

Now if we want to calculate $r$ we need an equation (in terms of $r$ ) equivalent to the above one

$\sqrt {2r^2} + r$

That should do it, so that means that

$\sqrt {2R^2} - R = \sqrt {2r^2} + r$

If we substitute in $1$ for $R$ we get

$\sqrt {2} - 1 = \sqrt {2r^2} + r$

Let's make a SURD out of that square root on the right

$r\sqrt{2} + r = \sqrt{2} - 1$

Then let's factorise

$r(\sqrt{2} + 1) = \sqrt{2} - 1$

Next divide by $(\sqrt{2} + 1)$

$r = \frac {\sqrt{2} - 1}{\sqrt{2} + 1}$

Since we don't want the root on the bottom we'll multiply it out

$r = \frac {(\sqrt{2}-1)(1-\sqrt{2})}{(\sqrt{2}+1)(1-\sqrt{2})}$

$r = \frac {-3 + 2\sqrt{2}}{-1}$

$r =3 - 2\sqrt{2}$

Working this out gives us

$r \approx 0.172$