What's the radius? (Inscribed circle)

OK, but let's simplify the answer! If $R$ is the radius of the large circle and $r$ the radius of the small circle, then (considering a radius of the quarter circle at the point of intersection of the quarter circle and the large circle, $R + R\sqrt{2} = 1$ , so that $R = \sqrt{2}-1$ . By considering the right-angled triangle with hypotenuse the line segment between the centres of the small and large circles, then $\begin{aligned} (R + r)^2 & = \; (R - r)^2 + \big(\sqrt{1-2r} - R\big)^2 \\ 4Rr & = \; \big(\sqrt{1- 2r} - R\big)^2 \; = \; R^2 + 1 - 2r - 2R\sqrt{1-2r} \\ 2R\sqrt{1-2r} & = \; R^2 + 1 - (4R+2)r \\ 2(\sqrt{2}-1)\sqrt{1-2r} & = \; 4 - 2\sqrt{2} - (4\sqrt{2}-2)r \\ \sqrt{1-2r} & = \; \sqrt{2} - (3 + \sqrt{2})r \\ 1 - 2r & = \; 2 - 2\sqrt{2}(3 + \sqrt{2})r + (11 + 6\sqrt{2})r^2 \\ (11 + 6\sqrt{2})r^2 - (2 + 6\sqrt{2})r - 1 & = \; 0 \\ (r - \sqrt{2} + 1)\big((11 + 6\sqrt{2})r - \sqrt{2}-1\big) & = \; 0 \end{aligned}$ Since we cannot have $r = \sqrt{2}-1=R$ , we deduce that $R \; = \; \frac{\sqrt{2}+1}{11 + 6\sqrt{2}} \; = \; \boxed{\frac{5\sqrt{2}-1}{49}}$ which is equal to, but much more simply expressed than, the official answer of $\frac{9 - 7\sqrt{2} + 2\sqrt{34 - 24\sqrt{2}}}{4\sqrt{2}-9}$ To see that these are the same, note that $\sqrt{34-24\sqrt{2}} = 3\sqrt{2}-4$ .

Let point $C$ be the origin $(0,0)$ of $xy$ -plane and flip the diagram vertically. Let the radius of the big circle be $r_1$ , then $r_1 + \sqrt 2r_1 = 1$ $\implies r_1 = \frac 1{\sqrt 2 + 1} = \sqrt 2 - 1$ . Let the radius of the red circle be $r$ , and the coordinates of the center of the red circle be $O (x, r)$ . Note that $O$ is the intersection of circle $x^2 + y^2 = (1-r)^2$ and circle $(x-r_1)^2 + (y-r_1)^2 = (r_1+r)^2$ , when $y = r$ . That is the point $O (x,r)$ satisfies:

$\begin{cases} x^2 + r^2 = (1-r)^2 & \implies x^2 = 1 - 2r & ...(1) \\ (x-r_1)^2 + (r-r_1)^2 = (r_1+r)^2 & \implies (x-r_1)^2 = 4r_1r & ...(2) \end{cases}$

From (2):

$\begin{aligned} x^2 - 2r_1x + r_1^2 & = 4r_1r \\ \frac {x^2}{r_1} - 2x + r_1 & = 4r & \small \color{#3D99F6} \text{blue} \text{Divide both sides by }r_1 \\ (1-2r)(\sqrt 2+1) - 2\sqrt{1-2r} + \sqrt 2 - 1 & = 4r & \small \color{#3D99F6} \text{Note that }x^2 = 1-2r, r_1 = \sqrt 2 - 1 \\ \sqrt 2 - (3+\sqrt 2)r & = \sqrt{1-2r} & \small \color{#3D99F6} \text{Squaring both sides} \\ (11+6\sqrt 2)r^2 - (6\sqrt 2 + 4)r + 2 & = 1 - 2r \\ (11+6\sqrt 2)r^2 - (6\sqrt 2 + 2)r + 1 & = 0 \end{aligned}$

$\begin{aligned} \implies r & = \frac {\sqrt 2 + 1}{11+6\sqrt 2} = \frac {5\sqrt 2-1}{49} = \color{#3D99F6} \boxed{\frac {9-7\sqrt 2+2\sqrt{34-24\sqrt 2}}{4\sqrt 2 - 9}} \quad \small \text{See note.} \end{aligned}$

---

Note:

$\begin{aligned} X & = \frac {9-7\sqrt 2+2\color{#3D99F6} \sqrt{34-24\sqrt 2}}{4\sqrt 2 - 9} & \small \color{#3D99F6} \text{Hint from Mark Hennings: } (3\sqrt 2 - 4)^2 = 34-24\sqrt 2 \\ & = \frac {9-7\sqrt 2+2\color{#3D99F6} (3\sqrt 2-4)}{4\sqrt 2 - 9} \\ & = \frac {1-\sqrt 2}{4\sqrt 2 - 9} \\ & = \frac {(\sqrt 2-1)(9+4\sqrt 2)}{(9-4\sqrt 2)(9+4\sqrt 2)} \\ & = \frac {5\sqrt 2 - 1}{49} \end{aligned}$