Weird Incircle

Although this is a special case of the general sangaku, this particular problem can be solved a little more simply. The centre of the big circle lies on the perpendicular bisector of $AB$ . If we put the origin of our coordinate system at the midpoint of $AB$ , so that $A\;(-10,0)$ and $B\;(10,0)$ , then the centre $Q$ of the large circle has coordinates $(0,-24)$ . We also note the right-angled triangle $ABD$ , which is similar to a $(3,4,5)$ triangle with third coordinate $D\;(\tfrac{14}{5},-\tfrac{48}{5})$ . The triangle $BCD$ is a $(5,12,13)$ triangle, giving $C\;(-\tfrac65,-\tfrac{33}{5})$ , so that $AC = 11$ and $BC = 13$ .

The incentre $I$ of the triangle $ABC$ is given by the formula $\overrightarrow{OI} \; = \; \frac{1}{44}\left[20\overrightarrow{OC} + 13\overrightarrow{OA} + 11\overrightarrow{OB}\right]$ and hence $I$ has coordinates $(-1,-3)$ . Thus a general point $X$ on the angle bisector of $\angle ACB$ has position vector $\overrightarrow{OX} \; = \; \tfrac15(\lambda-6)\mathbf{i} + \tfrac15(18\lambda-33)\mathbf{j}$ for $\lambda \ge 0$ , and the distance of this point from the lines $AC$ and $BC$ is $3\lambda$ . We therefore need to find the value of $\lambda$ such that $3\lambda = 26 - QX$ , and hence $\begin{aligned} 26 - 3\lambda & = \; \tfrac15\sqrt{(\lambda-6)^2 + (18\lambda + 87)^2} \\ 100\lambda^2 + 7020\lambda - 9295 & = \; 0 \\ 5(10\lambda - 13)(2\lambda + 143) & = \; 0 \end{aligned}$ and hence the radius of the smaller circle is $3\lambda = \boxed{\tfrac{39}{10}}$ .

After wrong answer three times this is the fourth solution.
This solution is based on basic knowledge ONLY.

In General there are four solutions. One circle radius 26 has center P(-11.4,-13.2 ) BELOW side AB. Two cases. Internal Contact / External Contact
Two circle radius 26 has center P'(-11.4,-13.2 ) ABOVE side AB. Two cases. Internal Contact / External Contact
The above proof is for P(-11.4,-13.2 ) BELOW side AB. I Internal Contact. ( C is inside the 26 circle.
Figs Below is just to show the four cases. Radii given is approximate. the points of contact E,H; and U,V are shown.

Click here and here for the solution