Circle geometry by Guy Fox

Radius is exactly $\dfrac{72}{23}$ , pretty neat. See the figure below, where dotted lines from the center of the circumcribing circle passes through the midpoints of the sides of the triangle.

Let $B$ be the origin $(0,0)$ , and the center of circle of unknown radius $r$ be $(x,y)$ . Then we solve the following three equations to find $r$

$\dfrac{3}{2}+\sqrt{x^2+(\dfrac{3}{2}-y)^2}=r$

$2+\sqrt{(2-x)^2+y^2}=r$

$\dfrac{5}{2}+\sqrt{(2-x)^2+(\dfrac{3}{2}-y)^2}=r$

so that $(r, x, y)= \left(\dfrac{72}{23}, \dfrac{36}{23}, \dfrac{24}{23}\right)$

Note: The midpoints of the sides $(0,\dfrac{3}{2})$ , $(2,0)$ , and $(2,\dfrac{3}{2})$ are centers of circles of radii $\dfrac{3}{2}, 2, \dfrac{5}{2}$ .
Given any two tangent circles, their centers and point of tangency is colinear.

Bonus: If right triangle $ABC$ has rational sides, then $(r,x,y)$ are also all rationals.

Let's find tangent circle by applying inversion transformation, or equivalently working in complex domain and applying reciprocal transformation $w=1/\overline{z}$ .

Let (0,0), or the inversion centre, be the intersection of 3 circles. Other circle intersections are $(0,3)$ , $(4,0)$ and $\frac{12}{25}\left(3,4\right)$ .

Inversion transformation transform a circle passing through centre into a line, and other circles into circles. Thus, given three circles are transformed into lines, intersecting at 3 points; while big (still unknown) tangent circle transforms into circle tangent on those three lines. Transformed intersection coordinates are:

$\begin{array}{lcr} (0,3) && \longrightarrow&& \left(0,\frac{1}{3}\right)\\ (4,0) && \longrightarrow && \left(\frac{1}{4},0\right)\\ \frac{12}{25}\left(3,4\right) && \longrightarrow && \left(\frac{1}{4},\frac{1}{3}\right)\end{array}$

There are 4 circles that are tangent on all 3 lines: 1 inscribed and 3 escribed circle. The correct one has (0,0) in its interior. Finding coordinates of the centre and radius (solving quadratic equation) gives following circle:

$c=\left(-\frac{1}{6},-\frac{1}{4}\right)\quad r=\frac{1}{2}$

Transforming back escribed circle gives the required tangent circle. Take notice that, while points on a circle transform into a circle, a circle centre does not transform to a corresponding circle centre. Radius of transformed circle can be deduce by working on the line that connects origin and centre of the circle. Transforming 2 points of the circle that are collinear with circle centre and the origin gives.

$2R= \left|\frac{1}{|c|-r}-\frac{1}{|c|+r}\right|\\ R= \frac{r}{\left||c|^2-r^2\right|}\\ R=\frac{72}{23}$

Here is the picture with requested tangent circle, as well as one other tangent circle corresponding to the inscribed circle of the reciprocal space.

$M_{red}=\dbinom{0}{3/2}$ , $r_{red}=\dfrac{3}{2}$ , $M_{blue}=\dbinom{0}{2}$ , $r_{blue}=2$ , $M_{green}=\dbinom{2}{3/2}$ , $r_{green}=\dfrac{5}{2}$

$M_{black}=M=\dbinom{x}{y}$ , $r_{black}=r$

$length(v)=\dfrac{v}{\sqrt{v_{x}^{2}+v_{y}^{2}}}$ , $unitvec(v)=\dfrac{v}{length(v)}$

$d_{red}=length[M_{red}+unitvec(M_{red}-M)\times r_{red}-M]$

$d_{blue}=length[M_{blue}+unitvec(M_{blue}-M)\times r_{blue}-M]$

$d_{green}=length[M_{green}+unitvec(M_{green}-M)\times r_{green}-M]$

$\dbinom{0}{3/2}+\dfrac{\dbinom{0}{3/2}-\dbinom{x}{y}}{\sqrt{(0-x)^{2}+(3/2-y)^{2}}}\times \dfrac{3}{2}-M$ $\Rightarrow$ $d_{red}=\sqrt{\left(-x-\dfrac{3x}{\sqrt{4x^{2}+(3-2y)^{2}}}\right)^{2}+\left(3/2-y+\dfrac{9-6y}{2\sqrt{4x^{2}+(3-2y)^{2}}}\right)^{2}}$

$\dbinom{2}{0}+\dfrac{\dbinom{2}{0}-\dbinom{x}{y}}{\sqrt{(2-x)^{2}+(0-y)^{2}}}\times 2-M$ $\Rightarrow$ $d_{blue}=\sqrt{\left(2-x+\dfrac{4-2x}{\sqrt{(-2+x)^{2}+y^{2}}}\right)^{2}+\left(-y-\dfrac{2y}{\sqrt{(-2+x)^{2}+y^{2}}}\right)^{2}}$

$\dbinom{2}{3/2}+\dfrac{\dbinom{2}{3/2}-\dbinom{x}{y}}{\sqrt{(2-x)^{2}+(3/2-y)^{2}}}\times \dfrac{5}{2}-M$ $\Rightarrow$ $d_{red}=\sqrt{\left(2-x-\dfrac{5(-2+x)}{\sqrt{25+4x(-4+x)+4y(-3+y)}}\right)^{2}+\left(3/2-y+\dfrac{5(3/2-y)}{2\sqrt{(-2+x)^{2}+(-3/2+y)^{2}}}\right)^{2}}$

Setting these 3 equations to r and solving for x,y,r of the black circle results in:

$x=\dfrac{36}{23} , y=\dfrac{24}{23} , r=\dfrac{72}{23}$

It is simply the Apollonius Problem and solved by the following three quadratic equations using WolframAlpha for center (x,y) and radius 'r' of the biggest circle, which is tangent to other three circles;

(1) x^2+(y-3/2)^2=(r-3/2)^2, circle on side of length=3,

(2) (x-2)^2+y^2=(r-2)^2, circle on side of length =4,

(3) (x-2)^2+(y-3/2)^2=(r-5/2)^2. circle on side of length=5.

The solution is x=36/23, y=24/23 and radius 'r' is the Answer=72/23.

Argggghhhh. Couldn't come up with the centers and tangency being colinear thought.

I did a gross guess of center and radius and looked at the distance between intersection points with the smaller circles (w/ java). And then adjusted the center and radius while using the intersection distances in each circle as a guide to subsequent guesses until the accuracy was beyond 5 places. A decent mastery of geometry would have saved a lot of work.