Not Three, But Four Incircles!

Label the diagram as follows, and let $AF = ED = k$ :

By the law of cosines on $\triangle ADE$ , $AD = \sqrt{DE^2 + AE^2 - 2 \cdot DE \cdot AE \cdot \cos \angle DEA} = \sqrt{k^2 + 1 - 2 \cdot k \cdot \frac{1}{2}} = \sqrt{k^2 - 2k + 1}$ .

Since $\triangle ABF \sim \triangle AED$ by AA similarity, $AB = AF \cdot \cfrac{AE}{AD} = k \cdot \cfrac{1}{\sqrt{k^2 - 2k + 1}} = \cfrac{k}{\sqrt{k^2 - 2k + 1}}$ and $BF = AF \cdot \cfrac{DE}{AD} = k \cdot \cfrac{k}{\sqrt{k^2 - 2k + 1}} = \cfrac{k^2}{\sqrt{k^2 - 2k + 1}} = CD$ .

Then $BC = AD - AB - CD = \sqrt{k^2 - 2k + 1} - \cfrac{k}{\sqrt{k^2 - 2k + 1}} - \cfrac{k^2}{\sqrt{k^2 - 2k + 1}} = \cfrac{1 - 2k}{\sqrt{k^2 - 2k + 1}}$ .

As an incircle of the red equilateral, $r = \cfrac{1}{2\sqrt{3}} \cdot BC = \cfrac{1}{2\sqrt{3}} \cdot \cfrac{1 - 2k}{\sqrt{k^2 - 2k + 1}} = \cfrac{1 - 2k}{2\sqrt{3}\sqrt{k^2 - 2k + 1}}$ .

As an incircle of $\triangle ADE$ , $r = \cfrac{DE \cdot AE \cdot \sin \angle DEA}{DE + AE + AD} = \cfrac{k \cdot 1 \cdot \frac{\sqrt{3}}{2}}{k + 1 + \sqrt{k^2 - k + 1}} = \cfrac{\sqrt{3}k}{2(k + 1 + \sqrt{k^2 - 2k + 1})}$ .

So $r = \cfrac{1 - 2k}{2\sqrt{3}\sqrt{k^2 - 2k + 1}} = \cfrac{\sqrt{3}k}{2(k + 1 + \sqrt{k^2 - 2k + 1})}$ , which solves numerically to $r \approx 0.122761$ and $k \approx 0.311549$ .

Therefore, $\lfloor 10^5 r \rfloor = \boxed{12276}$ .

Label the unit equilateral triangle $ABC$ , the bottom red line be $AD$ , the centers of the central circle and the right circle be $O$ and $P$ respectively. Let $OM$ and $PN$ be perpendicular to $AB$ , $OM$ cuts $AD$ at $E$ , and the central circle tangent to $AD$ at $F$ .

Let $\angle DAB = \angle EOF = \theta$ . Then we have:

$\begin{aligned} AN + NB & = AB \\ r \cot \frac \theta 2 + r \cot 30^\circ & = 1 & \small \blue{\text{Let }t = \tan \frac \theta 2} \\ \frac rt + \sqrt 3t & = 1 \\ \implies t & = \frac 1{\frac 1t + \sqrt 3} = \frac t{\sqrt 3 t +1} \end{aligned}$

Also

$\begin{aligned} \frac {EM}{AM} & = \tan \angle DAB \\ \frac {OM - OE}{AM} & = \tan \angle DAB \\ \frac {\frac 1{2\sqrt 3} - r \sec \theta}{\frac 12} & = \tan \theta \\ \cos \theta - 2\sqrt 3 r & = \sqrt 3 \sin \theta \\ \implies r & = \frac {\cos \theta - \sqrt 3 \sin \theta}{2\sqrt 3} = \frac {1-t^2-2\sqrt 3 t}{2\sqrt 3 (1+t^2)} \end{aligned}$

Therefore we have:

$\begin{aligned} \frac {1-t^2-2\sqrt 3 t}{2\sqrt 3 (1+t^2)} & = \frac t{\sqrt 3 t +1} \\ (1-2\sqrt 3 t - t^2)(1+\sqrt 3 t) & = 2\sqrt 3 t (1+t^2) \\ 3\sqrt 3 t^3 + 7 t^2 + 3 \sqrt 3 t - 1 & = 0 \\ \implies t & \approx 0.155912545 \\ \implies r & = \frac t{\sqrt 3 t + 1} \approx 0.1227611 \end{aligned}$

Therefore $\lfloor 10^5r \rfloor = \boxed{12276}$ .