Circle Partially in Triangle

BC is a tangent to the circle & centre of the circle (O) lies on AB (which is perpendicular to BC).

Hence BC can be tangent to circle only at B.

Hence, OB=r & AO=(5-r)

Let CA (=13) be tangent to circle at M.

Then, CM=CB (tangents from same point C) = 12

Also, OM=r & AM=13-12=1

Consider now triangle AOM,

$AO^{2}=AM^{2}+OM^{2}$

$(5-r)^{2}=1^{2}+r^{2}$

On solving, $\boxed{r=2.4}$

We see that $\triangle{ABC}$ is a $5-12-13$ triangle.

If we reflect point $C$ over $AB$ to $C'$ , we find that the circle in question is now the incircle of $\triangle{AC'C}$ , which has side lengths $13, 13$ and $24$ .

Since the height is $5$ , the area of the new triangle is $\frac{5 * 24}{2} = 60$ . We also see that the semi-perimeter is $25$ . Using the formula $a = rs$ , we find $60 = 25r$ , and thus $r=\boxed{2.4}$ .