A Slight Deformation 2

First, we find the inradius, from the right triangle: $r = \dfrac{{a + b - c}}{2} = 6$ .

Now we move to the acute $\triangle ABC$ . Denote $I$ , $D$ , $E$ , $F$ , the incenter and the contact points of the incircle with the triangle, as seen on the figure. Let $CB=a$ , $CA=b$ , $(a<b)$ , $CD=CF=x$ , $\angle ICD=\angle ICF=\theta$ and area of $\triangle ABC=A$ .

The semiperimeter of $\triangle ABC$ is $s = \dfrac{{AB + BC + CA}}{2} = \dfrac{{a + b + \left( {a + b - 2x} \right)}}{2} = a + b - x \ \ \ (1)$

$a + b - c = 12 \Rightarrow a + b = 12 + c \ \ \ (2)$
Hence, $A = s \cdot r = \left( {a + b - x} \right) \cdot 6 \Rightarrow A = 6\left( {12 + c - x} \right) \ \ \ (3)$

$\left( 2 \right) \Rightarrow {\left( {a + b} \right)^2} = {\left( {12 + c} \right)^2} \Rightarrow {a^2} + {b^2} + 2ab = {\left( {12 + c} \right)^2} \Rightarrow {c^2} + 2ab = {\left( {12 + c} \right)^2} \Rightarrow ab = 72 + 12c \ \ \ (4)$

Again, $A = \frac{1}{2}CB \cdot CA \cdot \sin \left( {2\theta } \right) = \frac{1}{2}a \cdot b \cdot \sin \left( {2\theta } \right)\mathop = \limits^{\left( 4 \right)} \frac{1}{2}\left( {72 + 12c} \right) \cdot \sin \left( {2\theta } \right) \Rightarrow A = 6\left( {6 + c} \right) \cdot \sin \left( {2\theta } \right) \ \ \ (5)$

$\left( 3 \right),\left( 5 \right) \Rightarrow 12 + c - x = \left( {6 + c} \right)\sin \left( {2\theta } \right) \ \ \ (6)$

Alongside,
$\sin \left( {2\theta } \right) = \dfrac{{2\cot \theta }}{{1 + {{\cot }^2}\theta }} \ \ \ (7)$

From $\triangle CID$ we see that $\cot \theta = \dfrac{x}{6} \ \ \ (8)$ , hence $\left( 7 \right) \Rightarrow \sin \left( {2\theta } \right) = \dfrac{{12x}}{{36 + {x^2}}} \ \ \ (9)$

Combining $(6)$ , $(7)$ , $(8)$ , $(9)$ and solving for $c$ we get $c = \dfrac{{{x^2} - 6c + 72}}{{x - 6}} \ \ \ (10)$

Since the area loss is $\dfrac{3}{2}\left( {53 - \sqrt {1657} } \right)$ , we have $\dfrac{1}{2}ab - 6\left( {6 + c} \right) \cdot \sin \left( {2\theta } \right) = \dfrac{3}{2}\left( {53 - \sqrt {1657} } \right)$ and using $(4)$ , $(9)$ and $(10)$ , we find $x = \dfrac{{77 - \sqrt {1657} }}{4}$ .
We plug this in $(10)$ to get $c = \dfrac{{65}}{2}.$

Solving simultaneously $(2)$ and $(4)$ we find $a = \dfrac{{33}}{2}$ , $b=\boxed{28}$

For the first (right) triangle, the semi-perimeter is $s_1 = \frac{1}{2}(a + b + c)$ , and the area is $A_1 = \frac{1}{2}ab$ . For the second (acute) triangle, the semi-perimeter is $s_2 = \frac{1}{2}(a + b + x)$ , and the area is $A_2 = \frac{1}{4}\sqrt{(a + b + x)(-a + b + x)(a - b + x)(a + b - x)}$ .

Using the equation $r = \frac{A}{s}$ we have $r = \frac{A_1}{s_1} = \frac{A_2}{s_2}$ , so $\frac{\frac{1}{2}ab}{\frac{1}{2}(a + b + c)} = \frac{\frac{1}{4}\sqrt{(a + b + x)(-a + b + x)(a - b + x)(a + b - x)}}{\frac{1}{2}(a + b + x)}$ , which rearranges to:

$(a + b + c)^2(x - c)(x + a + b)(x - \frac{1}{2}(a + b - c + \sqrt{q}))(x - \frac{1}{2}(a + b - c - \sqrt{q})) = 0$

for $q = c^2 + 10(a + b)c - 7(a + b)^2$ , and has one solution $x = \frac{1}{2}(a + b - c + \sqrt{q})$ for $0 < x < c$ .

The difference of areas is then $\frac{3}{2}(53 + \sqrt{1657}) = A_1 - A_2 = \frac{1}{2}ab - \frac{1}{4}\sqrt{(a + b + x)(-a + b + x)(a - b + x)(a + b - x)}$ . Substituting the $x$ and $q$ equations above and simplifying gives $\frac{3}{2}(53 + \sqrt{1657}) = \frac{1}{8}(a + b - c)(3c - a - b - \sqrt{c^2 + 10(a + b)c - 7(a + b)^2})$ . Equating rational parts tells us $\frac{3}{2}\cdot 53 = \frac{1}{8}(a + b - c)(3c - a - b)$ , and substituting $a + b - c = 12$ and simplifying gives $3c - a - b = 53$ . Combining $a + b - c = 12$ and $3c - a - b = 53$ gives us $c = \frac{65}{2}$ and $a + b = \frac{89}{2}$ .

Using the Pythagorean Theorem on the first (right) triangle now gives $a^2 + b^2 = (\frac{65}{2})^2$ . Combining this with $a + b = \frac{89}{2}$ gives $2a^2 - 89a + 924 = 0$ and solves to $a = 28$ or $a = \frac{33}{2}$ , which are the values of the two legs of the right triangle, the longer one being $\boxed{28}$ .