Don't trip

The side lengths will be of the form $k(m^{2} - n^{2}), 2kmn$ and $k(m^{2} + n^{2})$ for positive integers $k,m,n$ with $m \gt n.$ Now we don't know at this point which of the first two of these terms will be the greater, so we can't say which is $a$ and which is $b$ , but since the condition we must adhere to, namely $ab = 6(a + b + c)$ , is symmetric for $a,b$ we don't need to know which term is $a$ and which is $b$ until the end of the solution. We do know, however, that the third term will be $c.$

Now regardless of which term is $a$ and which is $b$ , the given condition becomes

$2k^{2}mn(m^{2} - n^{2}) = 6k(2m^{2} + 2mn)$

$\Longrightarrow kn(m - n)(m + n) = 6(m + n) \Longrightarrow kn(m - n) = 6.$

Now we need to look in turn at the cases of $k = 1,2,3,6$ and then determine the possible values for $m,n.$ After these values are determined, we can then determine the associated triples, gather those that are distinct, and then add the $c$ values to obtain the desired solution. We have the following cases:

• $k = 1 \Longrightarrow (m,n) = (7,1), (5,2), (5,3), (7,6)$ ;

• $k = 2 \Longrightarrow (m,n) = (4,1), (4,3)$ ;

• $k = 3 \Longrightarrow (m,n) = (3,1), (3,2)$ ;

• $k = 6 \Longrightarrow (m,n) = (2,1).$

Plugging all these values in to our original side length expressions and eliminating the duplicate triples, we find that

$S = \{(14,48,50), (20,21,29), (16,30,34), (13,84,85), (18,24,30), (15,36,39)\}.$

The sum of all hypotenuse values is then $50 + 29 + 34 + 85 + 30 + 39 = \boxed{267}.$

$36(a+b)^2=(6c)^2=(ab-6(a+b))^2 \\36(a+b)^2=a^2b^2-12ab(a+b)+36(a^2+2ab+b^2) \\0=a^2b^2-12ab(a+b)+72ab \\0=ab-12(a+b)+72$

Now:

$(a-b)^2=(a+b)^2-4ab=(a+b)^2-48(a+b)+288=(a+b-24)^2-288 \\(a+b+24)^2-(a-b)^2=288 \\(2a-24)(2b-24)=288 \\(a-12)(b-12)=72$

And from here, we can check all factors of $72=2^3 \cdot 3^2$ . Here is a shortcut, though arguably unwarranted:

$12(a+b)-72=ab=6(a+b+c) \\a+b-12=c$

$\sum\limits_{0<j<k,jk=72} (j+12)+(k+12)-12 \\=\sum\limits_{0<j<k,jk=72} j+k+12 \\=(\sum\limits_{0<j<k,jk=72} j+k) +( \sum\limits_{0<j<k,jk=72} 12) \\=\sum\limits_{0<k|72} k +12 (N_{factors-of-72}/2) \\=(1+2+4+8)(1+3+9)+12((3+1)(2+1)/2) \\=15\cdot 13+12(6) \\=267$

Since $c = \sqrt{a^2 + b^2}$ the equation is

$ab = 6\left(a + b + \sqrt{a^2 + b^2} \right)$

$ab -6a - 6b = 6\sqrt{a^2 + b^2}$

Setting $ab > 6(a+b)$ we can square the sides and get

$ab(ab - 12a - 12b + 72) = 0$

This means $ab -12a - 12 b + 72 = a(b - 12) -(12b - 72) = 0$

So $(b -12) \vert (12b - 72) = (12b - 144 )+ 72 \ \Rightarrow \ (b - 12) \vert 72 = 2^33^2$ .

Keeping in mind that $b > a = 12 + \dfrac{72}{b-12} > 0$ we get the following acceptable values for $b$ , and consequently values for $a$ , and hence for $c$

$\displaystyle{(a,b,c) = \cases{ (18,24,30) \\ (15,36,39) \\ (20,21,29) \\ (16,30,34) \\ (14,48,50) \\ (13,84,85) }}$

Summing the $c$ s we get the right answer.

My soln is not as fancy as yours... Observe A(ABC)÷s=6(where s is semiperimeter). But A(ABC)÷s=r (where r is inradius) We know that for right angled triangle 2r=a+b+c. So a+b+c=12. So b >a> 12. Also we can get an equation in (b-a) as

(b-a)^2 + (12 - c)(b-a) + 72 = 0

put condition for (b-a) is real. we get c^2 - 24c - 144 > 0. that is c>28.97 Check value of a from 13 ,14, 15, ... So the values of c are 85 50 39 34 30 29.

$b = \frac{-6a - 6c}{6 - a}$

$a^2 - c^2 + b^2 = a^2 - c^2 + {(\frac{-6a - 6c}{6 - a}})^2 = 0$

$a \neq 6, (a^2 - c^2)(6-a)^2 + (-6a - 6c)^2 = 0$

$a(a + c)(72 - 12a + 12c - ac + a^2) = 0$

$a = 0$ : breaks constraints

$a = -c$ : breaks constraints

$72 - 12a + 12c - ac + a^2 = 0$ : Diophantine equation which can be solved using standard methods, and then filtered to fit the constraints.