Solving right triangles

Let the two legs be $a$ and $b$ , and the hypotenuse be $c$ . By pythagorean theorem , we have

$c^2=a^2+b^2$

Thus,

$a^2+b^2+c^2=338$

$c^2+c^2=338$

$2c^2=338$

$c^2=169$

$c=13$

From $a+b+c=30$ ,

$a=30-b-c$

$a=30-b-13$

$a=17-b$

Substituting, we get

$a^2+b^2+c^2=338$

$(17-b)^2+b^2+169=338$

$289-34b+b^2+b^2+169=338$

$b^2-17b+60=0$

$(b-12)(b-5)=0$

$b-12=0$

$b=12$

$b-5=0$

$b=5$

Therefore, $b$ can either be $12$ or $5$ . Accordingly, $a$ can be either $12$ or $5$ .

Thus, the length of the smallest side is $5$ .

Conclusion: The right triangle is $5-12-13$ right triangle.

The problem does not specify that the solution is over the integers or even positive values. But, in fact, the only non-complex solution of: $\text{Solve}\left[x+y+z=30\land x^2+y^2+z^2=338\land x^2+y^2=z^2,\{x,y,z\}\right]$ where the third constraint comes from the right triangle specification gives the $(5, 12, 13)$ solution.

The full solution set is $\{\{x\to 5.,y\to 12.,z\to 13.\},\{x\to 12.,y\to 5.,z\to 13.\}, \\ \{x\to 21.5\, -19.4358 i,y\to 21.5\, +19.4358 i,z\to -13.\}, \\ \{x\to 21.5\, +19.4358 i,y\to 21.5\, -19.4358 i,z\to -13.\}\}$

If the right triangle constraint is removed, then there is a smaller solution: $\frac{2}{3} \left(15-\sqrt{57}\right)$ or about $4.96677704315$ .