Where does the smallest square lie?

Figure 1 Let $AB=c=13$ , $BC=a=14$ and $CA=b=15$ . Denote $\angle A$ , $\angle B$ and $\angle C$ by $\alpha$ , $\beta$ and $\gamma$ respectively.
Square $KLMN$ has its vertices $K$ , $L$ and $N$ on sides $AB$ , $AC$ and $BC$ respectively as seen in figure 1. While $K$ takes all possible positions on $AB$ , angle ${{\theta }_{c}}=\angle AKL$ varies, with $90{}^\circ -\alpha \le {{\theta }_{c}}\le \beta$ . At the same time, the side length ${{s}_{c}}$ of the square changes. We want to determine the minimum value ${{s}_{c}}$ can take. Denote the length of segment $BK$ by $x$ .
Using Sine Rule on $\triangle AKL$ ,

$\frac{AK}{\sin L}=\frac{KL}{\sin A}\Rightarrow \frac{c-x}{\sin \left( 180{}^\circ -\alpha -{{\theta }_{c}} \right)}=\frac{{{s}_{c}}}{\sin a}\Rightarrow \frac{c-x}{\sin \left( \alpha +{{\theta }_{c}} \right)}=\frac{{{s}_{c}}}{\sin a} \ \ \ \ \ \ (1)$ Using Sine Rule on $\triangle BKN$ ,

$\frac{BK}{\sin N}=\frac{KN}{\sin B}\Rightarrow \frac{x}{\sin \left( 90{}^\circ +\theta -\beta \right)}=\frac{{{s}_{c}}}{\sin \beta }\Rightarrow x=\frac{{{s}_{c}}}{\sin \beta }\cos \left( \beta -{{\theta }_{c}} \right) \ \ \ \ \ \ (2)$ Combining $(1)$ and $(2)$ we get

${{s}_{c}}\left[ \sin \left( \alpha +{{\theta }_{c}} \right)\sin \beta +\sin \alpha \cos \left( \beta -{{\theta }_{c}} \right) \right]=c\sin \alpha \sin \beta \ \ \ \ \ \ (3)$ Hence, ${{s}_{c}}$ is minimised when the function $f\left( {{\theta }_{c}} \right)=\sin \left( \alpha +{{\theta }_{c}} \right)\sin \beta +\sin \alpha \cos \left( \beta -{{\theta }_{c}} \right)$ , $90{}^\circ -\alpha \le {{\theta }_{c}}\le \beta$ takes its maximum value.

Taking the derivative, ${f}'\left( {{\theta }_{c}} \right)=\cos \left( \alpha +{{\theta }_{c}} \right)\sin \beta +\sin \alpha \sin \left( \beta -{{\theta }_{c}} \right)$ For the critical point,

$\begin{aligned} {f}'\left( {{\theta }_{c}} \right)=0 & \Leftrightarrow \cos \left( \alpha +{{\theta }_{c}} \right)\sin \beta +\sin \alpha \sin \left( \beta -{{\theta }_{c}} \right)=0 \\ & \Leftrightarrow \left( \cos \alpha \cos {{\theta }_{c}}-\sin \alpha \sin {{\theta }_{c}} \right)\sin \beta +\sin \alpha \left( \sin \beta \cos {{\theta }_{c}}-\sin {{\theta }_{c}}\cos \beta \right)=0 \\ & \Leftrightarrow \sin {{\theta }_{c}}\sin \alpha \left( \sin \beta +\cos \beta \right)=\cos {{\theta }_{c}}\sin \beta \left( \sin \alpha +\cos \alpha \right) \\ & \Leftrightarrow \frac{\sin {{\theta }_{c}}}{\cos {{\theta }_{c}}}=\frac{\sin \beta \left( \sin \alpha +\cos \alpha \right)}{\sin \alpha \left( \sin \beta +\cos \beta \right)} \\ & \Leftrightarrow \tan {{\theta }_{c}}=\dfrac{\dfrac{\sin \beta \left( \sin \alpha +\cos \alpha \right)}{\sin \alpha \sin \beta }}{\dfrac{\sin \alpha \left( \sin \beta +\cos \beta \right)}{\sin \alpha \sin \beta }} \\ & \Leftrightarrow \boxed{\tan {{\theta }_{c}}=\dfrac{1+\cot \alpha }{1+\cot \beta }} \\ \end{aligned}$

Thus, the only zero of the function ${f}'$ is at ${{\theta }_{c}}^{*}={{\tan }^{-1}}\left( \frac{1+\cot \alpha }{1+\cot \beta } \right)$ Furthermore, ${f}''\left( {{\theta }_{c}} \right)=-\sin \left( \alpha +{{\theta }_{c}} \right)\sin \beta -\sin \alpha \cos \left( \beta -{{\theta }_{c}} \right)<0$ for all ${{\theta }_{c}}\in \left[ 90{}^\circ -\alpha ,\ \beta \right]$ . This means that the function $f$ has a maximum at ${{\theta }_{c}}^{*}$ . Consequently, from $(3)$ we get the minimum for the side length of the square ${{s}_{c}}^{*}=\frac{c\sin \alpha \sin \beta }{\sin \left( \alpha +{{\theta }_{c}}^{*} \right)\sin \beta +\sin \alpha \cos \left( \beta -{{\theta }_{c}}^{*} \right)} \ \ \ \ \ \ (4)$

In order to calculate the exact value of ${{s}_{c}}^{*}$ we proceed to evaluate the trigonometric numbers of the angles of $\triangle ABC$ .

---

Figure 2 $\triangle ABC$ is a compound of a $5$ - $12$ - $13$ and a $9$ - $12$ - $15$ right triangle (see figure 2).

Hence, $\sin \beta =\dfrac{12}{13}$ , $\cos \beta =\dfrac{5}{13}$ , $\cot \beta =\dfrac{5}{12}$ , $\sin \gamma =\dfrac{12}{15}=\dfrac{4}{5}$ , $\cos \gamma =\dfrac{9}{15}=\dfrac{3}{5}$ , $\cot \gamma =\dfrac{9}{12}=\dfrac{3}{4}$ .

Moreover, $\sin \alpha =\sin \left( \beta +\gamma \right)=\sin \beta \cos \gamma +\sin \gamma \cos \gamma =\dfrac{56}{65}$ , $\cos \alpha =-\cos \left( \beta +\gamma \right)=-\cos \beta \cos \gamma -\sin \beta \sin \gamma =\dfrac{33}{65}$ and $\cot \alpha =\dfrac{\sin \alpha }{\cos \alpha }=\dfrac{33}{56}$ .

Using these values we find $\tan {{\theta }_{c}}^{*}=\dfrac{1+\cot \alpha }{1+\cot \beta }=\dfrac{1+\dfrac{33}{56}}{1+\dfrac{5}{12}}=\dfrac{267}{238}$ Now, $\begin{aligned} & \sin \left( \alpha +{{\theta }_{c}}^{*} \right)=\sin \alpha \cos {{\theta }_{c}}^{*}+\sin {{\theta }_{c}}^{*}\cos \alpha \\ & =\sin \alpha \cdot \dfrac{1}{\sqrt{1+{{\tan }^{2}}{{\theta }_{c}}^{*}}}+\tan {{\theta }_{c}}^{*}\cdot \dfrac{1}{\sqrt{1+{{\tan }^{2}}{{\theta }_{c}}^{*}}}\cdot \cos \alpha \\ & =\dfrac{56}{65}\cdot \dfrac{1}{\sqrt{1+{{\left( \dfrac{267}{238} \right)}^{2}}}}+\dfrac{267}{238}\cdot \dfrac{1}{\sqrt{1+{{\left( \dfrac{267}{238} \right)}^{2}}}}\cdot \dfrac{33}{65} \\ & =\dfrac{131}{5\sqrt{757}} \\ \end{aligned}$ And similarly, $\cos \left( \beta -{{\theta }_{c}}^{*} \right)=\dfrac{26}{\sqrt{757}}$ .

---

Having all the ingredients,
$\left( 4 \right)\Rightarrow {{s}_{c}}^{*}=\dfrac{13\cdot \dfrac{56}{65}\cdot \dfrac{12}{13}}{\dfrac{131}{5\sqrt{757}}\cdot \dfrac{12}{13}+\dfrac{56}{65}\cdot \dfrac{26}{\sqrt{757}}}=\dfrac{168}{\sqrt{757}}\approx 6.1$

Thus, the area of the smallest square with one vertex on side $c$ and its opposite vertex moving inside the square is $A={{\left( {{s}_{c}}^{*} \right)}^{2}}=\dfrac{28224}{757}\approx 37.28$ Working similarly, we find that the smallest square with one vertex on side $a$ and its opposite vertex moving inside the square has area $\dfrac{14112}{365}\approx 38.66$ and the smallest square with one vertex on side $b$ and the opposite one moving inside the square has area $\dfrac{28224}{701}\approx 40.26$ .

Comparing the results, we know that the area of the square in question is $A=\dfrac{28224}{757}$ .

For the answer, $p+q=28224+757=\boxed{28981}$ .

$\ \ \$

Note :

${f}''\left( {{\theta }_{c}} \right)<0$ , for all ${{\theta }_{c}}\in \left[ 90{}^\circ -\alpha ,\ \beta \right]$ , hence ${f}'$ is a decreasing function. Therefore, ${f}'\left( {{\theta }_{c}} \right)>0$ for ${{\theta }_{c}}\in \left[ 90{}^\circ -\alpha ,\ {{\theta }_{c}}^{*} \right)$ and ${f}'\left( {{\theta }_{c}} \right)<0$ for ${{\theta }_{c}}\in \left( {{\theta }_{c}}^{*},\ \beta \right]$ . Consequently, $f$ is increasing in $\left[ 90{}^\circ -\alpha ,\ {{\theta }_{c}}^{*} \right]$ and decreasing in $\left[ {{\theta }_{c}}^{*},\ \beta \right]$ ,giving its minimum when ${{\theta }_{c}}=90{}^\circ -\alpha$ , or ${{\theta }_{c}}=\beta$ .

Eventually, when the largest square occurs, all its vertices lay on the sides of the triangle. This result holds, not only for the $13$ - $14$ - $15$ triangle, but for all acute triangles.