Largest Square inside the Triangle

Let one of the vertex of the square be at the distance of $a$ from one vertex of the equilateral triangle as shown in figure. Now the two sides of square from this vertex is at angle of $\theta$ and $90\degree - \theta$ as shown. Let the side length of square be $s$ .

Now applying sine rule in both the smaller triangle, we get

$\Large \frac{s}{\text{sin}60\degree} = \frac{a}{\text{sin}(30\degree + \,\theta)}\hspace{50pt}$ and $\hspace{50pt}\Large \frac{s}{\text{sin}60\degree} = \frac{1-a}{\text{sin}(120\degree - \, \theta)}$

From both the equation we get

$a\,\text{sin}(120\degree - \theta) = (1-a)\text{sin}(30\degree + \theta)\newline \Rightarrow (\text{sin}(30\degree+ \theta) + \text{sin}(120\degree - \theta))a = \text{sin}(30\degree + \theta)\newline \Rightarrow 2a\,\text{sin}75\degree\text{cos}(45\degree-\theta) = \text{sin}(30\degree+\theta)$

$\Rightarrow a = \Large \frac{\text{sin}(30\degree+\theta)}{2\,\text{sin}75\degree\text{cos}(45\degree-\theta)}$

From $a$ we get the expression of $s$ as

$s = \Large\frac{\text{sin}60\degree}{ 2\,\text{sin}75\degree\text{cos}(45\degree-\theta)}$

Now, in order to maximize $s$ , we have to minimize $\text{cos}(45\degree-\theta)$ over some range of $\theta$ . We have to determine that range.

Two sides of the square are done, now the other two sides can be constructed inside the triangle if and only if the red marked angles in the figure is greater than or equal to $90\degree$ . Expressing this condition in the form of equation :

$60\degree+\theta \geq 90\degree \hspace{50pt}$ and $\hspace{50pt} 150\degree-\theta \geq 90\degree$

$\Rightarrow 30\degree\leq\theta\leq60\degree\newline \Rightarrow -15\degree\leq45\degree-\theta\leq 15\degree\newline \Rightarrow\text{cos} 15\degree \leq \text{cos}(45\degree-\theta)\leq1$

The minimum value of $\text{cos}(45\degree-\theta)$ is $\text{cos}15\degree$ which occurs either at $\theta = 30\degree$ or $\theta = 60\degree$

So, $\, s_{max} = \Large\frac{\text{sin}60\degree}{2\,\text{sin}75\degree\text{cos}15\degree} = \frac{\text{sin}60\degree}{2\,\text{cos}^2 15\degree} = \frac{\text{sin}60\degree}{1 + \text{cos}30\degree} = \frac{\text{sin}60\degree}{1 + \text{sin}60\degree} = \frac{\sqrt{3}}{2+\sqrt{3}}$ $= \sqrt{3}(2 - \sqrt{3}) = 2\sqrt{3} - 3 \approx 0.4641$