Squaring the pentagon?

We need to find the measure of $\angle EDF = \theta$ . But first let us find the measure of $\frac 12 \angle B)$ :

$\begin{aligned} \frac {2 \tan \frac B2}{1-\tan^2 \frac B2} & = \tan B = \frac 43 \\ 2 \tan^2 \frac B2 + 3 \tan \frac B2 - 2 & = 0 \\ \implies \tan \frac B2 & = \frac 12 \\ \tan \angle BDA & = 2 \\ \tan \angle BDC & = -2 \\ \implies \frac {2\tan \theta}{1-\tan^2 \theta} & = -2 \\ \tan^2 \theta - \tan \theta - 1 & = 0 \\ \implies \tan \theta & = \blue \varphi = \frac {1+\sqrt 5}2 & \small \blue{\text{the golden ratio}} \end{aligned}$

Now consider the sizes of the pentagon and square. Let the radius of the pentagon be $1$ and its height $h = 1 + \cos 36^\circ$ and its area $A_5 = \frac 52 \sin 72^\circ \approx 2.377641291$ . Let the side length of the square be $a$ and its left side is a distance $b$ from the central axis of the pentagon. Then at the bases of the pentagon and square $a\cot \theta = \sin 36^\circ + b \implies b = a \cot \theta - \sin 36^\circ$ and the height of the pentagon:

$\begin{aligned} b \tan 36^\circ + a & = h \\ (a \cot \theta - \sin 36^\circ)\tan 36^\circ + a & = 1 + \cos 36^\circ \\ \implies a & = \frac {1+\cos 36^\circ + \sin 36^\circ \tan 36^\circ}{1+\cot \theta \tan 36^\circ} \\ & \approx 1.543150314 \\ \implies \blue {A_\square} = a^2 & \approx 2.381312893 \end{aligned}$

Therefore, the area of square $A_\square \boxed > A_p$ .

Let the top vertex of the pentagon be $G$ and the top left vertex of the square be $H$ . Place the whole diagram on a coordinate plane so that $A$ is at $A(0, 0)$ , $B$ is at $B(0, 3)$ , and $C$ is at $C(4, 0)$ .

Then $BC$ is on the line through $B(0, 3)$ and $C(4, 0)$ , which is $y = -\frac{3}{4}x + 3$ .

Since $BD$ bisects $\angle ABC$ , $AD = 3 \tan (\frac{1}{2} \tan^{-1} \frac{4}{3}) = \frac{3}{2}$ , so $D$ is at $D(\frac{3}{2}, 0)$ .

Since $DE$ bisects $\angle BDC$ , $\tan \angle EDF = \tan \frac{1}{2} \angle BDC = \tan \frac{1}{2} (180° - \angle BDA) = \cot (\frac{1}{2} \angle BDA) = \cot (\frac{1}{2} \tan^{-1} 2) = \frac{1}{2}(1 + \sqrt{5}) = \phi$ , so $DE$ is on the line $y = \phi(x - \frac{3}{2})$ .

$E$ is at the intersection of $BC$ and $DE$ , or $y = -\frac{3}{4}x + 3$ and $y = \phi(x - \frac{3}{2})$ , which solves to $E(9 - 3\sqrt{5}, \frac{1}{4}(9\sqrt{5} - 15))$ .

The side of the pentagon is $DF = 9 - 3\sqrt{5} - \frac{3}{2} = \frac{1}{2}(15 - 6\sqrt{5})$ , so its area is $A_{\text{pentagon}} = \frac{1}{4}\sqrt{5(5 + 2\sqrt{5}}(\frac{1}{2}(15 - 6\sqrt{5}))^2 \approx \boxed{1.0786}$ .

The height of the pentagon is $\frac{1}{2}\sqrt{5 + 2 \sqrt{5}} \cdot \frac{1}{2}(15 - 6\sqrt{5}) = \frac{3}{4}\sqrt{5(5 - 2\sqrt{5})}$ and the midpoint of $DF$ is $(\frac{1}{2}(\frac{3}{2} + 9 - 3\sqrt{5}), \frac{1}{2}(0 + 0)) = (\frac{1}{4}(21 - 6\sqrt{5}), 0)$ , so $G$ is at $G(\frac{1}{4}(21 - 6\sqrt{5}), \frac{3}{4}\sqrt{5(5 - 2\sqrt{5})})$ .

By the properties of a regular pentagon, the slope of $GH$ is $-\tan 36° = -\sqrt{5 - 2\sqrt{5}}$ , and since it is through $G(\frac{1}{4}(21 - 6\sqrt{5}), \frac{3}{4}\sqrt{5(5 - 2\sqrt{5})})$ , it is on the line $y = -\sqrt{5 - 2\sqrt{5}}(x - \frac{1}{4}(21 - 6\sqrt{5})) + \frac{3}{4}\sqrt{5(5 - 2\sqrt{5})}$ .

$H$ is at the intersection of $GH$ and $DE$ , or $y = -\sqrt{5 - 2\sqrt{5}}(x - \frac{1}{4}(21 - 6\sqrt{5})) + \frac{3}{4}\sqrt{5(5 - 2\sqrt{5})}$ and $y = \phi(x - \frac{3}{2})$ , which solves to a $y$ -coordinate of $\cfrac{3\sqrt{5(5 - 2\sqrt{5})}}{1 + \sqrt{5} + 2\sqrt{5 - 2\sqrt{5}}}$ , so the area of the square is $A_{\text{square}} = \Bigg(\cfrac{3\sqrt{5(5 - 2\sqrt{5})}}{1 + \sqrt{5} + 2\sqrt{5 - 2\sqrt{5}}}\Bigg)^2 \approx \boxed{1.0803}$ .

Therefore, the square has a larger area than the pentagon.

Here is a (lengthy) solution, which avoids coordinate geometry. First we prove the

Lemma :
Figure 1 In figure 1, let $a$ , $b$ , $c$ , $x$ and $y$ be the lengths of the line segments $BC$ , $CA$ , $AB$ , $BD$ and $DC$ respectively, with $AD$ being the bisector of $A$ .
Then, the following relations hold $x=\dfrac{ac}{b+c} \text{\ \ \ \ \ and\ \ \ \ \ }y=\dfrac{ab}{b+c}$ Proof
By the angle bisector theorem , $\begin{aligned} \dfrac{x}{y}=\dfrac{c}{b} & \Rightarrow \dfrac{x}{a-x}=\dfrac{c}{b} \\ & \Rightarrow xb=c\left( a-x \right) \\ & \Rightarrow xb=ca-cx \\ & \Rightarrow xb+xc=ac \\ & \Rightarrow x=\dfrac{ac}{b+c} \\ \end{aligned}$ Similar proof for $y=\dfrac{ab}{b+c}$ .

Figure 2 Applying the lemma on $\triangle ABC$ we have $AD=\frac{AC\cdot BA}{BA\cdot BC}=\frac{4\cdot 3}{3+5}\Rightarrow AD=\frac{3}{2}$ Easily, $DC=AC-AD=4-\frac{3}{2}\Rightarrow DC=\frac{5}{2}$ .
Moreover, by the angle bisector theorem, $B{{D}^{2}}=BA\cdot BC-AD\cdot DC=3\cdot 5-\frac{3}{2}\cdot \frac{5}{2}\Rightarrow BD=\frac{3\sqrt{5}}{2}$ (Pythagoras’ theorem on $\triangle ABD$ could have been used as well.)

Now, we focus on $\triangle BDC$ .
Using the lemma we find $EC$ : $EC=\dfrac{BC\cdot DC}{BD+DC}=\dfrac{5\cdot \dfrac{5}{2}}{\dfrac{3\sqrt{5}}{2}+\dfrac{5}{2}}\Rightarrow EC=\dfrac{5}{4}\left( 3\sqrt{5}-5 \right)$ Then, the angle bisector theorem gives $\begin{aligned} & D{{E}^{2}}=BD\cdot DC-BE\cdot EC=\dfrac{3\sqrt{5}}{2}\cdot \dfrac{5}{2}-\left( 5-\dfrac{5}{4}\left( 3\sqrt{5}-5 \right) \right)\cdot \dfrac{5}{4}\left( 3\sqrt{5}-5 \right) \\ & \Rightarrow DE=\dfrac{3}{4}\sqrt{250-110\sqrt{5}} \\ \end{aligned}$ Triangles $\triangle EFC$ and $\triangle BAC$ are similar, thus,
$\dfrac{EF}{AB}=\dfrac{EC}{BC}\Rightarrow EF=\dfrac{AB\cdot EC}{BC}=\frac{3\cdot \dfrac{5}{4}\left( 3\sqrt{5}-5 \right)}{5}\Leftrightarrow EF=\dfrac{3}{4}\left( 3\sqrt{5}-5 \right)$ Hence,
$\sin \left( \angle EDF \right)=\frac{EF}{DE}=\frac{\frac{3}{4}\left( 3\sqrt{5}-5 \right)}{\frac{3}{4}\sqrt{250-110\sqrt{5}}}=\sqrt{\frac{1}{10}\left( 5+\sqrt{5} \right)}$

For the comparison of the areas of the two polygons and in order to simplify the calculations needed, we change the scale, setting the side of the pentagon to be of unit length, i.e. $DF=1$ .

Then for the area of the pentagon

${{A}_{5}}=\frac{\sqrt{25+10\sqrt{5}}}{4}DF=\frac{\sqrt{25+10\sqrt{5}}}{4}\cdot 1\Rightarrow \boxed{{{A}_{5}}\approx 1.720}$ Let $DR$ be perpendicular to one side of the pentagon, $I$ be the upper left vertex of the square and $\theta =\angle IDR$ , as seen in figure 2.
Then $DR$ is the height of the pentagon, hence $DR=\dfrac{1}{2}\sqrt{5+2\sqrt{5}}\cdot DF\Rightarrow DR=\dfrac{1}{2}\sqrt{5+2\sqrt{5}}$ .
Moreover, $\angle FDR$ is half the angle of the pentagon, hence $\angle FDR=\dfrac{3\pi }{10}$ .

For the area of the square,

$\begin{aligned} & {{A}_{4}}=I{{O}^{2}}={{\left( DI\cdot \sin \left( \angle IDF \right) \right)}^{2}}={{\left( \dfrac{DR}{\cos \theta }\cdot \sin \left( \angle EDF \right) \right)}^{2}}=\dfrac{\dfrac{5+2\sqrt{5}}{4}}{{{\cos }^{2}}\left( si{{n}^{-1}}\sqrt{\dfrac{1}{10}\left( 5+\sqrt{5} \right)}-\dfrac{3\pi }{10} \right)}\cdot \dfrac{1}{10}\left( 5+\sqrt{5} \right) \\ & \Rightarrow \boxed{{{A}_{4}}\approx 1.723} \\ \end{aligned}$

Comparing the two areas we see that the square has a greater area.