A Square Problem!

Let the side length of square $ABCD$ be $a$ . Then $\tan \theta = \dfrac {64}a$ and

$\begin{aligned} \tan (90^\circ - 2\theta) & = \cot 2\theta = \frac 1 {\tan 2\theta} = \frac {36}a \\ \frac {1-\tan^2 \theta}{2 \tan \theta} & = \frac {36}a \\ \frac {1-\left(\frac {64}a\right)^2}{2 \times \frac {64}a} & = \frac {36}a \\ 1 - \frac {64^2}{a^2} & = \frac {2 \times 64\times 36}{a^2} \\ \implies a^2 & = 8704 \end{aligned}$

The area of $\triangle AEF$ :

$\begin{aligned} [AEF] & = a^2 - \left(\frac {64a}2 + \frac {36a}2 + \frac {(a-64)(a-36)}2 \right) \\ & = a^2 - \left(\frac {a^2}2 + 1152\right) \\ & = \frac {a^2}2 - 1152 = \frac {8704}2 - 1152 = \boxed{3200} \end{aligned}$

Let the coordinates of $A, B, C, D$ be $(0,0), (0, a), (a, a), (a, 0)$ respectively, where $a$ is the side length of the square. Then $\tan \theta =\dfrac{64}{a}, \tan 2\theta=\dfrac{a}{36}=\dfrac{2\tan \theta}{1-\tan^2 \theta}$ . Solving we get $a=\sqrt {8704}$ . So the coordinates of $E$ and $F$ are $(36,\sqrt {8704})$ and $(\sqrt {8704},64)$ respectively. Hence the area of $\triangle {AEF}$ is $\dfrac{1}{2}|36\times 64+\sqrt {8704}\times (-\sqrt{8704})|=\boxed {3200}$

In square $ABCD$ let $AB = x \implies$

$AE = \sqrt{x^2 + 36^2}$ , $AF = \sqrt{x^2 + 64^2}$ and $\sin(2\theta) = \dfrac{x}{\sqrt{x^2 + 36^2}} = 2\sin(\theta)\cos(\theta)$ , where $\sin(\theta) = \dfrac{64}{\sqrt{x^2 + 64^2}}$

and $\cos(\theta) = \dfrac{x}{\sqrt{x^2 + 64^2}}$

$\implies \sin(2\theta) = \dfrac{x}{\sqrt{x^2 = 36^2}} = 2\sin(\theta)\cos(\theta) = \dfrac{128x}{x^2 + 64^2}$

$\implies 128\sqrt{x^2 + 36^2} = x^2 + 64^2 \implies x^4 - 2 * 64^2 x^2 - 64^2 * 8^2 * 17 = 0$

$\implies x^2 = 4096 \pm 4608$

Choosing the positive real root $\implies x = \sqrt{8704} = 16\sqrt{34} \implies$

$EF = \sqrt{(16\sqrt{34} - 36)^2 + (16\sqrt{34} - 64)^2} = \sqrt{22800 - 3200\sqrt{34}} =$

$20\sqrt{57 - 8\sqrt{34}}$ .

$\vec{U} = (\sqrt{8704} - 36)\vec{i} + (64 - \sqrt{8704})\vec{j} + 0\vec{k}$

$\vec{V} = \sqrt{8704} \:\ \vec{i} + 64\vec{j} + 0\vec{k}$

$\implies |\vec{U} X \vec{V}| = |64\sqrt{8704} - 2304 - 64\sqrt{8704} + 8704| = 6400$

and $\vec{U} = 4\sqrt{(4\sqrt{34} - 9)^2 + (16 - 4\sqrt{34})^2} = 20\sqrt{57 - 8\sqrt{34}}$

$\implies d = \dfrac{320}{\sqrt{57 - 8\sqrt{34}}}$

$\implies A_{\triangle{AEF}} = \dfrac{1}{2}(EF)(d) = \dfrac{1}{2}(20\sqrt{57 - 8\sqrt{34}})(\dfrac{320}{\sqrt{57 - 8\sqrt{34}}}) = \boxed{3200}$

Note: I wanted to find the $\perp$ distance $d$ . Of course you could have used:

$A_{\triangle{AEF}} = A_{\square} - (A_{\triangle{ABE}} + A_{\triangle{ECF}} + A_{\triangle{ADF}}) =$

$8704 - \dfrac{1}{2}(576\sqrt{34} + 8704 - 1600\sqrt{34} + 2304 + 1024\sqrt{34}) =$

$8704 - 5504 = 3200$