Concentric circles and equilateral triangle

Let $A,B,C$ be the vertices of the triangle on the circles of radii $8,15,17$ respectively. Since we can reflect the triangle if necessary, we can assume that $A,B,C$ occur on the triangle in anticlockwise order. We can then assume that $A$ is represented by the complex number $8$ and that $B$ is represented by the complex number $15e^{i\theta}$ for some $-\pi < \theta \le \pi$ . As the third vertex of an equilateral triangle, occurring in anticlockwise order, $C$ will then be represented by the complex number $8 - \omega^2(15e^{i\theta} - 8) \; = \; -8\omega - 15\omega^2e^{i\theta} \; = \; -\omega(8 + 15\omega e^{i\theta})$ Since $C$ must lie on the circle of radius $17$ , we deduce that $|8 + 15\omega e^{i\theta}| = 17$ , which implies that $\cos(\theta + \tfrac23\pi) = 0$ . Thus we deduce that $\theta = \tfrac56\pi\,,\,-\tfrac16\pi$ .

The side of the equilateral triangle is $h$ , where $h^2 = |15e^{i\theta}-8|^2 = 289 - 240\cos\theta = 289 \pm120\sqrt{3}$ , and hence the area of the triangle is $A \; = \; \tfrac14h^2\sqrt{3} \; = \; \tfrac14\sqrt{3}\big(289 \pm 120\sqrt{3}\big) \; = \; \frac{289\sqrt{3}}{4} \pm 90$ Thus there are two types of equilateral triangle possible, and the area of the larger one is $90 + \frac{289\sqrt{3}}{4}$ , making the answer $90 + 289 = \boxed{379}$ .

Notice that the side length of this largest equilateral triangle is equal to the side length of equilateral triangle $ABC$ shown below

where $O$ is the centre of the concentric circles. Now rotate the triangle $60^{\circ }$ anticlockwise about $A$ to form the points $B'$ and $O'$ .

Since it is evident that $\angle O'AO=60^{\circ }$ and $AO'=AO=17$ , we deduce that $\triangle AO'O$ is equilateral, and thus $O'O=17$ . Notice that $8^2+15^2=17^2$ , so we can now apply the converse of Pythagoras's Theorem to $\triangle O'BO$ to deduce that $\angle O'BO=90^{\circ }$ . Then $\angle AO'B +\angle AOB =360-60-90=210^{\circ }$ , meaning that $\angle BOC=360-210=150^{\circ }$ .

So we can apply the Cosine rule to $\triangle BOC$ to obtain $BC=\sqrt{8^2+15^2-2\times 8\times 15\times \textrm{cos}(150)}=\sqrt{120\sqrt{3}+289}$ and thus the area of the triangle is $(120\sqrt{3}+289)\times \dfrac{\sqrt{3}}{4}=90+\dfrac{289\sqrt{3}}{4}\textrm{ cm}^{2}$ So the answer is $90+289=\boxed{379}$ .