Making scalene to equilateral!

I agree, maybe put a link to the theory as a hint?

One of the principles of brilliant.org is to cultivate a culture of learning, and one way to do this is to read about a new theorem and investigate it more. For example, a reader who has never been taught trigonometry before may see the law of cosines for the first time, and use this as a learning opportunity to research it more. In the same way, if you see Napoleon’s Triangle Theorem for the first time, then you have the opportunity to click on its link and learn more about it, which I think is fantastic.

For a more rigorous solution without using Napoleon’s Triangle Theorem, one can take a trigonometric approach. If $M$ is the midpoint of $AB$ , then since $R$ is the center of equilateral triangle $\triangle ABF$ , $\triangle RMB$ is a $30°-60°-90°$ triangle, and $RB$ = $\frac{AB\sqrt{3}}{3}$ . Similarly, $BP = \frac{BC\sqrt{3}}{3}$ . Using the law of cosines on $\triangle BRP$ , $RP = \sqrt{(\frac{AB\sqrt{3}}{3})^2 + (\frac{BC\sqrt{3}}{3})^2 - 2(\frac{AB\sqrt{3}}{3})(\frac{BC\sqrt{3}}{3})\cos(\angle ABC + 60°)}$ . Using the cosine addition formula, Heron’s formula, and a lot of algebraic manipulation, this can be simplified to $RP = \sqrt{\frac{AB^2 + AC^2 + BC^2}{6} + \frac{2K\sqrt{3}}{3}}$ , where $K$ is the area of $\triangle ABC$ . Since $RP$ is symmetric according to sides $AB$ , $AC$ , and $BC$ , similar reasoning gives $RP = RQ = QP$ , making $\triangle PQR$ an equilateral triangle. In this case, $AB = 3$ , $AC = 6$ , and $BC = 5$ , so the area of $\triangle ABC = \sqrt{\frac{3 + 5 + 6}{2}(\frac{3 + 5 + 6}{2} - 3)(\frac{3 + 5 + 6}{2} - 5)(\frac{3 + 5 + 6}{2} - 6)} = 2\sqrt{14}$ (using Heron's formula), and each side $s$ of equilateral $\triangle PQR$ is $s = \sqrt{\frac{3^2 + 6^2 + 5^2}{6} + \frac{2(2\sqrt{14})\sqrt{3}}{3}} = \sqrt{\frac{35 + 4\sqrt{42}}{3}}$ , and the area of equilateral $\triangle PQR = \frac{s^2\sqrt{3}}{4} = \frac{\sqrt{\frac{35 + 4\sqrt{42}}{3}}^2\sqrt{3}}{4} = \frac{35\sqrt{3}}{12} + \sqrt{14} \approx \boxed{8.793472242}$ .

Personally, I thought a solution using Napoleon’s Triangle Theorem was more elegant (not to mention briefer) than one that doesn’t use it, which is why I chose to post it in my solution. Hopefully someone more clever than me can post a better solution without using Napoleon’s Triangle Theorem.

That is true. Since they are centers of equilateral triangles you could use the orthocenter, centroid, circumcenter, or incenter as they are all the same point.