Inspired by Rama Devi

Let $\vec{a} = \vec{OA}$ , $\vec{b} = \vec{OB}$ , $\vec{c} = \vec{OC}$ . Then $|\vec{a}| = 3$ , $|\vec{b}| = 5$ , $|\vec{c}| = 7$ , and

$\begin{aligned} AB^2 + AC^2 + BC^2 &= |\vec{a} - \vec{b}|^2 + |\vec{a} - \vec{c}|^2 + |\vec{b} - \vec{c}|^2 \\ &= 2|\vec{a}|^2 + 2|\vec{b}|^2 + 2|\vec{c}|^2 - 2 \vec{a} \cdot \vec{b} - 2 \vec{a} \cdot \vec{c} - 2 \vec{b} \cdot \vec{c}. \end{aligned}$

Also, $|\vec{a} + \vec{b} + \vec{c}|^2 \ge 0$ , which gives $|\vec{a}|^2 + |\vec{b}|^2 + |\vec{c}|^2 + 2 \vec{a} \cdot \vec{b} + 2 \vec{a} \cdot \vec{c} + 2 \vec{b} \cdot \vec{c} \ge 0.$ Then $AB^2 + AC^2 + BC^2 \le 3|\vec{a}|^2 + 3|\vec{b}|^2 + 3|\vec{c}|^2 = 249$ . Thus, $AB^2 + AC^2 + BC^2$ is maximized when $\vec{a} + \vec{b} + \vec{c} = \vec{0}$ .

Then $\vec{a} + \vec{b} = -\vec{c}$ , so $|\vec{a} + \vec{b}|^2 = |\vec{c}|^2$ , and $\vec{a} \cdot \vec{b} = \frac{49 - 9 - 25}{2} = \frac{15}{2}.$ Then $\cos \angle AOB = \frac{\vec{a} \cdot \vec{b}}{|\vec{a}||\vec{b}|} = \frac{15/2}{3 \cdot 5} = \frac{1}{2}.$ This means $\angle AOB = 60^\circ$ , and $\sin \angle AOB = \frac{\sqrt{3}}{2}$ .

There is the question of whether such a triangle exists, i.e. if $\vec{a} + \vec{b} + \vec{c} = \vec{0}$ can occur. This is left as an exercise for the reader. (Note that this condition implies the centroid of triangle $ABC$ is $O$ .)

This solution was once correct, when the problem was misworded.

Here is how to get the right answer.

Notice how it is never given which point is on which circle, so the answer must be the same in all cases. Hence $\sin \angle AOB=\sin \angle BOC=\sin\angle COA,$ where $\angle AOB+\angle BOC+\angle COA=2\pi$ . It is not that hard to conclude this only happens when all angles are equal to $120^\circ$ which gives an answer of $\dfrac{\sqrt{3}}{2}$ . Thanks Calvin :)