Maximizing with an Orthic Condition

It is well known that the medial triangle of $ABC$ is similar to the original triangle $ABC$ . Since $ABC$ 's medial and orthic triangles have the same circumcircles (the nine-point circle), the orthic triangle is congruent to the medial triangle if and only if the orthic triangle of $ABC$ is similar to $ABC$ .

We have two cases to consider: when $ABC$ is acute, and when $ABC$ is obtuse (if $ABC$ is right, the orthic triangle is degenerate).

When $ABC$ is acute, an angle chase with all the cyclic quadrilaterals formed by the altitudes shows that the set of angles of the orthic triangle is $\{\pi - 2A, \pi-2B, \pi-2C\}$ , where $A,B,C$ are the angles of the corresponding vertices of the original triangle.

WLOG setting $A\ge B\ge C$ , we get the system of equations $A+2C=\pi$ $3B=\pi$ $2A+C=\pi$ which only has the solution when $ABC$ is an equilateral triangle.

In this case, by drawing the circumcircle and dividing $ABC$ into three congruent triangles by drawing the radii to each vertex, we find by the Law of Cosines that $a^2=b^2=c^2 = R^2+R^2-2R^2\cos\frac{2\pi}{3} = 3R^2.$

In this case, $\dfrac{a^2+b^2+c^2}{R^2}=9.$

When $ABC$ is obtuse, a similar angle chase gives that the angle set of the orthic triangle is $\{2C-\pi, 2B, 2A-2C+\pi\}$ , when $C$ has the obtuse angle.

Doing some casework on order and solving the resulting equations, the only solution reveals itself to be $A=\frac{\pi}{7}$ $B=\frac{2\pi}{7}$ $C=\frac{4\pi}{7}.$

To compute the desired ratio for this heptagonal triangle, we invoke three well known theorems (well, two are well known).

Law of Sines: $\frac{a}{\sin A} = \frac{b}{\sin B}=\frac{c}{\sin C} = 2R$ .

Theorem 1: If $A,B,C$ are angles of a triangle, then $\cos^2A+\cos^2B+\cos^2C+2\cos A \cos B \cos C=1.$

Theorem 2: We have $cos\frac{\pi}{7}\cos\frac{2\pi}{7}\cos\frac{4\pi}{7}=-\frac{1}{8}$ . Proof: Write $P=cos\frac{\pi}{7}\cos\frac{2\pi}{7}\cos\frac{4\pi}{7}$ $P\sin\frac{\pi}{7}\sin\frac{2\pi}{7}\sin\frac{4\pi}{7}=cos\frac{\pi}{7}\cos\frac{2\pi}{7}\cos\frac{4\pi}{7}\sin\frac{\pi}{7}\sin\frac{2\pi}{7}\sin\frac{4\pi}{7}$ $P\sin\frac{\pi}{7}\sin\frac{2\pi}{7}\sin\frac{4\pi}{7} = \dfrac{1}{8}\sin\frac{2\pi}{7}\sin\frac{4\pi}{7}\sin\frac{8\pi}{7}$ $P=\dfrac{1}{8}\cdot \dfrac{\sin\frac{8\pi}{7}}{\sin\frac{\pi}{7}} = -\dfrac{1}{8}.$

Now, we have $\dfrac{a^2+b^2+c^2}{R^2} = 4(\sin^2A+\sin^2B+\sin^2C) =$ $=4(3-(\cos^2A+\cos^2B+\cos^2C))=$ $=4(2+2\cos A\cos B\cos C)=$ $=8(1+\cos\frac{\pi}{7}\cos\frac{2\pi}{7}\cos\frac{4\pi}{7})= 7.$

Thus the minimum possible value is $\boxed{7}$ .