Only looks like the Archery aim!

Good Problem!, the key is If we fix any one vertex keeping other two vertex as variable then maximum area won't affect . Since It is independent of reference frame. So Let us Fixed Vertex $A(3,0)$ and also let parametric Points as $B(5\cos { \alpha } ,5\sin { \alpha } )\& C(7\cos { \beta } ,7\sin { \beta } )$ .

So area of Triangle ABC in determinant form is given by: $\Delta =\cfrac { 1 }{ 2 } \begin{vmatrix} 5\cos { \alpha } & 5\sin { \alpha } & 1 \\ 7\cos { \beta } & 7\sin { \beta } & 1 \\ 3 & 0 & 1 \end{vmatrix}$ Using Operation: ${ C }_{ 1 }\rightarrow { C }_{ 1 }-3{ C }_{ 3 }$ and then expanding about 3rd row we get :

$\displaystyle{\Delta =\cfrac { 1 }{ 2 } \left| 35\cos { \alpha } \sin { \beta } -35\sin { \alpha \cos { \beta } } -21\sin { \beta } +15\sin { \alpha } \right| }$

Now Remind , Maximum value of $a\sin { x } +b\cos { x }$ is nothing but $\sqrt { { a }^{ 2 }+{ b }^{ 2 } }$ So let's rearrange above expression of area in this form:

$\displaystyle{\Delta =\cfrac { 1 }{ 2 } \left| (35\cos { \alpha } -21)\sin { \beta } -(35\sin { \alpha )\cos { \beta } +15\sin { \alpha } } \right| \\ { \Delta }_{ m }=\cfrac { 1 }{ 2 } \left| \sqrt { { (35\cos { \alpha } -21) }^{ 2 }+{ (35\sin { \alpha } ) }^{ 2 } } +15\sin { \alpha } \right| \\ { \Delta }_{ m }=\cfrac { 1 }{ 2 } \left| 7\sqrt { 34-30\cos { \alpha } } +15\sin { \alpha } \right| }$

Now simple differentiating it and equate to zero $\cfrac { d({ \Delta }_{ m }) }{ d\alpha } =0$ , we find It's Maximum value as:

${ \Delta }_{ max }=31.314\quad \\ \left\lceil { \Delta }_{ max } \right\rceil =32$

Imgur

Suppose we have two of the points (B & C) of the optimum triangle. The third point A lies on a circle. If we were to move A keeping the area ABC constant, A must move along a line parallel to BC (Shown dotted). Moving to right of the dotted line gives more area, on the left we get less. Since area is maximized, the path of A must wholly lie on the left of the dotted line.

So the dotted line must be a tangent to the path of A - the circle, since the radius OA is perpendicular to it, it must be so to the base BC as well! Hence line AOm must be perpendicular to the base BC - an altitude.

For the same reason, Bn, Cp must also be altitudes that meet at O. Hence O is the Orthocenter of triangle ABC.

Taking OA as the x axis and Om = x; equate slope of CO to negative reciprocal of slope AB gives -

$\sqrt{9-x^2}\sqrt{25-x^2} = -x(7-x)$

Solved this numerically to get x = 1.47 and the area 31.31

the question proposed is incomplete!! add the point that one can use a calc to find the roots of a cubic polynomial. well a pure geometric solution-> show that triangle ABC will be such that 0 (centre) is the orthocentre of ABC.

After pages and pages of calculations, I found that if the radii have values $r_1$ , $r_2$ and $r_3$ , and if we name $s=r_1^2+r_2^2+r_3^2$ and $p=r_1r_2r_3$ , then the square of the circumradius $x=R^2$ is a solution of $16x^3-8sx^2+s^2x-p=0$ .

More over, as $c_i^2+r_i^2=4R^2$ for each side $c_i$ opposite of point lying on circle with radius $r_i$ and as area $=\frac{c_1c_2c_3}{4R}$ , we have $A=\frac{\sqrt{3}}{12}\sqrt{(12x-s)^2+2s^2-6(r_1^4+r_2^4+r_3^4)}$ .