A Constrained Triangle

Choose any line segment as a base for the triangle, the outer circle should pass through its endpoints. Maximal triangle surface is obtained with the top point at maximal height. Therefore the top should be above the centerpoint of the line segment.

Choose the x-axis as the baseline of the triangle. (p, 0) and (-p, 0) as the two base points. The inner circle has radius 1, determine the top of the triangle (0, 2p^2/(p^2-1)). The centerpoint of the outer circle is (0, 2p^2/(p^2-1)-R).

The distance between the centerpoint of the outer circle and (p, 0) should be R=3, the radius of the outer circle. This yields to p^6-10p^4+13p^2=0. Surface of the triangle is A=p.2p^2/(p^2-1). Two solutions for p apart from 0 or nagative solutions. Largest surface for p^2=5-2sqrt(3). A=7.1038...

Let us consider triangles $ABC$ with inradius $r$ and circumradius $R > 2r$ (we exclude equilateral triangles by avoiding $R=2r$ ). Then standard results tell us that $R \; =\; \frac{r}{\cos A + \cos B + \cos C - 1} \; = \; \frac{r}{4\sin\frac12A \sin\frac12B \sin\frac12C}$ while the area of the triangle is $\Delta \; = \; 2R^2\sin A \sin B \sin C \; =\; 4Rr \cos\tfrac12A \cos \tfrac12B \cos\tfrac12C$ Putting $u=\tfrac12A$ , $v=\tfrac12B$ , $w=\tfrac12C$ , we need to maximize the quantity $\Delta \; = \; 4Rr \cos u \cos v \cos w$ subject to the constraints $u,v,w > 0 \hspace{1cm} u + v +w = \tfrac12\pi \hspace{1cm} \sin u \sin v \sin w = \tfrac{r}{4R}$ The method of Lagrange multipliers leads us to solve the equations $\begin{aligned} -4Rr \sin u \cos v \cos w - \lambda \cos u \sin v \sin w - \mu & = \; 0 \\ -4Rr \cos u \sin v \cos w - \lambda \sin u \cos v \sin w - \mu & = \; 0 \\ -4Rr \sin u \sin v \cos w - \lambda \sin u \sin v \cos w - \mu & = \; 0 \end{aligned}$ for some $\lambda,\mu$ . Eliminating $\mu$ leads to the equations $\sin(u-v)\big[\lambda \sin w - 4Rr\cos w\big] \; = \; \sin(v-w)\big[\lambda \sin u - 4Rr\cos u\big] \; = \; \sin(w-u)\big[\lambda \sin v - 4Rr\cos v\big] \; = \; 0$ If $\sin(u-v),\sin(v-w),\sin(w-u)$ are all nonzero, we deduce that $\tan u = \tan v = \tan w =\tfrac{\lambda}{4Rr}$ and hence, since $u,v,w$ are all acute, $u=v=w=\tfrac16\pi$ , which would imply that $R=2r$ . Since we have excluded this case, we deduce that one of the sines must be zero. Without loss of generality, we shall assume that $u=v$ , and so $\begin{aligned} \sin^2u \sin\big(\tfrac12\pi - 2u\big) \; = \; \sin u \sin v \sin w & = \; \tfrac{r}{4R} \\ (1 - \cos^2u)(2\cos^2u -1) \; = \; \sin^2u\cos 2u & = \; \tfrac{r}{4R} \\ 2\cos^4u - 3\cos^2u + 1 + \tfrac{r}{4R} & = \; 0 \\ 2\left(\cos^2u - \tfrac34\right)^2 & = \; \tfrac{R-2r}{8R} \; =\; \tfrac{d^2}{8R^2} \end{aligned}$ where $d = \sqrt{R^2 - 2Rr}$ is the distance between the incentre and circumcentre of $ABC$ . Thus $\Delta$ is extremized when $u = u_\pm$ , where $\cos^2u_\pm \; = \; \tfrac{3R \pm d}{4R} \hspace{2cm} \sin^2u_\pm \; = \; \tfrac{R \mp d}{4R}$ Thus the extremal areas of a triangle with inradius $r$ and circumradius $R$ are $\begin{aligned} \Delta_\pm & = \; 4Rr \cos^2u_\pm\cos(\tfrac12\pi-2u_\pm) \; =\; 4Rr \cos^2u_\pm \sin2u_\pm \; =\; 8Rr\cos^3u_\pm\sin u_\pm \\ & = \; 8Rr\left(\frac{3R \pm d}{4R}\right)^{\frac32} \left(\frac{R \mp d}{4R}\right)^{\frac12} \; = \; \frac{r}{2R}(3R \pm d)\sqrt{(3R\pm d)(R\mp d)} \\ & = \; \frac{r}{2R}(3R\pm d)\sqrt{3R^2 - d^2 \mp 2Rd} \; =\; r(3R \pm d)\sqrt{\frac{R + r \mp d}{2R}} \end{aligned}$ Since $\Delta^2_- - \Delta_+^2 \; =\; 4r^2d(R - 2r) \, \ge \, 0$ we deduce that the maximum triangle area is $\Delta_- \; = \; r(3R - d) \sqrt{\frac{R+r+d}{2R}}$ With $R=3$ , $r=1$ , we obtain $d=\sqrt{3}$ and $\Delta_- \; = \; \tfrac{1}{\sqrt{2}}(3\sqrt{3}-1)\sqrt{4+\sqrt{3}} \; \approx \; \boxed{7.103808951}$

The blue and brown isosceles triangles, with parallel bases, are the extreme-area triangles. The blue triangle yields the maximum area, while the brown triangle yields the minimum area. A triangle with inradius $1$ and circumradius $3$ (although not necessarily isosceles) can be found with one side parallel to the two bases of the extreme-area triangles and lying between them - the pink triangle is an example.