It's All Areas

Let $m(\angle{ACB}) = \dfrac{(n - 2)180}{n}$ be one of the interior angles of the larger $n-gon$ .

$\sin(\dfrac{(n - 2)180}{n}) = \sin(180 - \dfrac{360}{n}) = \sin(\dfrac{360}{n}) = \sin(\dfrac{2\pi}{n})$ .

Using the law of sines $\implies \dfrac{c}{\sin(\dfrac{2\pi}{n})} = \dfrac{a}{\sin(\alpha)} \implies$ $\sin(\alpha) = \dfrac{a\sin(\dfrac{2\pi}{n})}{c} \implies h^{*} = \dfrac{ab\sin(\dfrac{2\pi}{n})}{c} \implies$ $\sum_{j = 1}^{n} A_{j} = n\dfrac{1}{2}(c)(\dfrac{ab\sin(\dfrac{2\pi}{n})}{c}) = \dfrac{n}{2} ab\sin(\dfrac{2\pi}{n})$ .

Using the inequality $\cos(x) < \dfrac{\sin(x)}{x} < 1 \implies \pi\cos(\dfrac{\pi}{n}) < \dfrac{n}{2}\sin(\dfrac{2\pi}{n}) < \pi \implies \lim_{n \rightarrow \infty} \sum_{j = 1}^{n} A_{j} = \boxed{\pi ab}$ $\implies \sum_{j = 1}^{n} A_{j}$ converges to the area of an ellipse.

$\lim_{n \rightarrow \infty} \sum_{j = 1}^{n} A_{j} = \pi ab = \pi(a^2 + 2ab - 12b^2) \implies a^2 + ab - 12b^2 = 0 \implies (a - 3b)(a + 4b) = 0$ since $a,b > 0 \implies \dfrac{a}{b} = \boxed{3}$ .

In the triangle $ABC$ , $\angle C=\frac{n-2}{n} \pi = \pi-\frac{2\pi}{n}$ .

The area of one of the red triangles is therefore $A_j=\frac12 ab \sin C=\frac12 ab \sin \left(\pi-\frac{2\pi}{n} \right)=\frac12 ab \sin \frac{2\pi}{n}$

There are $n$ of these triangles; so the sum of their areas is $\frac{n}{2} ab \sin \frac{2\pi}{n}$

Since $\sin \theta \approx \theta$ for small $\theta$ , in the limit this becomes simply $\pi ab$

(which is interesting - I wonder how this is related to the area of an ellipse?)

Equating this to the given form, and letting $x=\frac{a}{b}$ , $\begin{aligned} \pi ab &= \pi \left(a^2+2ab-12b^2 \right) \\ x &=x^2+2x-12 \\ x^2+x-12 &=0 \end{aligned}$

whose unique positive root is $x=\boxed{3}$ .