Ellipses are trickier than Circles

Just to get the conversation started, here is an outline of my approach.

First note that in a circle of radius $r$ , the largest triangle with one vertex at the center of the circle that can be placed inside the circle is right-angled, and thus has area $\dfrac{r^{2}}{2}.$ The ratio of the area of this triangle and the area of the circle is then $\dfrac{1}{2\pi}.$

Now as an ellipse is just a linear transformation of a circle, this ratio will be preserved for the similar scenario with the ellipse, (although the triangle may no longer be right-angled). Since in general the area of an ellipse is $\pi ab,$ where $a$ and $b$ are the lengths of the semi-major and semi-minor axes, respectively, we see that our ellipse has an area of $8\pi,$ and thus the largest possible triangle with one vertex at the center $C$ of the ellipse will have an area of $4.$ We now have an upper bound for the answer to this question.

But now we must deal with the condition that $|AB| = 3.$ Guessing at a symmetric solution, I looked at the triangles formed when $AB$ was either horizontal or vertical. The vertical case proved to yield the largest triangle, so I will look at that calculation only. With the $y$ coordinates for $A$ and $B$ being $\frac{3}{2}$ and $-\frac{3}{2}$ respectively, we find that the corresponding $x$ coordinates are such that

$\dfrac{x^{2}}{16} + \dfrac{\frac{9}{4}}{4} = 1 \Longrightarrow x^{2} = 7 \Longrightarrow x = \pm \sqrt{7},$

where it was assumed that the major axis lay along the $x$ -axis and the ellipse was centered at the origin. The area of $\Delta ABC$ is then $\dfrac{3\sqrt{7}}{2} = 3.97$ to 2 decimal places. Since the answer field required an integer answer I entered $4$ as my answer, but I have yet to determine if this area value can be achieved under the given restriction of $|AB| = 3.$ My next attempt was going to apply vectors, i.e., $A = \frac{1}{2}|a||b|\sin(\theta)$ and then do some calculus, but an initial attempt proved to be unwieldy.

To be continued .....

We can assume from the problem that the coordinates of the vertices of the triangle are (4 cos alpha,2 sin alpha), (4 cos beta,2 sin beta) and (0,0).Now the area of the triangle with these vertices is coming as 4 cos(alpha - beta), whose maximum value is 4, irrespective of the magnitude of side AB

I'm building on Brian's solution, trying to fill in the small gap at the end.

Let's say the circle is given by $u^2+v^2=4$ , and we consider the linear transformation $T(u,v)=(2u,v)$ over to the ellipse, which doubles areas. We need to show that there exists a right triangle $A'B'O$ , with two vertices $A'$ and $B'$ on the circle and the third vertex at the origin, such that the side $AB$ of the corresponding triangle in the ellipse has length 3. Since the length of $A'B'$ is $\sqrt{8}$ , we need a scaling factor of $\frac{3}{\sqrt{8}}$ under transformation $T$ .

Now the scaling factor for horizontal line segments in the uv-plane is 2, and it is 1 for vertical line segments. It stands to reason, "by continuity", that all scaling factors between 1 and 2 are attained, including the one we need, $\frac{3}{\sqrt{8}}$ .

We can make this argument rigorous: If a line segment in the uv-plane makes an angle $\theta$ with the u-axis, it is not hard to see that the scaling factor is $\sqrt{3\cos^2\theta+1}$ , a continuous function whose range is $[1,2]$ , proving our claim.