Triangle rotating in triangle, supposedly hard

In fact, Guy Fox's solution is not a solution of the specified problem at all; but, it is a solution to a closely related problem. The problem statement specifies a Riemann sum of a polygon that is slightly larger than the area enclosed by the caustic curve swept by all the lines that could have swept the interior of the triangle. That problem also would have been a worthy problem. Guy Fox's solution is a worthy answer to that other problem.

Either this problem should be deleted as the answer is wrong or the problem statement should be editted to add a clarification that the Riemann sum is not intended and that the caustic curve result is the correct answer. This clarification must go into the problem statement itself. Adding the clarification in a comment is not sufficient.

Now, a solution to the specified problem. I also have Wolfram Mathematica 11.3.

The lines that actually enclose the polygon. But the corners of the polygon present an additional problem and will be handled separately.

The intersections of the green lines with themselves:

$g=\text{Table}\left[ \\ \ \ \text{RegionIntersection}\left[\text{Line}\left[\left( \begin{array}{cc} t & 0 \\ 120-\frac{t}{2} & 120 \sqrt{3}-\sqrt{3} \left(120-\frac{t}{2}\right) \\ \end{array} \right)\right],\text{Line}\left[\left( \begin{array}{cc} t+5 & 0 \\ 120-\frac{t+5}{2} & 120 \sqrt{3}-\sqrt{3} \left(120-\frac{t+5}{2}\right) \\ \end{array} \right)\right]\right], \\ \{t,40,75,5\}\right] \Longrightarrow \\ \left\{\text{Point}\left[\left( \begin{array}{cc} \frac{125}{2} & \frac{15 \sqrt{3}}{2} \\ \end{array} \right)\right],\text{Point}\left[\left( \begin{array}{cc} \frac{535}{8} & \frac{75 \sqrt{3}}{8} \\ \end{array} \right)\right],\text{Point}\left[\left( \begin{array}{cc} \frac{565}{8} & \frac{275}{8 \sqrt{3}} \\ \end{array} \right)\right],\text{Point}\left[\left( \begin{array}{cc} \frac{295}{4} & \frac{55 \sqrt{3}}{4} \\ \end{array} \right)\right],\text{Point}\left[\left( \begin{array}{cc} \frac{305}{4} & \frac{65 \sqrt{3}}{4} \\ \end{array} \right)\right],\text{Point}\left[\left( \begin{array}{cc} \frac{625}{8} & \frac{455}{8 \sqrt{3}} \\ \end{array} \right)\right],\text{Point}\left[\left( \begin{array}{cc} \frac{635}{8} & \frac{175 \sqrt{3}}{8} \\ \end{array} \right)\right],\text{Point}\left[\left( \begin{array}{cc} 80 & 25 \sqrt{3} \\ \end{array} \right)\right]\right\}$

The intersections of the blue lines with themselves.

$b=\text{Table}\left[ \\ \ \ \text{RegionIntersection}\left[\text{Line}\left[\left( \begin{array}{cc} 120-\frac{t}{2} & 120 \sqrt{3}-\sqrt{3} \left(120-\frac{t}{2}\right) \\ 60-\frac{t}{2} & \sqrt{3} \left(60-\frac{t}{2}\right) \\ \end{array} \right)\right],\text{Line}\left[\left( \begin{array}{cc} 120-\frac{t+5}{2} & 120 \sqrt{3}-\sqrt{3} \left(120-\frac{t+5}{2}\right) \\ 60-\frac{t+5}{2} & \sqrt{3} \left(60-\frac{t+5}{2}\right) \\ \end{array} \right)\right]\right], \\ \{t,40,75,5\}\right]$

$\left\{\text{Point}\left[\left( \begin{array}{cc} \frac{155}{2} & \frac{55 \sqrt{3}}{2} \\ \end{array} \right)\right],\text{Point}\left[\left( \begin{array}{cc} \frac{145}{2} & \frac{115 \sqrt{3}}{4} \\ \end{array} \right)\right],\text{Point}\left[\left( \begin{array}{cc} \frac{135}{2} & \frac{355}{4 \sqrt{3}} \\ \end{array} \right)\right],\text{Point}\left[\left( \begin{array}{cc} \frac{125}{2} & 30 \sqrt{3} \\ \end{array} \right)\right],\text{Point}\left[\left( \begin{array}{cc} \frac{115}{2} & 30 \sqrt{3} \\ \end{array} \right)\right],\text{Point}\left[\left( \begin{array}{cc} \frac{105}{2} & \frac{355}{4 \sqrt{3}} \\ \end{array} \right)\right],\text{Point}\left[\left( \begin{array}{cc} \frac{95}{2} & \frac{115 \sqrt{3}}{4} \\ \end{array} \right)\right],\text{Point}\left[\left( \begin{array}{cc} \frac{85}{2} & \frac{55 \sqrt{3}}{2} \\ \end{array} \right)\right]\right\}$

The intersections of the red lines with themselves.

$r=\text{Table}\left[\text{RegionIntersection}\left[\text{Line}\left[\left( \begin{array}{cc} 60-\frac{t}{2} & \sqrt{3} \left(60-\frac{t}{2}\right) \\ t & 0 \\ \end{array} \right)\right],\text{Line}\left[\left( \begin{array}{cc} 60-\frac{t+5}{2} & \sqrt{3} \left(60-\frac{t+5}{2}\right) \\ t+5 & 0 \\ \end{array} \right)\right]\right],\{t,40,75,5\}\right]$

$\left\{\text{Point}\left[\left( \begin{array}{cc} 40 & 25 \sqrt{3} \\ \end{array} \right)\right],\text{Point}\left[\left( \begin{array}{cc} \frac{325}{8} & \frac{175 \sqrt{3}}{8} \\ \end{array} \right)\right],\text{Point}\left[\left( \begin{array}{cc} \frac{335}{8} & \frac{455}{8 \sqrt{3}} \\ \end{array} \right)\right],\text{Point}\left[\left( \begin{array}{cc} \frac{175}{4} & \frac{65 \sqrt{3}}{4} \\ \end{array} \right)\right],\text{Point}\left[\left( \begin{array}{cc} \frac{185}{4} & \frac{55 \sqrt{3}}{4} \\ \end{array} \right)\right],\text{Point}\left[\left( \begin{array}{cc} \frac{395}{8} & \frac{275}{8 \sqrt{3}} \\ \end{array} \right)\right],\text{Point}\left[\left( \begin{array}{cc} \frac{425}{8} & \frac{75 \sqrt{3}}{8} \\ \end{array} \right)\right],\text{Point}\left[\left( \begin{array}{cc} \frac{115}{2} & \frac{15 \sqrt{3}}{2} \\ \end{array} \right)\right]\right\}$

The three corner points as they involve lines from two different colored line sets.

$\text{p1}=\text{RegionIntersection}\left[\text{Line}\left[\left( \begin{array}{cc} t & 0 \\ 120-\frac{t}{2} & 120 \sqrt{3}-\sqrt{3} \left(120-\frac{t}{2}\right) \\ \end{array} \right)\right]\text{/.}\, t\to 40,\text{Line}\left[\left( \begin{array}{cc} 60-\frac{t}{2} & \sqrt{3} \left(60-\frac{t}{2}\right) \\ t & 0 \\ \end{array} \right)\right]\text{/.}\, t\to 80\right] \Longrightarrow \text{Point}\left[\left( \begin{array}{cc} 60 & \frac{20}{\sqrt{3}} \\ \end{array} \right)\right]$

$\text{p2}=\text{RegionIntersection}\left[\text{Line}\left[\left( \begin{array}{cc} 120-\frac{t}{2} & 120 \sqrt{3}-\sqrt{3} \left(120-\frac{t}{2}\right) \\ 60-\frac{t}{2} & \sqrt{3} \left(60-\frac{t}{2}\right) \\ \end{array} \right)\right]\text{/.}\, t\to 40,\text{Line}\left[\left( \begin{array}{cc} t & 0 \\ 120-\frac{t}{2} & 120 \sqrt{3}-\sqrt{3} \left(120-\frac{t}{2}\right) \\ \end{array} \right)\right]\text{/.}\, t\to 80\right] \Longrightarrow \text{Point}\left[\left( \begin{array}{cc} 80 & \frac{80}{\sqrt{3}} \\ \end{array} \right)\right]$

$\text{p3}=\text{RegionIntersection}\left[\text{Line}\left[\left( \begin{array}{cc} 60-\frac{t}{2} & \sqrt{3} \left(60-\frac{t}{2}\right) \\ t & 0 \\ \end{array} \right)\right]\text{/.}\, t\to 40,\text{Line}\left[\left( \begin{array}{cc} 120-\frac{t}{2} & 120 \sqrt{3}-\sqrt{3} \left(120-\frac{t}{2}\right) \\ 60-\frac{t}{2} & \sqrt{3} \left(60-\frac{t}{2}\right) \\ \end{array} \right)\right]\text{/.}\, t\to 80\right] \Longrightarrow \text{Point}\left[\left( \begin{array}{cc} 40 & \frac{80}{\sqrt{3}} \\ \end{array} \right)\right]$

Then, assemble the corner points of the polygon: $p=\text{Flatten}[\{\{\text{p1}\},g,\{\text{p2}\},b,\{\text{p3}\},r\},1];$ .

Then, assemble the polygon: $\text{polygon}=\text{Polygon}[\text{Flatten}[p\text{/.}\, \text{Point}\to \text{List},2]];$ .

Then, get the Riemann sum area (the area of the polygon): $\text{area}=\text{Evaluate}[\text{Area}[\text{polygon}]] \Longrightarrow \frac{2675 \sqrt{3}}{4}$ .

Then, convert to requested form: $\text{Solve}\left[\text{area}=\frac{u}{\sqrt{3}},u\right] \Longrightarrow \left\{\left\{u\to \frac{8025}{4}\right\}\right\}$ .

The answer to the specified problem is $2006.25$ .