Expected value of triangle area

Let's position the circle so that its centre is at the origin. Let's assume that one of the points is at $C = (1, 0)$ , since we can always rotate the triangle to make this true.

Let the other two points be represented by $A = (\cos(a), \sin(a)), B = (\cos(b), \sin(b))$ where $0 \le a,b < 2\pi$ . Using the Shoelace Formula, the area of $\triangle ABC$ is: $f(a,b) = \dfrac{\left|\sin(a) + \sin(b)\cos(a) - \sin(a)\cos(b) - \sin(b)\right|}{2} = \dfrac{\left|\sin(a) + \sin(b-a) - \sin(b)\right|}{2}$ We can use some trigonometric manipulation to show that $\sin(a) + \sin(b-a) - \sin(b) = 4\sin\left(\dfrac{b-a}{2}\right)\sin\left(\dfrac{a}{2}\right)\sin\left(\dfrac{b}{2}\right)$ . Hence $\sin(a) + \sin(b-a) - \sin(b) \ge 0$ when $b \ge a$ and vice versa.

To calculate the expected value, I am going to use the formula $v = \left(\dfrac{1}{2\pi}\right)^2\displaystyle\int_{0}^{2\pi} \displaystyle\int_{0}^{2\pi} f(a,b) d b \text{ }da$ . I will calculate the inside integral first: $\begin{array}{lll} F(a) & = & \displaystyle\int_{0}^{2\pi} f(a,b) db \\ & = & \displaystyle\int_{0}^{a} \dfrac{\sin(b) + \sin(a-b) - \sin(a)}{2} db + \displaystyle\int_{a}^{2\pi} \dfrac{\sin(a) + \sin(b-a) - \sin(b)}{2} db \\ & = & \dfrac{-\cos(b) + \cos(b-a) - b\sin(a)}{2} \biggr\rvert_{0}^{a} + \dfrac{b\sin(a) -\cos(b-a) + \cos(b)}{2} \biggr\rvert_{a}^{2\pi} \\ & = & \dfrac{-\cos(a) + 1 - a\sin(a)}{2} - \left(\dfrac{-1 + \cos(a)}{2}\right) + \dfrac{2\pi\sin(a) -\cos(a) + 1}{2} - \left(\dfrac{a\sin(a) -1 + \cos(a)}{2}\right) \\ & = & 2 - 2\cos(a) - a\sin(a) + \pi \sin(a) \end{array}$

Now plug in this formula to the double integral:

$\begin{array}{lll} v & = & \dfrac{1}{4\pi^2} \displaystyle\int_{0}^{2\pi} \displaystyle\int_{0}^{2\pi} f(a,b) d b \text{ }da \\ & = & \dfrac{1}{4\pi^2} \displaystyle\int_{0}^{2\pi} \left(2 - 2\cos(a) - a\sin(a) + \pi \sin(a)\right) da \\ & = & \dfrac{1}{4\pi^2} \left(2a - 2\sin(a) + a\cos(a) - \sin(a) - \pi \cos(a)\right) \biggr\rvert_{0}^{2\pi} \\ & = & \dfrac{1}{4\pi^2}\left(6\pi \right) \\ & = & \boxed{\dfrac{3}{2\pi}} \end{array}$

As the co-primes had to be small, like others below i got it right within the 3 tries allowed