Don't forget the 3rd law

Let us first consider the force on the red ball. It feels a repulsive force from B and a attractive force towards G, both of magnitude $\frac{1}{L} N$ . The vertical components of these two forces are equal and opposite while the horizontal component of each is equal to $\frac{\cos 60 ^ \circ}{L}$ = $\frac{1}{2L} N$ . Since there are two horizontal forces of this magnitude, the net force felt by B has magnitude $\frac{1}{L} N$ and is horizontal to the left.

Note: Since the applied force is always aligned with the direction of motion, the force directly contributes to the objects speed at all times. Also notice that there is not centripetal force acting on either of the objects. This is because centripetal force describes the motion of an object rotating with constant speed and centripetal force is required to keep the object within the rotational path. This does not apply in this case because there is a constant external force acting in the tangential direction and the speed is not constant, therefore no centripetal force is required.

A similar analysis on the other 2 points yields forces of the same magnitude. The directions are forcing each point to go counterclockwise and as each point moves, the direction of the forces will also move causing the system to spin, and accelerate, in a counterclockwise motion. Since the force on each is $\frac{1}{L} N$ , we can divide by the mass of $0.001\; kg$ to get acceleration of $\frac{1000}{L} \frac{m}{s^2}$ , which means, if starting from rest, the speed after 5 seconds is $\frac{1000}{L} \frac{m}{s^2} * 5 s = \frac{5000}{L} \frac{m}{s}$ .

Now we need to find the radius of rotation. Recall that the circumcenter of an equilateral triangle is intersection of the altitudes. Drawing all three altitudes, we can analyze a subtriangle from the resulting diagram. If we call the 3 outer vertices RGB, the midpoints of these outersegments where the altitudes intersect the sides as ABC, and the circumcenter as D, we can analyze the triangel RDA. This is a right triangle where the leg RA has length L/2 and is opposite a $\cos 60 ^ \circ$ angle. The hypotenuse, RD, is the length we are trying to find because it denotes the distance from the point to the circumcenter. Using trigonometry, we find $RD = \frac{RA}{\cos 60 ^ \circ} = \frac{L}{2} * \frac{2}{\sqrt{3}} = \frac{L}{\sqrt{3}}$

We now need to find the angular momentum $L = I \omega$ . Since we have 3 point masses all of mass $0.001 kg$ , $I = 3*m*r^2 = 3*0.001 * (\frac{L}{\sqrt{3}})^2 = 0.003 * \frac{L^2}{3} = 0.001 L^2$ . $\omega = \frac{v}{r} = \frac{5000}{L} \frac{m}{s^2} * \frac{\sqrt{3}}{L} = \frac{8660}{L^2} \frac{rad}{s^2}$ .

We now get $L = I \omega = 0.001 L^2 * \frac{8660}{L^2} = 8.66 \frac{kg \; m^2}{s}$ .

Firstly, we must find the effective force acting on three balls, which is given by

$3F=3\times 2 \cos 60^\circ 1/L = 3/L N$

Now imagine the equilateral triangle is inscribed inside a circle. The radius of this circle is easily determined by the sine rule

$\frac{r}{ \sin 30^\circ } = \frac{Lm}{ \sin 120^\circ }$

$r= \frac{L m}{ \sqrt{3}}$

Therefore the effective torque acting on the 3 balls is $T= Fr = \frac{3}{L} N \times \frac{L m}{ \sqrt{3}} = \sqrt{3} Nm = Ia$

Where I is the moment of inertia, a is the angular acceleration. The moment of inertia of the 3 balls is

$I = 3Mr^{2} = 3M \times \frac{L^{2}}{3} = 3ML^2$ (M is the mass of one ball)

therefore, $a = \frac{ \sqrt{3} Nm }{ 3ML^2 } = \frac{1 Nm}{ \sqrt{3} ML^2 }$

The angular velocity $\omega$ after 5 seconds is therefore

$\omega = at = \frac{5 Nms}{ \sqrt{3} ML^2 }$

The angular momentum of the system of 3 particle is therefore

$I \omega = 3ML^{2} \times \frac{5 Nms}{ \sqrt{3} ML^2} = 5\sqrt{3} kgm^{2}/s$

Which is actually around $8.66kgm^{2}/s$ to 3 significant figures.

At t=0, there are two forces acting on R. One is from R->G while the other is from B->R. Hence their resultant has the direction of B->G and has magnitude 1/L. Since the direction of acceleration is perpendicular to the position vector from the centre of the triangle to R, the distance from R to the centre doesn't change (instantaneously). But the same can be said about G and B. this means that the net effect of the forces is that the triangle rotates as a whole. Hence the force and acceleration on each object is invariant with time.

$|F|=\frac{1}{L}$ $|a|=\frac{1}{mL}$ $|v|=\frac{t}{mL}$ Angular momentum $=mvr=m\times\frac{t}{mL}\times{L}{\sqrt{3}}$ $=\sqrt(3)t$

It is easily seen that the resultant force on each particle is $\frac{1}{L}$ , parallel to the opposite side of the triangle. However, due to these forces on each particle, the triangle has rotated at the next instant and the direction of the force also changes. However, it always remains parallel to the opposite side, i.e. perpendicular to the the line joining the particle to the centre of mass of the three particles. Hence the only effect of this force is to produce angular acceleration; the size of the triangle does not change. Linear acceleration of each particle is $\frac{1}{mL}$

Hence angular acceleration about the centre of mass of the three particles is $\frac{1}{mL}.\frac{1}{\frac{L}{\sqrt{3}}}$

$=\frac{\sqrt{3}}{mL^{2}}$

Hence angular velocity after time $t$

$=\frac{\sqrt{3}t}{mL^{2}}$

Total moment of inertia about centre of mass

$=3(m(\frac{L}{\sqrt{3}})^{2})$

$=mL^{2}$

Hence, angular momentum = $\sqrt{3}t$

in SI units. Plugging in the values, the answer follows.

By symmetry, the three balls will remain a distance L away from each other as they rotate about the centre of mass. To get the change in angular momentum, simply derive the net torque $T$ on the three balls. Let the radius of the inradius of the triangle be $R = \frac{\sqrt{3}\times L}{6}$ . There are 2 forces acting on each ball, one repelling the ball away from the previous ball, the other attracting the ball towards the next ball. These 2 forces act at the same perpendicular distance $R$ away from the centre. As such, there are $2 \times 3 = 6$ identical forces acting at a distance $R$ away from the centre. Since each ball is $L$ away from each other, then $F = \frac{1}{L}$ So the net torque is simply $T = 6 \times (\R \times F)$ . Evaluating this statement, we get $T = \sqrt{3}$ Now, since Torque is change in angular momentum divided by change in time, the change in angular momentum is $\Delta t \times T = 5\sqrt{3}$

We know that $F = \frac{1}{L}$ . Also, using properties of equilateral triangles, we can show that the distance to the center (which is the center of mass) is $\frac{L}{\sqrt{3}}$ .

Since the force is always perpendicular to the radial vector, the torque on each object is just $Fr = \frac{1}{L}\frac{L}{\sqrt{3}} = \frac{1}{\sqrt{3}}$ , so the total torque is $\tau_{\mathrm{net}} = \frac{3}{\sqrt{3}} = \sqrt{3}$ .

The angular momentum, then, is just $L = \int_0^5 \tau_\mathrm{net} \, dt = \int_0^5 \sqrt{3} \, dt = 5\sqrt{3} \approx \boxed{8.66025}$ kg m^2/s.

By symmetry it can be said that the three balls would exhibit similar motion and they would always form an equilateral triangle and the centroid of the triangle would not shift.

Resultant of the repulsive and attractive forces on any ball will be in a direction parallel to the side of the triangle opposite to the ball (this can be seen easily)

Let one of the balls move a small distance dx in this direction. Let us observe the motion from the centroid of the triangle. Let its instantaneous side length be l.

the angle between the direction of motion and of any ball and the line connecting it to the centroid = $60^\circ + 30^\circ =90^\circ$

therefore distance of the ball from the centroid = $\frac{l}{\sqrt{3}}$

new distance = $\sqrt{\frac{l^{2}}{3}+dx^{2}}$ = $\frac{l}{\sqrt{3}}$ ( $dx^2$ is too small to be considered)

thus the distance remains virtually constant and the path traced by the balls is circular.

thus the side length remains same as the distance of any vertex from the centroid remains same. therefore l = L at all instants

Hence we come to the conclusion that the balls move in a circular path of radius $\frac{l}{\sqrt{3}}$ the force being tangential and $\frac{1}{l}$ in magnitude.

therefore we can say that: F= $\frac{1}{l}$

or $\frac{m \times v \times dv}{dx}$ = $\frac{1}{l}$

or $\frac{m \times v \times dv}{\frac{l}{\sqrt{3}} \times d\theta}$ = $\frac{1}{l}$

or $\sqrt{3} \times m \times v \times dv$ =d $\theta$

$\Rightarrow$ $\sqrt{3} \times m \times v^2$ = $2 \times \theta$

but v= $\alpha \times \frac{l}{\sqrt{3}}$ where $\alpha$ is the angular velocity of the ball abut the centroid of the triangle

$\Rightarrow$ $\sqrt{3} \times m \times \alpha^2 \times \frac{l^2}{3}$ = $2 \times \theta$

$\alpha = \sqrt{\frac{2 \sqrt{3} \theta}{m L^2}}$

$\Rightarrow \frac{d\theta}{dt} = \sqrt{\frac{2 \sqrt{3} \theta}{m L^2}}$

Solving the above differential equation

$\theta =\frac{\sqrt{3} t^2}{2 m L^2}$

$\Rightarrow \alpha = \frac{\sqrt{3} t}{m L^2}$

angular momentum of one ball= $m \alpha l^2 = \frac{\sqrt{3} t}{m L^2}*\frac{m L^2}{3}$ = $\frac{t}{\sqrt{3}}$ = $\frac{5}{\sqrt{3}}$

total angular momementum = $3 \times \frac{5}{\sqrt{3}}=5 \times \sqrt{3}$ =8.66