Doodle Probability

Without the loss of generality, let's assume the side length of the square to be $1$

We start by selecting four points inside the square.

The figure on left shows the cases where the four points have a concave hull (It can be seen that none of the colored configurations intersect).

The figure on right shows the cases where the four points have a convex hull (It can be seen that one out of the three colored configurations intersect).

Hence, the lines segments will intersect if and only if the four points form a convex figure (i.e none of the selected points is inside the triangle formed by the other three)

Let $P_{\text{convex}}$ be the probability that the four points form a convex figure.

If a convex figure is obtained, then the probability that the line segments intersect is $\dfrac{1}{3}$ .

If a concave figure is obtained, then the probability that the line segments intersect is $0$ .

Hence the probability we are looking for is $P=\dfrac{1}{3}\times P_{\text{convex}}$ .

Now we need to find $P_{\text{convex}}$ .

First, Let's calculate the probability $P_\alpha$ that the first point is inside the triangle formed by the other three.

$P_\alpha=\dfrac{\mathrm{E}\left(\Delta_\text{Triangle}\right)}{\Delta_\text{Square}}$

where $\mathrm{E}\left(\Delta_\text{Triangle}\right)$ is the expected value of the area of the triangle formed by the three points, and $\Delta_\text{Square}$ is the area of the square.

Same goes for $P_\beta, P_\gamma$ and $P_\delta$ . Obviously $P_\alpha=P_\beta=P_\gamma=P_\delta$ .

Probability that any point is inside of triangle formed by other three is $P_\alpha+P_\beta+P_\gamma+P_\delta=4P_\alpha$ because no more than one of these outcomes can happen simultaneously.

$P_{\text{convex}}=1-4P_\alpha=1-4\mathrm{E}\left(\Delta_\text{Triangle}\right)$

Now, we only need to find $\mathrm{E}\left(\Delta_\text{Triangle}\right)$ , which is given by

$\displaystyle\mathrm{E}\left(\Delta_\text{Triangle}\right)=\dfrac{\displaystyle\int_0^1\int_0^1\int_0^1\int_0^1\int_0^1\int_0^1|\Delta|dx_1dx_2dx_3dy_1dy_2dy_3}{\displaystyle\int_0^1\int_0^1\int_0^1\int_0^1\int_0^1\int_0^1dx_1dx_2dx_3dy_1dy_2dy_3}$

$=\displaystyle\int_0^1\int_0^1\int_0^1\int_0^1\int_0^1\int_0^1\dfrac{\left|-x_2y_1+x_3y_1+x_1y_2-x_3y_2-x_1y_3+x_2y_3\right|}{2}\,dx_1dx_2dx_3dy_1dy_2dy_3=\dfrac{11}{144}$

Here, $(x_i,y_i)$ represent the polygon vertices of the triangle for $i=1, 2, 3$ , and the (signed) area of these triangles is given by the determinant

$\Delta=\dfrac{1}{2!} \left|\begin{matrix} x_1 & y_1 & 1 \\ x_2 & y_2 & 1 \\ x_3 & y_3 & 1 \end{matrix}\right|=\dfrac{1}{2}\left(-x_2y_1+x_3y_1+x_1y_2-x_3y_2-x_1y_3+x_2y_3\right)$

$\implies\displaystyle\mathrm{E}\left(\Delta_\text{Triangle}\right)=\dfrac{11}{144}$

$\implies P_{\text{convex}}=1-4\mathrm{E}\left(\Delta_\text{Triangle}\right)=\dfrac{25}{36}$

$\implies P=\dfrac{1}{3}\times P_{\text{convex}}=\boxed{\dfrac{25}{108}}$

---

A better approach on finding $\mathrm{E}\left(\Delta_\text{Triangle}\right)$ can be found here and here

Click here for more information.