Let's do it using coordinate geometry.

Let square be of side $a$

Now, each side is symmetric and so, solving using one side gives answer for all sides. So, let's solve considering $\color{#3D99F6}{P}$ on side $AB$ .

So, other two vertices will be on $AD$ and $BC$ .

Now, when $\color{#3D99F6}{P}$ is at mid point of $AB$ , $\color{#3D99F6}{Q}$ and $\color{#3D99F6}{R}$ will be on $AD$ and $BC$ such that they are at same distance from side $AB$ , by symmetry.

Now, as $\color{#3D99F6}{P}$ will move to left, $\color{#3D99F6}{Q}$ will move up and $\color{#3D99F6}{R}$ will move down. Similar is true when $\color{#3D99F6}{P}$ will move right.

At one point, $\color{#3D99F6}{P}$ will be at such point that $\color{#3D99F6}{Q}$ will coincide with $D$ and $\color{#3D99F6}{R}$ will be at some point. Similar is true when $\color{#3D99F6}{P}$ is moved right. At that point, $\color{#3D99F6}{P}$ can't be moved more as $\color{#3D99F6}{Q}$ will go off the sides. Let it be $\color{#3D99F6}{P'}$

So, let's find coordinate of $\color{#3D99F6}{P'}$ at that point.

$P'Q' = Q'R' = R'P' \text{ or } P'Q'^2 = Q'R'^2 = R'P'^2$

$\sqrt{a^2 + x^2} = \sqrt{a^2 + x^2} = \sqrt{(a - x)^2 + (a - x)^2}$

Solving, we get $x = a(2 - \sqrt3)$

---

Now, Probability that Equilateral triangle can be selected will be twice of $\frac{PP'}{AB}$ as $\color{#3D99F6}{P}$ could also be moved right and we have only considered left.

$Probability = 1 - 2\frac{PP'}{AB}$

$Probability = 1 - 2\frac{\frac{a}{2} - a(2-\sqrt3)}{a}$

$\color{#3D99F6}{Probability = 4 - \sqrt{12}}$

So, $a = 4$ and $b = 12$ and $\boxed{a^2 + b^2 = 160}$