Just a theorem

Trigonometric solution!

Since $ABPC$ is cyclic, $\angle BPC=180^{\circ}-\angle BAC = 120^{\circ}$ . By Cosine rule,

$\begin{aligned} BC^2 &= BP^2 + CP^2 - 2\cdot BP\cdot CP \cdot \cos 120^{\circ}\\ &= a^2 + b^2 + ab \end{aligned}$

Therefore, $BC=\sqrt{a^2 + b^2 + ab}$ .

Since either $a,b$ can be $0$ , this rules out $\dfrac{ab}{a+b}$ and $\sqrt{ab}$ as they would yield a nonsensical $0$ . $\:\:a+b$ can be ruled out because the sum of $2$ sides of a triangle is greater than the $3$ rd. $\;\;$ Finally, $\sqrt{{a}^{2}-ab+{b}^{2}}$ can be ruled out because it yields a result that is less than the greater of $a$ or $b$ . $\;\;$ So, we are left with $\sqrt{{a}^{2}+ab+{b}^{2}}$ to choose as the "correct:" answer, without proof that it is indeed the correct expression.

Consult Alan's proof of the expression.

Let $L$ be the side of the triangle $\triangle ABC$ . Using Ptolemy's theorem on quadrilateral $ACPB$ we get:

$aL+bL=AP\cdot L \implies AP=a+b$

Now use Ptolemy's second theorem to get:

$\dfrac{AP}{L}=\dfrac{L^2+ab}{aL+bL} \implies AP(a+b)=L^2+ab \implies L^2=(a+b)^2-ab$

Finally, $L=\boxed{\sqrt{a^2+ab+b^2}}$ .