Brilliant Geometric Probability

We are told that $a,b,1$ form a triangle, and so we know that $a+b > 1$ . For the triangle to be obtuse, its largest angle must be obtuse, and so the cosine of its largest angle must be negative. The largest angle is the one opposite the longest side (which is $c=1$ ). Thus we require that $\frac{a^2 + b^2 - 1}{2ab} < 0$ so that $a^2 + b^2 < 1$ .

In words, we want to know $P[A|B]$ , where $A$ is the event that $a^2 + b^2 < 1$ and $B$ is the event that $a+b > 1$ , where $a,b$ are uniformly (and, presumably, independently ) distributed over $[0,1]$ . Thus

$P[A \cap B]$ is the area of the yellow segment, and $P[B]$ is the area of the upper triangle (the yellow and red areas combined), and so

$P[A \cap B] \; = \; \tfrac14\pi - \tfrac12 \hspace{2cm} P[B] \;= \; \tfrac12$ making the required answer $\tfrac12(\pi-2) = 0.5707963268$ .

Alternatively, after one reaches the constraints $a^2+b^2<1\equiv b<\sqrt{1-a^2}\text{ and } a+b>1\equiv b>1-a$ , one can solve directly: (noting that $1-x<\sqrt{1-x^2}$ for all $x\in(0,1)$ :

$P(1-a<b<\sqrt{1-a^2}\cup 1-b<a<\sqrt{1-b^2})=2\cdot\displaystyle\int_{0}^{1}\sqrt{1-x^2}-(1-x)\cdot dx=\frac{\pi-2}{2}$