This problem may remind you of the Pythagoras' Theorem but in 3D. Instead of the length of two legs, we have the areas of three triangular faces $a=\triangle OXY$ , $b=\triangle OYZ$ , and $c=\triangle OXZ$ . Instead of finding the length of the hypotenuse, we are trying to find the area of the triangle "hypotenuse" $\triangle XYZ$ . Coincidentally, the formula is similar to the 2D Pythagoras' Theorem but with an extra variable: $\sqrt{a^2+b^2+c^2}$ .

Let's try to prove it formally. There's a couple of methods that we can use here, but I will try working with vectors. (Another method involves finding the base and subsequently the height of $\triangle XYZ$ through repeated use of Pythagoras' Theorem.)

If we are given two sides of a triangle in vector form $\vec u, \vec v$ , we can find its area using the cross product with $\frac{1}{2}|\vec u \times \vec v|$ .

Suppose that the vector from $O$ to $X$ is $\vec x=\begin{pmatrix}x\\0\\0\end{pmatrix}$ , that from $O$ to $Y$ is $\vec y=\begin{pmatrix}0\\y\\0\end{pmatrix}$ , and that from $O$ to $Z$ is $\vec z=\begin{pmatrix}0\\0\\z\end{pmatrix}$ .

If you're wondering why we introduced new variables here, realize that the areas of $a=\triangle OXY$ , $b=\triangle OYZ$ , and $c=\triangle OXZ$ can be easily expressed in terms of $x,y,z$ . In fact, $a=\frac{1}{2}xy$ , $b=\frac{1}{2}yz$ , and $c=\frac{1}{2}xz$ , respectively, which we will use later.

Now, we try to find two sides of $\triangle XYZ$ . In this example, I'm going to use $XY$ and $XZ$ .

Using vector algebra, we have $\vec {XY}=\vec y-\vec x=\begin{pmatrix}0-x\\y-0\\0-0\end{pmatrix}=\begin{pmatrix}-x\\y\\0\end{pmatrix}$ and $\vec {XZ}=\vec z-\vec x=\begin{pmatrix}0-x\\0-0\\z-0\end{pmatrix}=\begin{pmatrix}-x\\0\\z\end{pmatrix}$ .

With this, we can finally find the area:

$\frac{1}{2}|\vec {XY} \times \vec {XZ}|$ $=\frac{1}{2}\left|\begin{pmatrix}-x\\y\\0\end{pmatrix} \times \begin{pmatrix}-x\\0\\z\end{pmatrix}\right|$

Computing the cross product, we have $=\frac{1}{2}\left|\begin{pmatrix}yz\\xz\\-xy\end{pmatrix}\right|$ $=\frac{1}{2}\sqrt{(yz)^2+(xz)^2+(-xy)^2}$

Now, bring in the $\frac{1}{2}$ from the outside: $=\sqrt{\left(\frac{1}{2}yz\right)^2+\left(\frac{1}{2}xz\right)^2+\left(\frac{1}{2}xy\right)^2}$

Remember how we said that $a=\frac{1}{2}xy$ , $b=\frac{1}{2}yz$ , and $c=\frac{1}{2}xz$ ? Substitute these in to get the final result:

$\sqrt{a^2+b^2+c^2}$

A Traditional Way

Approach

For this question, we could start by find some calculation techniques for triangle's area.

And substitute the area of three triangle $a$ , $b$ , and $c$ in.

Triangle's area

Here, I choose Heron's formula.

Wikipedia

Which is

$area=\sqrt{s\left(s-A\right)\left(s-B\right)\left(s-C\right)}$

$s=\frac{\left(A+B+C\right)}{2}$

Where $A$ , $B$ , and $C$ is three side of the triangle.

Find side of the triangle $A$ , $B$ , and $C$

By Pythagoras Theorem, we could find three side of the triangle as:

$A=\sqrt{x^2+y^2}$

$B=\sqrt{y^2+z^2}$

$C=\sqrt{z^2+x^2}$

Find side $A$ , $B$ , and $C$ in term's of area $a$ , $b$ , and $c$

As we could see, area $a$ , $b$ , and $c$ could be express as

$a=\frac{x\cdot y}{2}$

$b=\frac{y\cdot z}{2}$

$c=\frac{x\cdot z}{2}$

Since we would like to take $x$ , $y$ , and $z$ out from the function.

We could see that:

$\frac{\left(x\cdot y\right)\left(x\cdot z\right)}{\left(y\cdot z\right)}=x^2$

So if we apply this to a, b, and c, we could pull $x^2$ , $y^2$ , and $z^2$ out.

$2\frac{a\cdot c}{\ b}=2\frac{\left(\frac{x\cdot y}{2}\right)\left(\frac{x\cdot z}{2}\right)}{\left(\frac{y\cdot z}{2}\right)}=x^2$

$2\frac{a\cdot b}{c}=2\frac{\left(\frac{x\cdot y}{2}\right)\left(\frac{y\cdot z}{2}\right)}{\left(\frac{x\cdot z}{2}\right)}=y^2$

$2\frac{c\cdot b}{a}=2\frac{\left(\frac{x\cdot z}{2}\right)\left(\frac{y\cdot z}{2}\right)}{\left(\frac{x\cdot y}{2}\right)}=z^2$

So we could substitute $x^2$ , $y^2$ , and $z^2$ with $a$ , $b$ , and $c$ in side length $A$ , $B$ , and $C$ .

$A=\sqrt{\frac{a\cdot c}{b}\cdot2+\frac{a\cdot b}{c}\cdot2}$

$B=\sqrt{\frac{a\cdot b}{c}\cdot2+\frac{b\cdot c}{a}\cdot2}$

$C=\sqrt{\frac{b\cdot c}{a}\cdot2+\frac{a\cdot c}{b}\cdot2}$

Use Heron's formula

Let's simplify Heron's formula by substitute s as $A$ , $B$ , and $C$ .

$area=\sqrt{s\left(s-A\right)\left(s-B\right)\left(s-C\right)}$

Where

$s=\frac{\left(A+B+C\right)}{2}$

So

$area=\sqrt{\left(\frac{\left(A+B+C\right)}{2}\right)\left(\frac{\left(A+B+C\right)}{2}-A\right)\left(\frac{\left(A+B+C\right)}{2}-B\right)\left(\frac{\left(A+B+C\right)}{2}-C\right)}$

$area=\sqrt{\left(\frac{\left(A+B+C\right)}{2}\right)\left(\frac{\left(-A+B+C\right)}{2}\right)\left(\frac{\left(A-B+C\right)}{2}\right)\left(\frac{\left(A+B-C\right)}{2}\right)}$

$area=\sqrt{\left(\frac{\left(A+B+C\right)\left(-A+B+C\right)\left(A-B+C\right)\left(A+B-C\right)}{16}\right)}$

$area=\sqrt{\left(\frac{-A^4+2A^2B^2+2A^2C^2-B^4+2B^2C^2-C^4}{16}\right)}$

Then we could substitute $A$ , $B$ , and $C$ as $a$ , $b$ , and $c$ .

$\sqrt{\left(\frac{-\left(\frac{a\cdot b}{c}\cdot2+\frac{a\cdot c}{b}\cdot2\right)^2+2\left(\frac{a\cdot b}{c}\cdot2+\frac{a\cdot c}{b}\cdot2\right)\left(\frac{a\cdot b}{c}\cdot2+\frac{b\cdot c}{a}\cdot2\right)-\left(\frac{a\cdot b}{c}\cdot2+\frac{b\cdot c}{a}\cdot2\right)^2+2\left(\frac{a\cdot b}{c}\cdot2+\frac{a\cdot c}{b}\cdot2\right)\left(\frac{b\cdot c}{a}\cdot2+\frac{a\cdot c}{b}\cdot2\right)+2\left(\frac{a\cdot b}{c}\cdot2+\frac{b\cdot c}{a}\cdot2\right)\left(\frac{b\cdot c}{a}\cdot2+\frac{a\cdot c}{b}\cdot2\right)-\left(\frac{b\cdot c}{a}\cdot2+\frac{a\cdot c}{b}\cdot2\right)^2}{16}\right)}$

And we simplify:

$\sqrt{\frac{1}{4}\frac{a\cdot b}{c}\cdot2\frac{a\cdot c}{b}\cdot2+\frac{1}{4}\frac{b\cdot c}{a}\cdot2\frac{a\cdot c}{b}\cdot2+\frac{1}{4}\frac{a\cdot b}{c}\cdot2\frac{b\cdot c}{a}\cdot2}$

$\sqrt{\frac{a\cdot c}{b}\cdot\frac{b\cdot c}{a}+\frac{a\cdot b}{c}\left(\frac{a\cdot c}{b}+\frac{b\cdot c}{a}\right)}$

$\sqrt{a^2+b^2+c^2}$

Boom we got the answer!