What is there to do

Note that $x^2 + 3xy + 3y^2= 25$ $\implies x^2 - 2 \left ( -\frac{\sqrt{3}}{2} \right ) \times x \times \sqrt{3}y + (\sqrt{3}y)^2= 25= 5^2$ $\implies x^2 - 2 x (\sqrt{3}y) \cos(150^{\circ}) + (\sqrt{3}y)^2= 25$ Similarly, $z^2 + 3yz + 3y^2= 169$ $\implies z^2 - 2 z (\sqrt{3}y) \cos(150^{\circ}) + (\sqrt{3}y)^2= 169= 13^2$ From the third equation, we obtain $2x^2 + 3xy - xz = 3yz$ From the first two equations, we know that $3xy= 25 - 3y^2 - x^2$ $3yz= 169 - 3y^2 - z^2$ Plugging them in the third equation, we obtain $2x^2 + 25 - 3y^2 - x^2 - xz= 169 - 3y^2 - z^2$ $\implies x^2 - xz + z^2 = 144$ $\implies x^2 - 2 \times \frac{1}{2} xz + z^2= 144$ $\implies x^2 - 2xz \cos(60^{\circ}) + z^2= 144= 12^2$ Now consider the triangle in the following figure.
Go here in case the image doesn't load: http://s24.postimg.org/jp6t8ehph/Untitled.png
Notice that the angles and lengths are consistent from the cosine rule and our first three observations.

Since $12^2+5^2= 13^2$ , we deduce that $\angle ABC= 90^{\circ}$ . Let $P$ be the point in the interior. We have $[\triangle APB]= \frac{\sqrt{3}}{2} xy$ $[\triangle APC]= \frac{\sqrt{3}}{2}yz$ $[\triangle BPC]= \frac{\sqrt{3}}{2}zx$ $[\triangle ABC]= 12 \times 5 = 60$ Note that $[\triangle APB] + [\triangle BPC + [\triangle APC] = [\triangle ABC]$ $\implies \frac{\sqrt{3}}{2} (xy+yz+zx)= 60$ $\implies xy+yz+zx= \frac{60 \times 2}{\sqrt{3}}= 40\sqrt{3}$ Thus, $a= 40$ , $b=3$ , $a+b= 40+3= \boxed{43}$ .

Motivation :Firstly,one must notice that the given values, $25$ and $169$ are perfect squares of $5$ and $13$ ,and they also form a pythagorean triplet with $12$ ,so one must think of using geometry to solve this problem.The idea will become more clear by reading my solution.

Firstly,as we are using geometry in this problem,we must use some concepts of geometry like similarity,cosine rule,sine rule etc.The another hardest part of this problem is to identify that concept which we have to apply in this problem.Let's see what is given!Also notice that the given information contains a lots of squares( $x^{2},z^{2},25,169$ ),and cosine rule is the only one which contains many squares.So,by simple hit and trial and observations,the given data can be rewritten as, $(5)^{2}=x^{2} +(y\sqrt{3})^{2} -2(\frac{-\sqrt{3}}{2})(x)(y\sqrt{3})$

$(13)^{2} = z^{2} +(y\sqrt{3})^{2} -2(\frac{-\sqrt{3}}{2})(z)(y\sqrt{3})$ and by subtracting these two equations and using the third given equation,we get $(12)^{2}=x^{2} + z^{2} - 2(\frac{\sqrt{3}}{2})(x)(z)$ The above 3 equations looks like cosine rule,so the above three equations,we need to arrange them such that these equations are geometrically related. So,it can be arranged by setting a point inside the triangle. For final diagram,download this file and if you don't have geogebra, google it and download it.Sorry,because I feel lazy to type,so please download the file.

From the diagram,the problem is converted into a geometry problem which is very simple to solve. The area of the 3 smaller triangles equal to the area of the bigger triangle.So, $(\frac{1}{2} \times \sin\ 150 \times y\sqrt{3} \times x) + (\frac{1}{2} \times \sin\ 150 \times z \times x) + (\frac{1}{2} \times \sin\ 60 \times y\sqrt{3} \times z) = \frac{1}{2} \times 5 \times 12$ Simplifying the above equation we get our answer. $xy+yz+zx = \boxed{40\sqrt{3}}$

Subtracting the first equation from the second equation, we get $z^2 - x^2 + 3yz - 3xy = 144.$ From the third equation, $3yz - 3xy = 2x^2 - xz$ . Substituting, we get $x^2 - xz + z^2 = 144.$

Let $A$ , $B$ , $C$ , and $P$ be points such that $AP = x$ , $BP = y \sqrt{3}$ , $CP = z$ , $\angle APB = \angle BPC = 150^\circ$ , and $\angle APC = 60^\circ$ . Then by the cosine law on triangle $ABP$ ,

$\begin{aligned} AB^2 &= x^2 - 2x(y \sqrt{3}) \cos 150^\circ + (y \sqrt{3})^2 \\ &= x^2 + 3xy + 3y^2 \\ &= 25, \end{aligned}$

so $AB = 5$ . Similarly, by the cosine law on triangle $BCP$ , $BC^2 = 3y^2 + 3yz + z^2 = 169,$ so $BC = 13$ , and by the cosine law on triangle $ACP$ , $AC^2 = x^2 - xz + z^2 = 144,$ so $AC = 12$ .

Since $AB^2 + AC^2 = BC^2$ , triangle $ABC$ is a right-triangle, and has area $\frac{1}{2} \cdot AB \cdot AC = \frac{1}{2} \cdot 5 \cdot 12 = 30.$

The area of triangle $ABP$ is $\frac{1}{2} \cdot AP \cdot BP \cdot \sin 150^\circ = \frac{1}{2} \cdot x \cdot y \sqrt{3} \cdot \frac{1}{2} = \frac{\sqrt{3}}{4} xy.$ Similarly, the areas of triangles $ACP$ and $BCP$ are $\frac{\sqrt{3}}{4} xz$ and $\frac{\sqrt{3}}{4} yz$ , respectively. Adding these areas up, we get $\frac{\sqrt{3}}{4} xy + \frac{\sqrt{3}}{4} xz + \frac{\sqrt{3}}{4} yz = 30,$ so $xy + xz + yz = \frac{120}{\sqrt{3}} = 40 \sqrt{3}.$

Although this was probably the intended solution, the given system does not actually have any solutions where $x$ , $y$ , and $z$ are all positive. If it did, then our construction would give us a triangle $ABC$ with sides 5, 12, and 13, and a point $P$ inside the triangle such that $\angle APC = 60^\circ$ . But $\cos \angle ABC = \frac{5}{13} < \frac{1}{2} = \cos 60^\circ,$ so $\angle ABC > 60^\circ$ , which means that no such point $P$ exists.

Let $3y^2=t^2$ . Then we have from the first equation, $5^2 = x^2 + t^2 + \sqrt{3}xt = x^2 + t^2 - 2xt \cos{150^{\circ}}$ .

Similarly from second equation, we get $13^2 = z^2 + t^2 + \sqrt{3}zt = z^2 + t^2 - 2zt \cos{150^{\circ}}$ .

Substituting the values of $zt$ & $xt$ from the first two equations into the third, we get:

$13^2 - z^2 - t^2 = 2x^2 +5^2 - x^2 - t^2 - xz \Rightarrow 12^2=x^2+z^2 -xz \Rightarrow 12^2=x^2+z^2 -2xz \cos{60}$ .

Now consider a right triangle $ABC$ with sides $c=5,a=12,b=13$ . There is a point $P$ inside such that $\angle APB = 150^{\circ}, \angle APC = 150^{\circ}, \angle BPC = 60^{\circ}$ . Applying cosine rule in these smaller triangles gives us the above three equations.

We are asked for $xy+yz+zx = \frac{xt}{\sqrt{3}} + \frac{zt}{\sqrt{3}} +zx = \frac{1}{\sqrt{3}}(xt + zt + \sqrt{3}xz)$ , which we get by adding the areas of the above three triangles computed using sine rule. Indeed, we get: $\frac{1}{2} (\frac{xt}{2} + \frac{\sqrt{3}xz}{2} + \frac{zt}{2}) = 30$ , whence we get the answer $\boxed{40\sqrt{3}}$ .