Weird inequality problem.

We first consider the cubic polynomial $P(t)=tb(t^2-b^2)+bc(b^2-c^2)+ct(c^2-t^2).$ It is easy to check that $P(b)=P(c)=P(-b-c)=0,$ and therefore $P(t)=(b-c)(t-b)(t-c)(t+b+c),$ since the cubic coefficient is $b-c$ . The LHS of the proposed inequality can therefore be written in the form $\left| ab(a^2-b^2)+bc(b^2-c^2)+ca(c^2-a^2) \right|= |P(a)|= \left| (b-c)(a-b)(a-c)(a+b+c) \right|.$ The problem comes down to finding the smallest number $\xi$ that satisfied the inequality $\left| (b-c)(a-b)(a-c)(a+b+c) \right|\leqslant\xi\cdot(a^2+b^2+c^2)^2. \tag{1}$ Note that this expression is symmetric, and we can therefore assume $a\leqslant b\leqslant c$ without loss of generality. With this assumption, $|(a-b)(b-c)|=(b-a)(c-b)\leqslant\left(\dfrac{(b-a)+(c-b)}2\right)^2=\dfrac{(c-a)^2}4,\tag{2}$ with equality iff $b-a=c-b$ , i.e. $2b=a+c.$ Also $\left( \dfrac{(c-b)+(b-a)}2 \right)^2 \leqslant \dfrac{(c-b)^2+(b-a)^2}2 ,$ or equivalently, $3(c-a)^2\leqslant2\cdot\left[ (b-a)^2+(c-b)^2+(c-a)^2 \right], \tag{3}$ again with equality only for $2b=a+c.$ From $(2)$ and $(3)$ we get \eqalign{ &\qquad |(b-c)(a-b)(a-c)(a+b+c)| \\ & \leqslant\ \ \dfrac14\cdot\left|(c-a)^3(a+b+c)\right|\\ &= \ \ \dfrac14\cdot\sqrt{(c-a)^6(a+b+c)^2}\\ &\leqslant \ \ \dfrac14\cdot\sqrt{\left( \dfrac{2\left[ (b-a)^2+(c-b)^2+(c-a)^2 \right]}3 \right)^3\cdot(a+b+c)^2}\\ &=\ \ \dfrac{\sqrt{2}}2\cdot\left(\sqrt[4]{ \left( \dfrac{(b-a)^2+(c-d)^2+(c-a)^2}3 \right)^3\cdot(a+b+c)^2 }\right)^2. } By AM-GM inequality this estimate continues as follows: \eqalign{ &\qquad |(b-c)(a-b)(a-c)(a+b+c)|\\ &\leqslant \ \ \dfrac{\sqrt{2}}2\cdot\left( \dfrac{(b-a)^2+(c-b)^2+(c-a)^2+(a+b+c)^2} 4 \right)^2\\ &= \ \ \dfrac{9\sqrt{2}}{32}\cdot(a^2+b^2+c^2)^2. } We see that the inequality $(1)$ is satisfied for $\xi=\tfrac9{32}\sqrt{2}$ , with equality iff $2b=a+c$ and $\dfrac{(b-a)^2+(c-b)^2+(c-a)^2}3=(a+b+c)^2.$ Plugging $b=(a+c)/2$ into the last equation we bring it to the equivalent form $2(c-a)^2=9(a+c)^2.$ The conditions for equality can now be restated as $2b=a+c\quad\text{and}\quad(c-a)^2=18b^2.$ Setting $b=1$ yiels $a=1-\tfrac32\sqrt{2}$ and $c=1+\tfrac32\sqrt{2}.$ We see that $\xi=\tfrac9{32}\sqrt{2}$ is indeed the smallest constant satisfying the inequality, with inequality for any triple $(a,b,c)$ proportional to $(1-\tfrac32\sqrt{2},1,1+\tfrac32\sqrt{2}),$ up to permutation.