You could find the maximum, but can you prove it?

Playing around with the expression by substituting values seems to give the maximum value as $\frac{3}{\sqrt{2}}$ , which occurs when $a=b$ . Let's try to prove this. It suffices to prove $\sqrt{2}\left (\sqrt{a(a+b)^3} + b\sqrt{a^2+b^2} \right ) \leq 3(a^2+b^2)$ The square roots on the LHS give a nice reason why we should apply QM-AM. However, we also want equality to occur at $a=b$ . Thus, we can't directly apply QM-AM. Rather, we first determine how we could break down the LHS such that we would be able to apply Weighted QM-AM with equality at $a=b$ . Substituting $a=b$ gives $\sqrt{a(a+b)^3} = 2a^2\sqrt{2}$ and $b\sqrt{a^2+b^2} = a^2\sqrt{2}$ . Thus, we want the first square root halved so that we get equality at $a=b$ . Thus, we have $\begin{aligned} \dfrac{\frac{1}{2} \sqrt{a(a+b)^3} + \frac{1}{2} \sqrt{a(a+b)^3} + b \sqrt{a^2+b^2}}{3} &\leq \sqrt{\dfrac{\frac{1}{4} a(a+b)^3 + \frac{1}{4} a(a+b)^3 + b^2(a^2+b^2)}{3}}\\ \implies \sqrt{2}\left (\sqrt{a(a+b)^3} + b\sqrt{a^2+b^2} \right ) &\leq \sqrt{3a(a+b)^3 + 6b^2(a^2+b^2)} \end{aligned}$ Thus, it suffices to prove $\begin{aligned} \sqrt{3a(a+b)^3 + 6b^2(a^2+b^2)} &\leq 3(a^2+b^2)\\ a(a+b)^3+2b^2(a^2+b^2) &\leq 3a^4+6a^2b^2+3b^4\\ a^4+3a^3b+5a^2b^2+ab^3+2b^4 &\leq 3a^4 + 6a^2b^2+3b^4\\ 2a^4 - 3a^3b + a^2b^2 - ab^3 + b^4 &\geq 0\\ (a-b)^2(2a^2+ab+b^2) &\geq 0 \end{aligned}$ The final line is true since squares are non-negative and $a, b$ are positive so the second bracket is never $\leq 0$ . Therefore, this inequality is proven, so the maximum value of the expression is $\frac{3}{\sqrt{2}}$ , occuring at $a=b$ .