Try Newton sums, maybe!

Manipulating second eqn:

$3((x_1^3+x_2^3)(x_1^2+x_2^2)-x_1^2x_2^2(x_1+x_2))=11(x_1^3+x_2^3)$ Using $\small{\color{teal}{x_1^2+x_2^2=5}}$ $4(x_1^3+x_2^3)-3x_1^2x_2^2(x_1+x_2)=0$

Using $\color{teal}{\small{x_1^3+x_2^3=(x_1+x_2)(\underbrace{x_1^2+x_2^2}_5-x_1x_2)}}$

$4((\color{#D61F06}{x_1+x_2})(5-x_1x_2)-3x_1^2x_2^2(\color{#D61F06}{x_1+x_2})=0$

$\implies ( \color{#D61F06}{x_1+x_2})(20-4x_1x_2-3x_1^2x_2^2)=0$ Two case arise:-

$\large{\star\star\star}~~~~~x_1+x_2=0$ Using this and $x_1^2+x_2^2=5$ we can find: $\large\color{#3D99F6}{F(x):x^2-\dfrac{5}{2}}$

---

$\large{\star\star\star}~~~20-4x_1x_2-3x_1^2x_2^2=0$ This is a quadratic in $x_1x_2$ - solve it to get $x_1x_2=2,\dfrac{-10}{3}$

Now, $x_1+x_2=\pm\sqrt{\underbrace{x_1^2+x^2}_5+2\underbrace{x_1x_2}_2}=\pm 3$

(Here $x_1x_2=\dfrac{-10}{3}$ is rejected because $x_1+x_2$ is taking imaginary values for that. Hence : $\large \color{#3D99F6}{F(x):x^2\pm 3x+2=0}$

---

Hence there are $3$ possible values of $F(x)$ . Direct substitution of $x=3$ to find $F(3)$ and whose product is:

$\large \dfrac{13}{2}\times 20\times 2=260$

$\huge\therefore\sqrt{256}=\boxed{\color{#D61F06}{16}}$

Let the polynomial be $F(x) = x^2 - ax + b$ . To make it easy for me to type, let $x_1 = p,\;x_2 = q$

$p+ q = a,\;pq = b$

Using Newton's sums:

$p^2 + q^2 = (p+q)^2 - 2pq\\ 5=a^2 - 2b$

$p^3 + q^3 = (p+q)(p^2 + q^2) - pq(p+q)\\ p^3 + q^3 = 5a - ab$

$p^4 + q^4 = (p+q)(p^3 + q^3) - pq(p^2 + q^2)\\ p^4 + q^4 = a(5a -ab) - b(5) = 5a^2 - a^2b - 5b$

$p^5 + q^5 = (p+q)(p^4 + q^4) - pq(p^3 + q^3)\\ p^5 + q^5 = a(5a^2 - a^2b - 5b) - b(5a-ab) = 5a^3 - a^3b -10ab + ab^2$

Given the 2 conditions:

$5=a^2 - 2b \implies b=\dfrac{a^2-5}{2} \quad$ Eq.(1)

$3(p^5 + q^5) = 11 (p^3 + q^3)\\ 3 (5a^3 - a^3b -10ab + ab^2)= 11(5a - ab)\\ 15a^3 - 3a^3b - 30ab + 3ab^2 = 55a - 11ab\\ 15a^3 - 3a^3b - 19ab + 3ab^2 - 55a = 0$

Substitute Eq.(1) in here:

$15a^3 - 3a^3\left(\dfrac{a^2-5}{2}\right) -19a\left(\dfrac{a^2-5}{2}\right) + 3a\left(\dfrac{a^2-5}{2}\right)^2 - 55a = 0\\ 60a^3 - 6a^5 + 30a^3 - 38a^3 + 190a + 3a(a^4 - 10a^2 + 25) - 220a =0\\ -3a^5 + 22a^3 + 45a = 0\\ 3a^5 - 22a^3 -45a = 0\\ a(3a^4 - 22a^2 - 45) = 0\\ a(3a^2 + 5)(a^2 - 9) = 0\\ a(3a^2 + 5)(a-3)(a+3) = 0\\ a=0,\;3,\;-3 \quad a^2 = -\dfrac{5}{3}$

The last factor does not give real values for $a$ , therefore we have 3 different $F(x)$

$a=0, \;b=\dfrac{0^2-5}{2} = -\dfrac{5}{2}\\ F(x) = x^2 - \dfrac{5}{2} \implies F(3) = \dfrac{13}{2}$

$a=3, \;b=\dfrac{3^2-5}{2} = 2\\ F(x) = x^2 - 3x + 2 \implies F(3) = 2$

$a=-3, \;b=\dfrac{(-3)^2-5}{2} = 2\\ F(x) = x^2 + 3x + 2 \implies F(3) = 20$

$\psi = \dfrac{13}{2} \times 2 \times 20 = 260\\ \sqrt{\psi -4} = \sqrt{260 -4} = \sqrt{256} = \boxed{16}$