A composition of polynomials

Since $x-y$ divides $x^n-y^n$ for all positive integers $n$ , we deduce that $g(x)-x$ divides $\begin{array}{rcl} f(g(x)) - f(x) & = & x^6 + 3x^5 - 6x^3 + 6x^2 \\ & = & x^2(x^4 + 3x^3 - 6x + 6) \end{array}$ and this quartic polynomial is irreducible over the integers (using Eisenstein's criterion with $p=3$ ). If $g(x)-x$ had degree $m\ge4$ , then $g(x)$ would have the same degree, and hence $f(g(x))$ would have degree at least $2m\ge8$ . But this would force $f(x)$ to have the same degree as $f(g(x))$ , which is impossible. Thus $g(x)-x$ has degree at most $3$ , so is coprime to $x^4 + 3x^3 - 6x + 6$ , and hence it divides $x^2$ . There are $6$ options to consider:

• If $g(x)-x=1$ , then $g(x)=x+1$ . However, if $f(x)$ has leading term $ax^n$ , then $f(x+1)-f(x)$ has leading term $nax^{n-1}$ , and hence $f(x+1)-f(x)$ cannot be equal to a monic polynomial of degree $6$ . This case is not possible.

• If $g(x)-x=-1$ , then $g(x)=x-1$ . But, if $f(x)$ has leading term $ax^n$ , $f(x-1)- f(x)$ has leading term $-nax^{n-1}$ , and hence $f(x-1)-f(x)$ cannot be a monic polynomial of degree $6$ . This case is not possible.

• If $g(x)-x=x$ , then $g(x) = 2x$ . But. if $f(x)$ has leading term $ax^n$ , $f(2x)-f(x)$ has leading term $(2^n-1)ax^n$ , and so cannot be equal to a monic polynomial of degree $6$ . This case is not possible.

• We can have $g(x)-x=-x$ , so that $g(x)=0$ and $f(x)=-(x^6+3x^5-6x^3+6x^2)$ . This case gives $f(10)=-1294600$ .

• If $g(x)-x=x^2$ , we have $g(x)=x^2+x$ and therefore $f(x)$ must be cubic. Chasing coefficients, it follows that $f(x)=x^3-3x^2+6x$ . This gives $f(10)=760$ .

• If $g(x)-x=-x^2$ , we have $g(x)=x-x^2$ and therefore $f(x)$ must be cubic. However, if $f(x)$ has leading term $ax^3$ , then $f(x-x^2) - f(x) \,=\, -ax^6 + 3ax^5 + \cdots$ , and it is clear that no value of $a$ exists so that $f(x-x^2)-f(x) = x^6 + 3x^5 - 6x^3 + 6x^2$ . This case is also impossible.

Thus there are only two valid cases, and since $\big|-1294600\big| + \big|760\big| \; = \; 1295360$ so the required answer is $360$ .

Note that $g(x) - x \mid f(g(x)) - f(x) = x^6 + 3x^5 - 6x^3 + 6x^2$ . Further, note that $x^4 + 3x^3 - 6x + 6$ is irreducible, limiting the possible values of $g(x) - x$ .

Case 1: $g$ has degree at least 2:

Note that the degree of $f(g(x)) - f(x)$ is equal to the degree of $f(g(x))$ , since that is strictly larger than the degree of $f(x)$ .

Since $f$ has degree at least 2, $g$ has degree at most 3. However, there is no degree 3 factor of $x^6 + 3x^5 - 6x^3 + 6x^2$ , so $f$ has degree 3 and $g$ has degree 2. We know that $g$ is either $x + x^2$ or $x - x^2$ , since there are only two factors of $x^6 + 3x^5 - 6x^3 + 6x^2$ of degree 2, namely $x^2$ and $-x^2$ .

Now, it is easy to substitute those two expressions into $f(x) = Ax^3 + Bx^2 + Cx$ to determine that only the first works, giving a unique solution $f(x) = x^3 - 3x^2 + 6x$ , with $f(10) = 760$ .

Case 2: $g$ is constant. $g(x) = k \forall x$ .

Then $k$ is an integer such that $k^6 + 3k^5 - 6k^3 + 6k^2 = 0$ , so $k = 0$ . This gives $f(x) = -(x^6 + 3x^5 - 6x^3 + 6x^2)$ , with $|f(10)| = 1294600$ .

Case 3: $g$ is linear. Say $g(x) = px + q$ .

Note that $(p-1)x + q \mid x^6 + 3x^5 - 6x^3 + 6x^2$ , so either $p = 1$ , or $p - 1 = \pm 1$ . But $p \neq 0$ , so $p = 2$ in this case.

Case 3.1: $p = 2$ .

Then $x + q \mid x^6 + 3x^5 - 6x^3 + 6x^2$ , so $q = 0$ . It is easy to see, considering the coefficient of $x^6$ , that $g(x) = 2x$ has no solution.

Case 3.2: $p = 1$ .

Then the degree of $f(x+q) - f(x)$ is one less than the degree of $f(x)$ , so $f$ is of degree 7. However, this gives a leading coefficient of $f(x+q) - f(x)$ equal to seven times the leading coefficient of $f$ , which is patently impossible.

Hence, the required sum is $1294600 + 760 = 1295360$ , giving an answer of $\boxed{360}$ .

We can see that the degree of 2 sides must be equal. In the case deg(f) >= 6: we have f = (-x^6 +3x^5-6x^3+6x^2) In the case def(f) < 6 we have two cases of deg(f): 2 or 3 Solve we have f(x) = x^3 - 3x^2 + 6x

So that, result is 360