Calculus *and* Algebra

By differentiating both sides and applying the chain rule (twice in the case of the left side), we get the following:

$k (F(x)+x)^{k-1} \left(F'(x)+1\right) F'\left((F(x)+x)^k\right)=2 (F(x)+x) \left(F'(x)+1\right)-1$

Substituting $1$ in for $x$ , we then get

$k (F(1)+1)^{k-1} \left(F'(1)+1\right) F'\left((F(1)+1)^k\right)=2 (F(1)+1) \left(F'(1)+1\right)-1$

We must choose $F(1)$ such that $F'\left((F(1)+1)^k\right) = F'(1) \Rightarrow (F(1)+1)^k = 1$ , which gives $F(1) = 0$ . Thus, our equation above becomes $k F'(1) \left(F'(1)+1\right)=2 \left(F'(1)+1\right)-1$ , the solution to which is $F'(1) = \frac{2-k \pm \sqrt{k^2+4}}{2k}$ . For the case $F'(1) = \frac{2-k-\sqrt{k^2+4}}{2 k}$ , $F'(1)$ is asymptotic to $0$ as $k$ approaches $-\infty$ and $-1$ as $k$ approaches $\infty$ . Thus, for this case, $F'(1)$ will never be an integer. For the case $F'(1) = \frac{2-k+\sqrt{k^2+4}}{2 k}$ , $F'(1)$ is positive for positive $k$ and asymptotic to $0$ as $k$ approaches $\infty$ . Thus, the largest value of $k$ such that $F'(1)$ is an integer is the largest value of $k$ where $F'(1) = 1$ . $\frac{2-k+\sqrt{k^2+4}}{2 k} = 1 \Rightarrow k^2+4 = (3k-2)^2 \Rightarrow 2k^2 - 3k = 0 \Rightarrow k = \frac{3}{2}$ , so $a+b = \boxed{5}$ .