Is There A Function?

Let $g(n)=2f(n)-n$ .

From the question, we have $6f(f(n))=5f(n)-n \implies 3g(f(n))=6f(f(n))-3f(n)=2f(n)-n=g(n)$

Now, since $g(n)=3g(f(n))=9 g (f (f(n))= \dots= 3^{m} g(\underbrace{f(f(f \ldots f}_{m \textrm{ times}}(n))$

Since $g: \mathbb{N} \to \mathbb{N}$ for any given $n,m \in \mathbb{N}$ , we have that $g(n) \equiv 0 \pmod {3^m}$

This implies for all $n$ , we have $g(n)=0$ . Since this implies $f(n)=\frac{n}{2}$ , we have a contradiction as $f: \mathbb{N} \to \mathbb{N}$ .

A more natural approach using the theory of recurrence relations. Identical to Chaebum's, just in a different language.

Suppose that such a function exists.
Fix an integer $n \neq 0$ .
Define $U_1 = n, U_k = f(U_{k-1} (n) )$ .
Observe that we have the recurrence relation $6U_k = 5 U_{k-1} - U_{k-2}$ .
Since the characteristic equation is $6m^2 - 5m + 1 = 0$ , which has roots $m = \frac{1}{3}, \frac{1}{2}$ , it follows that there are some constants $A, B$ such that
$U_k = A \frac{1}{3^k} + B \frac{ 1} { 2^k }$

If $A, B$ are non-zero, there it is clear that there is a $K$ large enough such that $U_K$ is not an integer. This contradicts the condition that the function maps the integers to the integers. Hence, we must have $A = B = 0$ .

In turn, we get a contradiction with $k = 1$ , since

$n = U_1 = 0 \frac{1}{3^k} + 0 \frac{ 1}{2^k} = 0$