Does $f(x)$ not matter?

Let us take note of this integral:

$\large \int_a^b f(x) dx + \int_{f(a)}^{f(b)} f^{-1}(x) dx = b f(b) - a f(a)$

Well, it looks like this, graphically..

Now, the points $(a, f(a))$ and $(b,f(b))$ are, as stated in the problem, the intersections of the given function $f(x)$ and its inverse $f^{-1}(x)$ . Since every function's inverse is its reflection over the line $y=x$ , the intersections are undoubtedly part of this line. This will mean that $f(a)=a$ and $f(b)=b$ .

So now we know what $f(a)$ and $f(b)$ are.. And that simplifies the above integral as

$\large \int_a^b [f(x) dx + f^{-1}(x) ]dx = b^2 - a^2$

So now, we get this

$b^2 - a^2 = 17$

knowing that both $a$ and $b$ are integers, so should be the factors of $b^2 - a^2$ .

since 17 is prime, then...

$(b-a)(b+a) = 17$

can be broken down to a system such that

$b-a = 1$ $b+a = 17$

or vice versa.. For this case, it will give us $b=9$ and $a=8$ , so that $|a \times b| = \boxed{72}$