Which function has its derivative as sawtooth function?

First of all, $f(x)$ is decreasing iff $f'(x)<0$ , so, to answer the question, it suffices to find when $f'(x)<0$ .

The sawtooth function is a little hard to analyze, but I find it helpful to let $x=n+\epsilon$ , where $n$ is an integer and $0{\leq}{\epsilon}<1$ . This allows us to do more, because ${\lfloor}n+{\epsilon}{\rfloor}=n+{\lfloor}{\epsilon}{\rfloor}=n+0=n$ (you are allowed to pull integers out of the floor function, and $\epsilon$ is between 0 and 1, so it will be sent to 0).

With our new method, we can tackle the sawtooth function. First, input $x=n+\epsilon$ :

$\{n+\epsilon\}=n+\epsilon-{\lfloor}n+\epsilon{\rfloor}=n+\epsilon-n=\epsilon$

So $f'(n+\epsilon)=|n+\epsilon|-\{n+\epsilon\}=|n+\epsilon|-\epsilon$

This is much easier to work with. Let's try looking at it in different cases. We'll start with $n\geq0$ :

Because we also chose $\epsilon$ to be nonnegative, $n+\epsilon$ is nonnegative, so $|n+\epsilon|=n+\epsilon$ . Let's try and use that.

First, we're looking for $f'(x)<0\Rightarrow|n+\epsilon|-\epsilon<0\Rightarrow|n+\epsilon|<\epsilon$

Let's plug in our absolute value:

$|n+\epsilon|=n+\epsilon<\epsilon\Rightarrow n<0$

But we said $n\geq0$ ! Therefore, the interval we're looking for doesn't contain any $x=n+\epsilon$ such that $n\geq0$ .

Real quickly, take a step back: what values did we just eliminate? Well, we just eliminated $n\geq0$ and we know $\epsilon\geq0$ , so we eliminated all values where $n+\epsilon\geq0$ , meaning that our interval must only contain negative values.

[note: out of all the options listed, only one meets this criterion, so this would be enough to answer this multiple-choice question]

So, we just showed that $n\ngeq0\Rightarrow n<0$

Let $-m=n$ , so $m>0$ . Now, $n+\epsilon=(-m)+\epsilon=\epsilon-m$ which must be negative, so $|\epsilon-m|=m-\epsilon$ Like before, we will use this.

$f'(x)<0\Rightarrow|n+\epsilon|-\epsilon=|\epsilon-m|-\epsilon=m-\epsilon-\epsilon=m-2\epsilon<0\Rightarrow m<2\epsilon$

We know that $0{\leq}{\epsilon}<1$ , so $0{\leq}{2\epsilon}<2$ Therefore, $m<2\epsilon$ implies $m<2$ Remember that $m=-n$ , so $-n<2\Rightarrow n>-2$ This means that $n$ is an integer between $-2$ and $0$ , leaving only $n=-1$

Before we saw that $m<2\epsilon$ Because $m=-n=-(-1)=1$ , we know that $1<2\epsilon\Rightarrow\frac{1}{2}<\epsilon$

(note: because we are talking about $-1+\epsilon$ , larger values for $\epsilon$ correspond to smaller values after the decimal point)

So, $\frac{1}{2}<\epsilon<1$ , meaning that $f(x)$ is decreasing on the open interval $(-1+\frac{1}{2},-1+1)=(-\frac{1}{2},0)$