Matrices: $A\neq I$ and $A^2=I$

Since $A=\frac{1}{5}\left(\begin{array}{cc} -3& a\\b& c \end{array}\right)$ , then $A^2=\frac{1}{25}\left(\begin{array}{cc} 9+ab& a(c-3)\\b(c-3)& c^2+ab \end{array}\right)=\left(\begin{array}{cc} 1& 0\\0& 1 \end{array}\right)$ .

This means that $\begin{cases} 9+ab &=&25 \\a(c-3)&=&0\\b(c-3)&=&0\\c^2+ab&=&25 \end{cases}$ . If $c\neq 3$ , then (from 2nd and 3rd equations), $a=0$ or $b=0$ , which means that $9+ab=9\neq25$ , a contradiction. Hence, $c=3$ . Now, from 1st and 4th equations, $ab=16$ . Now either both of $a,b>0$ of $a,b<0$ . To obtain the maximum value of $a+b$ , we need $a,b>0$ . As $a$ and $b$ are (positive) integers, $a+b$ is maximum if $(a,b)=(1,16)$ or $(16,1)$ . Thus the maximum value of $a+b+c$ is $\boxed{20}$ .