Can you factorize the matrix?

Relevant wiki: Characteristic Polynomial

The equation given above is called characteristic polynomial of a matrix, and the following form applies:

$p_A(x) = \text{det}(xI - A),$

Clearly, if $x = A$ , it will result in zero matrix. By expanding the determinant of $xI - A$ , we will yield similar polynomial:

$p_A(x) = \text{det}(xI - A), = \begin{bmatrix} x-a & -b \\ -c & x-d \end{bmatrix} = (x-a)(x-d)-(-b)(-c) = x^2 - (a+d)x +(ad-bc)$ .

Since matrix $A$ results in zero matrix, then $a+d = 16$ and $ad-bc = -17$ .

Since all elements are positive integers, $a$ can vary from number $1$ to $7$ . Then if $a$ is even, $d = 16-a$ is even and will have common factor $2$ (but they are coprime in the condition), so $a$ is odd and can only be $1$ , $3$ , or $5$ .

If $a = 1$ , $d=15$ ; $bc = 32$ . The possible matrix is $\begin{bmatrix} 1 & 4 \\ 8 & 15 \end{bmatrix}$ , but $4$ and $8$ are not coprime. So it's not the solution matrix.

If $a = 3$ , $d=13$ ; $bc = 56$ . The possible matrix is $\begin{bmatrix} 3 & 7 \\ 8 & 13 \end{bmatrix}$ , but there should be two composites. So it's not the solution matrix.

Finally, if $a = 5$ , $d=11$ ; $bc = 72$ . The possible matrix is $\begin{bmatrix} 5 & 8 \\ 9 & 11 \end{bmatrix}$ , which fits all constraints and is our desired solution matrix.

As a result, $B=\begin{bmatrix} 8 & 5 \\ 9 & 11 \end{bmatrix}$ , so $det(B) = 8\cdot 11 - 5\cdot 9 = 88-45 = \boxed{43}$ .