Help the Librarian!

The total number of ways to arrange the books can be calculated by Permutations with Repetition : $\dfrac{12!}{(3!)^4} = 369600$ .

Now, we calculate how many ways there are to arrange the books such that there exists 3 identical books in a row . There are $4$ ways to choose which kind of books to put them together, e.g. Math Books. Consider only 10 things: a group of Math Books and 9 other books. Again by Permutations with Repetition, there are $4 \times \dfrac{10!}{(3!)^3} = 67200$ ways . Now, we calculate how many ways there are to arrange the books such that there exists 2 groups of 3 identical books in a row . Similarily, there are ${4 \choose 2} \times \dfrac{8!}{(3!)^2} = 6720$ ways.

The number of ways to arrange the books such that there exists 3 groups of 3 identical books in a row is ${4 \choose 3} \times \dfrac{6!}{3!}=480$ . And for 4 groups, there are $4! = 24$ ways.

Now using Principle of Inclusion and Exclusion , the number of ways which satisfies the problem's conditions is $369600 - 67200 + 6720 - 480 + 24 = \boxed{308664}$ .

I like this problem because it uses a lot of combinatorics functions.