Not AM-GM

Since $\frac{a}{c} + \frac{b}{d} + \frac{c}{a} + \frac{d}{b}$ and $\frac{a}{b} + \frac{b}{c} + \frac{c}{d} + \frac{d}{a} = 4$ are homogeneous of degree zero and $ac=bd$ is homogeneous of degree two, we may arbitrarily choose one of the variables, so we may assume $d=1$ . Then the equation $ac=bd$ becomes $b = ac$ .

Using both these equations to rewrite the rest of the problem, we see we want to find the maximum of $\frac{a}{c} + \frac{ac}{1} + \frac{c}{a} + \frac{1}{ac} = \left(a+\frac{1}{a}\right)\left(c+\frac{1}{c}\right)$ such that $4 = \frac{a}{ac} + \frac{ac}{c} + \frac{c}{1} + \frac{1}{a} = \frac{1}{c} + a + c + \frac{1}{a} = \left(a+\frac{1}{a}\right) + \left(c+\frac{1}{c}\right)$

For $x>0$ and $x\neq 1$ , we always have $x + \frac{1}{x} > 2$ . In this case, since $a,c\neq d=1$ , if we assume $a,c > 0$ , we would have $4 = \left(a+\frac{1}{a}\right) + \left(c+\frac{1}{c}\right) > 2 + 2 = 4,$ which is a contradiction. Therefore, at least one of $a,c$ is negative, so without loss of generality, we may assume $a < 0$ , so that, in particular, $a + \frac{1}{a} \leq -2.$

Finally, simplifying notation by setting $x = a + \frac{1}{a}, \quad y = c + \frac{1}{c}$ , we want to maximize $xy$ such that $x+y=4$ and $x\leq -2$ . Solving for $y$ in the constraint equation, we see we want to maximize $x(4-x)$ such that $x \leq -2$ . It's easy to see this maximum occurs at $x=-2$ , so the maximum is $-2\cdot(4-(-2)) = -12$ .

This shows that the maximum is at most $-12$ . To see the maximum is equal to $-12$ , we must unwrap the above work to check that the variable assignments I've made don't contradict the assumption that $a,b,c,d$ are distinct real numbers. I'll spare you the details, but once you unwrap all of the above work, we see that $(a,b,c,d) = (-1, -3-2\sqrt{2}, 3+2\sqrt{2}, 1)$ yields the maximum value of $\boxed{-12}$ .

If we write $x \; = \; \frac{a}{b} \hspace{1cm} y \; = \; \frac{b}{c} \hspace{1cm} z \; = \; \frac{c}{d} \hspace{1cm} w \; = \; \frac{d}{a}$ then we are interested in maximising $xy + yz + zw + wx \; = \; (x+z)(y+w)$ subject to the conditions $x+y+z+w \; = \; 4 \hspace{1cm} xz \; = \; yw \; = \; 1$ and with none of $x,y,z,w,xy,zw$ equal to $1$ . Putting $X = x + x^{-1}$ , $Y = y + y^{-1}$ we see that we are maximising $XY$ subject to the conditions that $X+Y=4$ and $X,Y \not\in (-2,2]$ and $X \neq Y$ . It is not possible for $X,Y$ both to be positive and satisfy these constraints; nor is it possible for both to be negative. Thus we (by symmetry) may assume that $Y = 4 - X$ where $X \ge 6$ , and we simply have to maximise $X(4-X)$ for $X \ge 6$ . This happens when $X=6$ , and the maximum value is $\boxed{-12}$ .

Note that this value of $-12$ is achieved for $x=3+2\sqrt{2}$ , $y=-1$ , $z = 3-2\sqrt{2}$ , $w=-1$ , so when $a \; = \; -1 \hspace{1cm} b \; = \; 2\sqrt{2} - 3 \hspace{1cm} c \; = \; 3-2\sqrt{2} \hspace{1cm} d \; = \; 1$