Diagonal Mystery

By drawing the perpendicular lines from points $A, B, C, D$ to meet one another as shown above, we can see that each blue triangle has the same area with each red neighboring triangle. The beige rectangle is, thus, the excess area to be computed.

Let the square side length = $s$ , the rectangle's width = $x$ , and length = $y$ . Then by Pythagorean theorem, $AC^2 = 104 = x^2 + s^2$ and $BD^2 = 116 = y^2 + s^2$ .

Hence, $y^2 - x^2 = 116-104 = 12 = (y-x)(y+x)$ .

Since all the points divide the side lengths into integers, $x$ & $y$ are also integers.

$12 = 1\times 12 = 2\times 6 = 3\times 4$ .

Since $(y+x) + (y-x) = 2y$ , which is even, the only combination that leads to even sum is $y-x = 2$ and $y+x = 6$ .

Therefore, $x = 2$ and $y = 4$ . Then $s = 10$ .

As a result the blue area = $\dfrac{10^2 - 2\times 4}{2} = 46$ .

Finally, the area of $ABCD$ = $46+8 = \boxed{54}$ .

The areas can be illustrated as shown below:

$Let ~horizontal ~ distance~ between ~A ~and ~C~ be ~X.\\ Let ~vertical ~ distance~ between ~B ~and ~D be ~Y.\\ S~ the~sides~of~the~square.\\ \therefore~S^2 + X^2=AC^2=(2\sqrt{26})^2=104.\\ \therefore~S^2 + Y^2=BD^2=(2\sqrt{29})^2=116\\ Since~S,~X,~Y ~are~integers,~and~AC>S,~and~BD>S, ~S<11.\\ If~S=10,~we~get~X=\sqrt{AC^2 - S^2}= 2,~and~Y=\sqrt{BD^2 - S^2}= 4.\\ This~satisfies~given~data.\\ Let~PQRS~be~the~square.\\ Let~A=P, ~~B=Q. ~So~CR=10-2=8,~and~SD=10-4=6.\\ So~area~out~of~ABCD~but~in~PQRS~is~as~under.\\ PSD=\frac 1 2*10*6=30.~~CRD=\frac 1 2*4*8=16.~~~30+16=46.\\ \therefore~Area~ABCD=10^2-46=\Large~~\color{#D61F06}{54}.$