Squares won't help you.

If the rectangle corners have coordinates $(0,0)$ , $(1,0)$ , $(1,X)$ and $(0,X)$ , where $0 < X < 1$ , then the fact that the marked angle inside the rectangle is a right angle means that the coordinate of the vertex of the blue triangle along the top edge is $(1-X^2,X)$ . Solving elementary simultaneous equations gives us the coordinates of the other two vertices of the blue triangle as $\left( \frac{1-X^2}{2-X^2},\frac{X}{2-X^2}\right) \hspace{1cm} \left(\frac{1}{X^2+1},\frac{X}{X^2+1}\right)$ Thus we can calculate the area of the blue triangle as $A_B \; = \; \frac{X^3 - X^5}{4 + 2 X^2 - 2 X^4}$ while the remaining orange area is $A_O \;= \; X - A_B \; = \; \frac{4 X + X^3 - X^5}{4 + 2 X^2 - 2 X^4}$ Thus we have the ratio $\frac{A_O}{A_B} \; = \; \frac{4 + X^2 - X^4}{X^2 - X^4} \; = \; 1 + \frac{4}{X^2(1-X^2)}$ which is minimized, taking the value $\boxed{17}$ , when $X^2 = \tfrac12$ .

If we go with the hunch that the minimum occurs when the figure is symmetric, i.e., the point where the lines meet at top is at center, then we solve the equation of the line from origin to that point to determine $s$ , the length of the rectangle of height $1$ :

$s(\frac{1}{2}s)=1$

which has the solution $s=\sqrt{2}$ . From this the ratio can readily be worked out.