A Special Locus

Let $E$ be the foot of the perpendicular to $AB$ from $P$ . Denote $AX$ , $XE$ , $EY$ , $YB$ and $PE$ by $k$ , $m$ , $n$ , $l$ and $h$ respectively, as seem on the figure.
Let $\angle XPE={{\theta }_{1}}$ and $\angle YPE={{\theta }_{2}}$ .

$\triangle PEX$ , $\triangle DAX$ and $\triangle PEY$ , $\triangle CBY$ are pairs of similar triangles, hence $\frac{h}{b}=\frac{m}{k}=\frac{n}{l} \ \ \ \ \ (1)$

$\left( 1 \right)\Rightarrow k=b\frac{m}{h} \ \ \ \text{and} \ \ \ l=b\frac{n}{h} \ \ \ \ \ (2)$ Since $\dfrac{m}{k}=\dfrac{m}{l}=\dfrac{m+n}{k+l}$ , $\left( 1 \right)\Rightarrow \frac{h}{b}=\frac{m+n}{k+l}\Rightarrow \frac{h}{b}=\frac{m+n}{a-\left( m+n \right)}\Rightarrow \frac{m}{h}+\frac{n}{h}=\frac{a-\left( m+n \right)}{b} \ \ \ \ \ (3)$ From the geometric progression,

$\begin{aligned} \frac{BY}{YX}=\frac{YX}{XA}\Rightarrow Y{{X}^{2}}=XA\cdot BY & \Rightarrow {{\left( m+n \right)}^{2}}=kl \\ & \overset{\left( 2 \right)}{\mathop{\Rightarrow }}\,{{\left( m+n \right)}^{2}}={{b}^{2}}\cdot \frac{m}{h}\cdot \frac{n}{h} \\ & \Rightarrow \frac{m}{h}\cdot \frac{n}{h}=\frac{{{\left( m+n \right)}^{2}}}{{{b}^{2}}} \ \ \ \ \ (4) \\ \end{aligned}$ Now, we have

$\tan \theta =\dfrac{\tan {{\theta }_{1}}+\tan {{\theta }_{2}}}{1-\tan {{\theta }_{1}}\cdot \tan {{\theta }_{2}}}=\dfrac{\dfrac{m}{h}+\dfrac{n}{h}}{1-\frac{m}{h}\cdot \dfrac{m}{h}}\overset{\left( 3 \right),\left( 4 \right)}{\mathop{=}}\,\dfrac{\dfrac{a-\left( m+n \right)}{b}}{1-\dfrac{{{\left( m+n \right)}^{2}}}{{{b}^{2}}}}=b\dfrac{a-\left( m+n \right)}{{{b}^{2}}-{{\left( m+n \right)}^{2}}}$ Setting $x=m+n$ , we have a function for $\tan \theta$ in terms of $x$ :

$f\left( x \right)=b\frac{a-x}{{{b}^{2}}-{{x}^{2}}}$ For the domain of $f$ we claim that $\text{dom}\left( f \right)=\left( 0,\dfrac{a}{3} \right]$ .
Proof: Let $r$ be the common ratio of the geometric progression, i.e. $\dfrac{l}{x}=\dfrac{x}{k}=r>0$ . Then, $l=rx$ , $k=\dfrac{x}{r}$ , thus we have $l+x+k=a\Rightarrow rx+x+\frac{x}{r}=a\Rightarrow x=\frac{r}{{{r}^{2}}+r+1}a$ An easy study of the function $g\left( r \right)=\dfrac{r}{{{r}^{2}}+r+1}$ shows that its range is $\text{ran}\left( g \right)=\left( 0,\dfrac{1}{3} \right]$ , hence the domain of $f$ is $\left( 0,\dfrac{a}{3} \right]$ .

The derivative of $f$ is
${f}'\left( x \right)=\dfrac{b}{{{\left( {{b}^{2}}-{{x}^{2}} \right)}^{2}}}\left( -{{x}^{2}}+2ax-{{b}^{2}} \right)$

The discriminant of the quadratic is $\Delta =4\left( {{a}^{2}}-{{b}^{2}} \right)$ .

Case 1 : $a\le b$
Then $\Delta \le 0$ and ${f}'\left( x \right) <0$ for all $x\in \left( 0,\dfrac{a}{3} \right)$ with an exception of a single root if $\Delta = 0$ . Hence $f$ is a decreasing function and, since $f$ is continuous, its range is $\left[ f\left( \dfrac{a}{3} \right),\underset{x\to {{0}^{+}}}{\mathop{\lim }}\,f\left( x \right) \right)$ Hence,
${{f}_{\min }}=f\left( \dfrac{a}{3} \right)=b\dfrac{a-\dfrac{a}{3}}{{{b}^{2}}-{{\left( \dfrac{a}{3} \right)}^{2}}}=\dfrac{6ab}{9{{b}^{2}}-{{a}^{2}}}$

Case 2 : $a>b$
Then,

$-{{x}^{2}}+2ax-{{b}^{2}}=0\Leftrightarrow x=\frac{-2a\pm 2\sqrt{{{a}^{2}}-{{b}^{2}}}}{-2}=a\mp \sqrt{{{a}^{2}}-{{b}^{2}}}$ The value $x=a+\sqrt{{{a}^{2}}-{{b}^{2}}}$ is greater than $a$ , hence it is rejected, therefore the only value we have to deal with is $x=a+\sqrt{{{a}^{2}}-{{b}^{2}}}$ .

Case 2i : $a-\sqrt{{{a}^{2}}-{{b}^{2}}}\ge \frac{a}{3}$ Then, $\dfrac{2a}{3}\ge \sqrt{{{a}^{2}}-{{b}^{2}}}\Rightarrow 4{{a}^{2}}\ge 9{{a}^{2}}-{{b}^{2}}\Rightarrow 5{{a}^{2}}\le 9{{b}^{2}}$ and in this case $-{{x}^{2}}+2ax-{{b}^{2}}<0$ for all $x\in \left( 0,\dfrac{a}{3} \right)$ , thus, like in case 1, $f$ is a decreasing function and ${{f}_{\min }}=\dfrac{6ab}{9{{b}^{2}}-{{a}^{2}}}$ .

Case 2ii : $a-\sqrt{{{a}^{2}}-{{b}^{2}}}<\dfrac{a}{3}\Leftrightarrow 5{{a}^{2}}>9{{b}^{2}}$
Then $f$ has a minimum at ${{x}_{0}}=a-\sqrt{{{a}^{2}}-{{b}^{2}}}$ , because ${f}'\left( x \right)<0$ for $x\in \left(0, {{x}_{0}} \right)$ and ${f}'\left( x \right)>0$ for $x\in \left( {{x}_{0}},\dfrac{a}{3} \right]$ .

The minimum value is

$\begin{aligned} {{f}_{\min }}=f\left( a-\sqrt{{{a}^{2}}-{{b}^{2}}} \right) & =\dfrac{b\sqrt{{{a}^{2}}-{{b}^{2}}}}{{{b}^{2}}-{{\left( a-\sqrt{{{a}^{2}}-{{b}^{2}}} \right)}^{2}}} \\ & =\dfrac{b\sqrt{{{a}^{2}}-{{b}^{2}}}}{{{b}^{2}}-{{a}^{2}}+{{b}^{2}}+2a\sqrt{{{a}^{2}}-{{b}^{2}}}} \\ & =\dfrac{b\sqrt{{{a}^{2}}-{{b}^{2}}}}{{{b}^{2}}-{{a}^{2}}+{{b}^{2}}+2a\sqrt{{{a}^{2}}-{{b}^{2}}}} \\ & =\dfrac{1}{2}\cdot \dfrac{b\sqrt{{{a}^{2}}-{{b}^{2}}}}{\sqrt{{{a}^{2}}-{{b}^{2}}}\left( a-\sqrt{{{a}^{2}}-{{b}^{2}}} \right)} \\ & =\dfrac{1}{2}\cdot \dfrac{b}{\left( a-\sqrt{{{a}^{2}}-{{b}^{2}}} \right)} \\ & =\dfrac{b\left( a+\sqrt{{{a}^{2}}-{{b}^{2}}} \right)}{2\left( {{a}^{2}}-{{a}^{2}}+{{b}^{2}} \right)} \\ & =\dfrac{a+\sqrt{{{a}^{2}}-{{b}^{2}}}}{2b} \\ \end{aligned}$

With $a=5$ and $b=3$ we are in case 2ii, hence our answer is: ${{f}_{\min }}=\dfrac{5+\sqrt{{{5}^{2}}-{{3}^{2}}}}{2\times 3}=\dfrac{3}{2}=\boxed{1.5}$

Let the midpoint of $AB$ be at the origin, with $A(\frac{1}{2}a, 0)$ , $B(-\frac{1}{2}a, 0)$ , $C(-\frac{1}{2}a, -b)$ , and $D(\frac{1}{2}a, -b)$ . Also draw $PR \perp CD$ with $R$ on $CD$ and $Q$ on $AB$ , and let $\theta_1 = \angle CPR$ and $\theta_2 = \angle DPR$ .

We can show that the locus of points for $P$ lie on the upper half of an ellipse with a major axis of $a$ , a minor axis of $b$ , and centered at the origin:

Using a parametric representation of the ellipse, $P$ will have coordinates $P(\frac{1}{2}a \cos t, \frac{1}{2}b \sin t)$ .

That means $\tan \theta_1 = \cfrac{CR}{PR} = \cfrac{\frac{1}{2}a \cos t + \frac{1}{2}a}{b + \frac{1}{2}b \sin t} = \cfrac{a(1 + \cos t)}{b(2 + \sin t)}$ and $\tan \theta_2 = \cfrac{RD}{PR} = \cfrac{\frac{1}{2}a - \frac{1}{2}a \cos t}{b + \frac{1}{2}b \sin t} = \cfrac{a(1 - \cos t)}{b(2 + \sin t)}$ .

Since $\triangle PYX \sim \triangle PCD$ by AA similarity, $\cfrac{YX}{CD} = \cfrac{PQ}{PR}$ , or $\cfrac{YX}{a} = \cfrac{\frac{1}{2}b \sin t}{b + \frac{1}{2}b \sin t}$ , which solves to $YX = \cfrac{a \sin t}{2 + \sin t}$ .

As alternate interior angles of parallel lines, $\angle BCY = \angle YPR = \theta_1$ , so that by $\triangle BCY$ , $BY = BC \tan \theta_1 = b \cdot \cfrac{a(1 + \cos t)}{b(2 + \sin t)} = \cfrac{a(1 + \cos t)}{2 + \sin t}$ .

Similarly, $\angle XDA = \angle QPX = \theta_2$ , so that by $\triangle XDA$ , $XA = AD \tan \theta_2 = b \cdot \cfrac{a(1 - \cos t)}{b(2 + \sin t)} = \cfrac{a(1 - \cos t)}{2 + \sin t}$ .

Since $(1 + \cos t)(1 - \cos t) = \sin^2 t$ , $BY \cdot XA = YX^2$ , which means $BY$ , $YX$ , and $XA$ are in a geometric progression.

Substituting $\tan \theta_1 = \cfrac{a(1 + \cos t)}{b(2 + \sin t)}$ and $\tan \theta_2 = \cfrac{a(1 - \cos t)}{b(2 + \sin t)}$ into $\tan (\theta_1 + \theta_2) = \cfrac{\tan \theta_1 + \tan \theta_2}{1 - \tan \theta_1 \tan \theta_2}$ gives $\tan \theta = \cfrac{2ab(2 + \sin t)}{b^2(2 + \sin t) - a^2\sin^2 t}$ .

The minimum and maximum $\theta$ values will occur when $\cfrac{d}{dt}(\tan \theta) = \cfrac{ab \cos t((a^2 - b^2) \sin^2 t + 4(a^2 - b^2) \sin t - 4b^2)}{(a \sin t - b(\sin t + 2))^2(a \sin t + b(\sin t + 2))^2} = 0$ , which is either at $\cos t = 0$ (which solves to $\sin t = 1$ ), or at $(a^2 - b^2) \sin^2 t + 4(a^2 - b^2) \sin t - 4b^2) = 0$ (which solves to $\sin t = -2 + \cfrac{2a}{\sqrt{a^2 - b^2}}$ ).

For the latter, since $-1 \leq \sin t \leq 1$ , then $-2 + \cfrac{2a}{\sqrt{a^2 - b^2}} \leq 1$ which solves to $5a^2 \geq 9b^2$ . Some careful analysis of the derivative function shows that the minimum occurs at $\sin t = 1$ when $5a^2 < 9b^2$ and at $\sin t = -2 + \cfrac{2a}{\sqrt{a^2 - b^2}}$ when $5a^2 \geq 9b^2$ .

Substituting these values of $\sin t$ back into $\tan \theta = \cfrac{2ab(2 + \sin t)}{b^2(2 + \sin t) - a^2\sin^2 t}$ gives us the following:

• when $5a^2 < 9b^2$ , the minimum $\theta$ occurs when $\sin t = 1$ , and $\tan \theta_{\text{min}} = \cfrac{2ab(2 + 1)}{b^2(2 + 1) - a^2 \cdot 1^2} = \cfrac{6ab}{9b^2 - a^2}$

• when $5a^2 \geq 9b^2$ , the minimum $\theta$ occurs when $\sin t = -2 + \cfrac{2a}{\sqrt{a^2 - b^2}}$ , and $\tan \theta_{\text{min}} = \cfrac{2ab(\frac{2a}{\sqrt{a^2 - b^2}})}{b^2(\frac{2a}{\sqrt{a^2 - b^2}}) - a^2 \cdot (-2 + \frac{2a}{\sqrt{a^2 - b^2}})^2} = \cfrac{a + \sqrt{a^2 - b^2}}{2b}$

In this problem, $a = 5$ and $b = 3$ , so $5a^2 \geq 9b^2$ , and $\tan \theta_{\text{min}} = \cfrac{5 + \sqrt{5^2 - 3^2}}{2 \cdot 3} = \boxed{1.5}$ .

Let $\overline{YX} = c$ and their common ratio be $p$ , so that $\overline {AX} = \frac{c}{p}$ and $\overline{YB} = cp$ . From that it follows:

$\large \displaystyle cp + c + \frac{c}{p} = a$

$\large \displaystyle p + 1 + \frac{1}{p} = \frac{a}{c}$

$\color{#20A900} (i) \ \ \boxed{ \large \displaystyle p + \frac{1}{p} = \frac{a-c}{c}}$

Also, let $\alpha$ and $\beta$ be marked as follows.

Notice also that $\angle{BYC} = \alpha$ and $\angle{AXD} = \beta$ . So:

$\large \displaystyle \tan(\alpha) = \frac{b}{cp}$

$\large \displaystyle \tan(\beta) = \frac{bp}{c}$

So:

$\large \displaystyle \tan(\alpha + \beta) = \frac{ \frac{b}{cp} + \frac{bp}{c}}{1 - \frac{b}{cp}\frac{bp}{c}}$

$\large \displaystyle \tan(\alpha + \beta) = b \left( p + \frac{1}{p} \right) \frac{c}{c^2-b^2}$

Plugging $(i)$ into this:

$\large \displaystyle \tan(\alpha + \beta) = b\left( \frac{a-c}{c} \right) \frac{c}{c^2-b^2}$

$\large \displaystyle \tan(\alpha + \beta) = \frac{b(a-c)}{c^2-b^2}$

Since $\theta$ and $\alpha+\beta$ add up to $180$ degrees, $\tan(\theta) = -\tan(\alpha + \beta)$ :

$\color{#20A900} (ii) \ \ \boxed{\large \displaystyle \tan(\theta) = \frac{b(c-a)}{c^2-b^2} }$

To minimize $\theta$ is the same as minimize $\tan(\theta)$ , since $0^o < \theta < 90^o$ and, in that interval, $\tan(\theta)$ is continuous and increasing . So, differentiating $(ii)$ with respect to $c$ , one gets:

$\large \displaystyle \frac{ b(c^2-b^2) - (bc-ab)2c}{ (c^2-b^2)^2} = 0$

If $c \neq b$ , then we can multiply both sides by $(c^2-b^2)^2$ . Then:

$\large \displaystyle c^2 - 2ac + b^2 = 0$

$\large \displaystyle [c - (a + \sqrt{a^2-b^2})]\cdot [c - (a - \sqrt{a^2-b^2})] = 0$

Since $c < a$ :

$\color{#20A900} \boxed{ \large \displaystyle c = a - \sqrt{a^2-b^2}}$ .

Which makes for the given case:

$\color{#3D99F6} \boxed{\large \displaystyle c=1}$

And:

$\color{#3D99F6} \boxed{\large \displaystyle \tan(\theta) = \frac{3}{2} = 1.5}$

Disclaimer: I've removed a flawed section of my solution because I made a wrong generalization.

---

To avoid the abuse of notation, I will relabel the vertex $A$ as $A'$ .

Let $(A'X,XY, YB) = \left( \frac AR, A, AR \right)$ , where $R > 1$ and $a = AB = \frac AR + A + AR = 5$ .

Then, $\angle YCD = \angle BYC = \tan^{-1} \left( \dfrac 3{AR} \right)$ .
Similarly, $\angle XDC = \angle A'XD = \tan^{-1} \left( \dfrac 3{A/R} \right)$ .

With $b = 3$ . $\angle CPD = \pi - \angle YCD - \angle XDC = \pi - \left [ \tan^{-1} \left( \dfrac 3{AR} \right) + \tan^{-1} \left( \dfrac 3{A/R} \right) \right ]$

Apply the compound angle formula, $\tan(x + y) = \dfrac{ \tan x + \tan y}{1 - \tan x \cdot \tan y } ,$ the equation above simplifies to $\angle CPD = \pi - \tan^{-1} \left( \dfrac{3A(R^2 + 1)}{R(A^2 - 9)} \right)$

Substitute $A = \dfrac 5{R + 1/R + 1}$ gives

$\angle CPD = \pi + \tan^{-1} \left [\dfrac{15(R^2+1)(R^2 + R + 1)}{(3R^2 -2R +3)(3R^2+8R+3)} \right ]$

So we want to minimize $f(R):= \dfrac{15(R^2+1)(R^2 + R + 1)}{(3R^2 -2R +3)(3R^2+8R+3)} \quad\quad R> 1$

A simple derivative test shows that $f(R)$ is minimized when $R^2-4R+1 = 0 \Rightarrow R = 2 + \sqrt3$ . Thus, $\min(f(R)) = f(2 + \sqrt3) = \frac32$ .

Hence, $\tan( \theta(5,3)) = \min( \angle CPD) = \tan \left [ \pi + \tan^{-1} \left( \frac32 \right) \right ] = \frac32 = \boxed{1.5}$