Brightest elongation of a planet.

Sun, planet and Earth are the vertices of a triangle SPE, of which the two sides adjacent to S are known in AU: $SP=\frac12$ and $SE=1$ . We label the side $EP$ as $s$ .

The angles are labeled as $φ$ at S, $ψ$ at E, and $α$ at P.

The phase, i.e. the illuminated fraction of the circular disk as seen from Earth is given by $f=\frac{1+\cos α}{2}$

Brightness is inversely proportional to the square of the distance s, so these proportions combine into a function we want to maximize (note that we can drop the 2, because it is a constant factor): $L= \frac{1+\cos α}{ s^2}$

Also, $\cos α = -\cos(φ+ψ) = -\cos φ \cos ψ + \sin φ \sin ψ$

By dropping a perpendicular from $P$ onto $SE$ , we have two expressions for the new length: $\frac{1}{2} \sin φ = s \sin ψ$ , so $\sin ψ= \frac{\sin φ}{2s}$

We also have $\frac{1}{2} \cos φ + s \cos ψ =1$ , so $\cos ψ= \frac{2-\cos φ}{2s}$

Combining these, we get an expression for L as a function of $φ$ only: $L(φ) = \frac{s(φ)-\cos φ + \frac{1}{2}} {s^3(φ)}$

Applying the quotient rule we get $\frac{dL}{dφ}=\frac{(\frac{ds}{dφ}+\sinφ)s^3-(s-\cosφ+\frac{1}{2})3s^2\frac{ds}{dφ}}{s^6}$

Now the cosine rule tells us that $s=\sqrt{\frac{5}{4}-\cos φ}$ and hence $ds/dφ=\frac{\sinφ}{2s}$ .

Because we want to set $\frac{dL}{dφ}=0$ , we only need to consider the numerator: $(\frac{\sinφ}{2s}+\sinφ)s^3-\frac{3}{2}s \sinφ (s-\cos φ+\frac{1}{2})=0$

so either $\sin φ=0$ or $(1/2+s)s-\frac{3}{2}(s-\cos φ+\frac{1}{2})=0$ $s+2s^2-3s+3\cos φ-\frac{3}{2}=0$ $4s^2-4s+6\cos φ-3=0$ Time to substitute $s=\sqrt{\frac{5}{4}-\cos φ}$ : $5-4\cos φ-4\sqrt{\frac{5}{4}-\cos φ}+6\cos φ-3=0$ $1+\cos φ=\sqrt{5-4\cos φ}$ $1+2\cos φ+\cos^2φ=5-4\cos φ$ $\cos^2φ+6\cos φ-4=0$ $\cosφ=\sqrt{13}-3$

Now that $\cos φ$ is known, because $\frac{1}{2}\cosφ= 1- s \cos ψ$ , we have $\cos ψ=\frac{2-\cosφ}{2s}$ ,

we find $s=\sqrt{\frac{5}{4}-\sqrt{13}+3}=\frac{1}{2}\sqrt{13}-1$

$\cos ψ=\frac{5-\sqrt{13}}{\sqrt{13}-2}=\frac{(5-\sqrt{13})(\sqrt{13}+2)}{9}=\frac{\sqrt{13}-1}{3}=\frac{4}{1+\sqrt{13}}$ So the requested answer is $4+1+13=\boxed{18}$

For other circular inner planet orbits

If the planet's orbit has radius $r \lt 1$ , then we get $\sin \psi\ = \frac{r}{s}\sin\phi$ and $\cos \psi\ = \frac{1-r\cos \phi}{s}$ .

We want to maximize $L(\phi)=\frac{s(\phi)+r-\cos \phi}{s^3(\phi)}$ with $s=\sqrt{1+r^2-2r\cos\phi}$ .

Using $\frac{ds}{d\phi}=\frac{r}{s}\sin\phi$ we find the condition $\frac{dL}{d\phi}=\sin\phi \frac{s^2 -2rs-3r^2+3r\cos \phi}{s^5(\phi)}=0$ , which is met when

$\sin\phi = 0 \vee s^2 -2rs-3r^2+3r\cos \phi=0$

Substituting the expression for s gives $1+r^2-2r\cos\phi - 2r\sqrt{1+r^2-2r\cos\phi} - 3r^2 + 3r\cos \phi=0$

So that the condition writes as $(1 - 2r^2 + r\cos \phi )^2 = 4r^2(1+r^2-2r\cos\phi)$ , resulting in a quadratic in $\cos\phi$ :

$r^2\cos^2 \phi + (2r+4r^3) \cos \phi + 1-8r^2 = 0$ which has solution $\cos \phi=\sqrt{12+4r^2}-2r-\frac{1}{r}$

From here, the elongation $\psi$ can be found, for example using $\psi = \arcsin(\frac{r}{s} \sin \phi )$

For planets closer to the sun than 0.25 AU or further away than 1 AU, the above expression gives a value outside the range [-1, 1], and no longer defines a local maximum. For those planets maximum brightness is at $ψ=0$ (behind the sun!) or $ψ=π$ (opposition).

This link does the business.