Be careful now

Quick solution: The graph $y = \sqrt{x - 5}$ is an increasing function lying entirely in the first quadrant, so the only way a straight line can intersect it at two distinct points is if the slope of that line is positive. Thus the given equation can only have two real solutions if $a \gt 0,$ and only one of the given options, namely $0 \lt a \lt \frac{1}{10},$ satisfies this (necessary but not sufficient) condition.

More thoroughly, now that we've established that we must have $a \gt 0$ and that any points of intersection will be in the first quadrant, we can use the "square both sides" approach without concern for extraneous solutions. Doing this, we have that

$x - 5 = (ax)^{2} + 4ax + 4 \Longrightarrow a^{2}x^{2} - (1 - 4a)x + 9 = 0$

$\Longrightarrow x = \dfrac{(1 - 4a) \pm \sqrt{(1 - 4a)^{2} - 36a^{2}}}{2a^{2}} = \dfrac{(1 - 4a) \pm \sqrt{-20a^{2} - 8a + 1}}{2a^{2}}.$

We will then obtain two distinct (positive) solutions for $x$ if the discriminant is $\gt 0:$

$-20a^{2} - 8a + 1 \gt 0 \Longrightarrow -20\left(a^{2} + \dfrac{2}{5}a + \dfrac{1}{25}\right) + \dfrac{20}{25} + 1 \gt 0$

$\Longrightarrow -20\left(a + \dfrac{1}{5}\right)^{2} + \dfrac{9}{5} \gt 0 \Longrightarrow \left(a + \dfrac{1}{5}\right)^{2} \lt \dfrac{9}{100}$

$\Longrightarrow a + \dfrac{1}{5} \lt \dfrac{3}{10} \Longrightarrow a \lt \dfrac{1}{10}.$

Thus we require that $\boxed{0 \lt a \lt \dfrac{1}{10}}.$

Brian Charlesworth's solution is shorter, but here is the way that I solved this problem. First note that $\sqrt{x-5}=ax+2\Rightarrow a^2x^2+(4a-1)x+9=0$ . The maximum number of solutions this equation can have is 2. And if one is particularly mindful, she will try to prove that $a$ must be a real number (assuming $a$ could possibly come from the set of complex numbers). This can be realized in the following way. Let $\alpha$ and $\beta$ be real roots of the equation. By Vieta's formulas we must have $\alpha\beta=\frac{9}{a^2}$ and $\alpha + \beta=\frac{1-4a}{a^2}$ . After dividing these equations and applying some simple manipulation, one can find that $a=\frac{1}{4}-9\frac{\alpha + \beta}{4\alpha\beta}$ . So, as long as both roots are nonzero (which they are since substituting $x=0$ in the quadratic leads to $9=0$ ), $a$ is a real number. Next, note $a^2x^2+(4a-1)x+9=0 \Leftrightarrow \pm\sqrt{x-5}=ax+2$ . However, we want something equivalent to $\sqrt{x-5}=ax+2$ . Leaving a bit of deciphering to the reader, one can come up with the relationship: $\sqrt{x-5}=ax+2 \Leftrightarrow ax+2\geq 0 \wedge \pm\sqrt{x-5}=ax+2$ $\Leftrightarrow ax+2 \geq 0 \wedge a^2x^2+(4a-1)x+9=0.$ Therefore, all one has to do now is ensure two things: the discriminant of the quadratic is positive and the leading coefficient is nonzero (so that there are two solutions to the quadratic) and $ax+2\geq 0$ for each solution. First I find the values of $a$ for which the discriminant is positive: $(4a-1)^2-4a^2(9)=-20a^2-8a+1>0\Leftrightarrow -\frac{1}{2}<a<\frac{1}{10}$ . Also $a\neq 0$ because, otherwise, the equation would reduce to a linear equation. Now, alongside that, we need to test the solutions via the quadratic formula in the inequality $ax+2\geq 0$ (noting that the inequality must be satisfied by both roots). $\frac{1-4a\pm\sqrt{-20a^2-8a+1}}{2a}+2\geq 0$ (The " $\pm$ " is in the sense that both must satisfy the inequality.) Since I am going to multiply each side of the inequality by $2a$ , it is necessary to consider separate cases for when $a$ is positive and $a$ is negative, as one always should whenever multiplying each side of an inequality by a variable. Assume $a>0$ . $\frac{1-4a\pm\sqrt{-20a^2-8a+1}}{2a}+2\geq 0 \Leftrightarrow 1-4a\pm\sqrt{-20a^2-8a+1}+4a\geq 0$ $\Leftrightarrow 1\geq \pm\sqrt{-20a^2-8a+1}\Leftrightarrow 1\geq \sqrt{-20a^2-8a+1}$ $\Leftrightarrow 1\geq -20a^2-8a+1 \Leftrightarrow a(5a+2)\geq 0 \Leftrightarrow a>0.$ Now assume $a<0$ . $\frac{1-4a\pm\sqrt{-20a^2-8a+1}}{2a}+2\geq 0\Leftrightarrow 1-4a\pm\sqrt{-20a^2-8a+1}+4a\leq 0$ $\Leftrightarrow 1\leq \pm\sqrt{-20a^2-8a+1}.$ Since $1$ is not less than or equal to a negative number, there can be no values of $a$ such that $a<0$ . So, in conclusion, $\sqrt{x-5}=ax+2$ has two distinct solutions iff $0<a<\frac{1}{10}$ .