Calculus, Geometry or Algebra?

Define vectors $\overrightarrow{a_1},\;\overrightarrow{a_2},\;\overrightarrow{a_3},\;\overrightarrow{a_4}$ as, $\begin{aligned} \overrightarrow{a_1}&=(3-x)\hat{i}+2\hat{j}&&\implies \mid \overrightarrow{a_1}\mid =\sqrt{x^2-6x+13} \\ \\ \overrightarrow{a_2}&=x\hat{i}+y\hat{j}&&\implies \mid \overrightarrow{a_2}\mid =\sqrt{x^2+y^2} \\ \\ \overrightarrow{a_3}&=1\hat{i}+(4-y)\hat{j} &&\implies \mid \overrightarrow{a_3}\mid =\sqrt{y^2-8y+17} \\ \\ \overrightarrow{a_4}&=\overrightarrow{a_1}+\overrightarrow{a_2}+\overrightarrow{a_3} \\ &=(3-x+x+1)\hat{i}+(2+y+4-y)\hat{j} \\ &=4\hat{i}+6\hat{j} \end{aligned}$ In a general quadrilateral, the sum of the lengths of any three sides is greater than the length of the fourth side , so we have, $\mid \overrightarrow{a_1}\mid+\mid \overrightarrow{a_2}\mid+\mid \overrightarrow{a_3}\mid\;\ge\; \mid \overrightarrow{a_4}\mid$ $\implies \sqrt{x^2-6x+13}+\sqrt{x^2+y^2}+\sqrt{y^2-8y+17}\ge \sqrt{4^2+6^2}=\sqrt{52}$ Therefore, the four vectors must lie on a straight line for equality to be achieved and the minimum value of the expression is $\sqrt{52}=2\sqrt{13}\implies AB=\boxed{26}$

Imagine the following diagram in which $AB = 2$ , $BD = 3$ , $DF = 4$ , and $FG = 1$ , and a path from $A$ to $G$ that passes through $BD$ at $C$ and $DF$ at $E$ :

If $x = CD$ and $y = DE$ , then by the Pythagorean Theorem on the three right triangles, the length of the path is $AC + CE + EG = \sqrt{2^2 + (3 - x)^2} + \sqrt{x^2 + y^2} + \sqrt{(4 - y)^2 + 1^2} = \sqrt{x^2 - 6x + 13} + \sqrt{x^2 + y^2} + \sqrt{y^2 - 8y + 17}$ , the given equation.

This path is a minimum when it is a straight line:

Since $AH = AB + BH = AB + FD = 2 + 4 = 6$ , and since $HG = HF + FG = BD + FG = 3 + 1 = 4$ , by Pythagorean Theorem on $\triangle AHG$ the minimum path will be $AG = \sqrt{AH^2 + HG^2} = \sqrt{6^2 + 4^2} = 2\sqrt{13}$ .

Therefore, $A = 2$ , $B = 13$ , and $AB = \boxed{26}$ .

Both Sathvik Acharya and David Vreken have posted remarkable solutions. To accentuate how incredible their solutions are, I will post an atrocious solution.

---

Let $f(x,y)$ denote the value of the expression that we want to minimize. Clearly $f(x,y)$ has no maximum value because we can set either $x$ or $y$ to be unboundedly large. In other words, if $f(x,y)$ has a critical point, it must be a minimum value.

At its critical points, both its partial derivatives must be 0, $\dfrac{\partial }{\partial x} f(x,y) = 0 \quad \Rightarrow\quad \dfrac x{\sqrt{x^2 + y^2}} + \dfrac{x-3}{\sqrt{x^2-6x+13}} = 0 \tag1$

$\dfrac{\partial }{\partial y} f(x,y) = 0 \quad \Rightarrow\quad \dfrac y{\sqrt{x^2 + y^2}} + \dfrac{y-4}{\sqrt{y^2-4y+17}} = 0 \tag2$

From $(1)$ , we can set $y>0$ as the subject (I've omitted the tedious algebra):

$y = \dfrac {2x}{\sqrt{x^2-6x+9}} \tag3$

Likewise, from $(2)$ , set $x>0$ as the subject: $x = \pm \dfrac y{y-4} \tag4$

Substitute $(4)$ into $(3)$ yields $(x,y) = (\tfrac73, 7), (5,5), (\tfrac53, \tfrac52), (7, \tfrac72) .$

Trial and error shows that among these 4 critical points, $(x,y) =(\tfrac53, \tfrac52)$ gives a smallest value of $f(x,y)$ .

Hence, $\min(f(x,y)) = f(\tfrac53, \tfrac52) = 2\sqrt{13} \Rightarrow A \times B = 2\times13=\boxed{26}.$

I am not gonna put a full explanation and it’s gonna use geometry and light ( physics )

The sum required is the sum of the distances in green

Here’s a Desmos link which is a little interactive on what the green distances are

If you notice it seems like there is a light ray coming from the origin bouncing off the point and landing on x=3 and y=4

If u keep the x coordinate constant and change the y coordinate you will also notice that point at which the light ray hits y=4 is constant

We can use fermats principle and see that when x=a acting as a mirror (purple vertical line) the first two distances ( from origin to point and point to y=4 ) can be minimised for a given x=a

Vice versa with y=b

The equations formed from these two will be that ( using similarity of triangles )

When x=a the sum of first two distances is minimised when $y=$ $\frac{4a}{a+1}$

When y=b the sum of the next two distances (from origin to point, point to x=3) is minimised when $x=$ $\frac{3b}{b+2}$

Solving these two equations with each other ( putting the value of x in a ) , we get $x,y=$ $\frac{5}{3}$ , $\frac{5}{2}$

Putting these values in the equation given gives $2\sqrt{13}$