Octagon in an Octagon

Let the side length of octagon $ABCDEFGH$ be $x$ . Then $A_L=2x^2 (1+\sqrt2)$ .

Consider $\triangle ABH$ : Let the length of line $BH$ be $y$ - this is the length of a side of the squares $ACEG$ and $BDFH$ . Using the cosine rule, we have $y^2=x^2+x^2-2x^2 \cos135° \Rightarrow y^2= 2x^2(1 + \sqrt2/2)$ .

Now let the side length of the smaller octagon be $z$ . Then $z(1+\sqrt2)=y$ and $A_S=2z^2 (1+\sqrt2)$ .

$\Rightarrow \dfrac{A_S}{A_L}=\dfrac{2z^2 (1+\sqrt2)}{2x^2 (1+\sqrt2)}=\dfrac{z^2}{x^2}=\dfrac{y^2}{x^2 (1+\sqrt2)^2}=\dfrac{2x^2(1 + \sqrt2/2)}{x^2 (1+\sqrt2)^2}$ $=\dfrac{2(1 + \sqrt2/2)}{(1+\sqrt2)^2} =\dfrac{2+\sqrt2}{3+2\sqrt2} = 2- \sqrt2$

Thus $a=b=2$ , and the required answer is $\large \color{#20A900}{\boxed{4}}$ .

One can easily prove that the ratio of side of smaller octagon to that of larger octagon is

$=\frac{1}{\sqrt2\cos\frac\pi8}$

we know that the ratio of areas of similar regular polygons is equal to the ratio of squares of their respective sides hence

$\frac{A_S}{A_L}=\left(\frac{1}{\sqrt2\cos\frac\pi8}\right)^2=\frac{\sec^2\frac\pi8}{2}=\frac{1+\tan^2\frac\pi8}{2}=\frac{1+(\sqrt2-1)^2}{2}=\frac{4-2\sqrt2}{2}=2-\sqrt2$

Let x the side of larger octagon, the square side y, y^2 = x^2(2 +√2), Let z the side of smaller octagon, (1+√2)z = y, z^2 = y^2/(3 +2√2), By similarity AS/AL= z^2 / x^2 = (2 +√2)/ (3 +2√2) = 2 - √2 = a - √b, a+b=4