A strange octagon

$$ See the picture below. Octagon in Square $$ Using $\Delta\text{MRC}$ : $$ $\begin{aligned} \overrightarrow{MR}&=\overrightarrow{RC}\tan 30^\circ\\ y&=\frac{3}{2}\sqrt{3}\cdot\frac{1}{\sqrt{3}}\\ &=\frac{3}{2}. \end{aligned}$ $$ Using $\Delta\text{CQL}$ : $$ $\begin{aligned} \overrightarrow{LQ}&=\overrightarrow{QC}\tan 30^\circ\\ x&=(3\sqrt{3}-x)\cdot\frac{1}{\sqrt{3}}\\ x\sqrt{3}+x&=3\sqrt{3}\\ x&=\frac{3\sqrt{3}}{\sqrt{3}+1}\\ &=\frac{9-3\sqrt{3}}{2}. \end{aligned}$ $$ Then, we calculate length $\overrightarrow{LN}$ . $$ $\begin{aligned} \overrightarrow{LN}&=\overrightarrow{BC}-2\overrightarrow{BQ}\\ z&=3\sqrt{3}-2x\\ z&=3\sqrt{3}-2\left(\frac{9-3\sqrt{3}}{2}\right)\\ &=6\sqrt{3}-9. \end{aligned}$ $$ Therefore, area of the polygon $\text{IJKLMNOP}$ is equal to the sum of the area square $\text{JLNP}$ and four times area of the triangle $\text{LMN}$ . $$ $\begin{aligned} [\text{IJKLMNOP}]&=[\text{JLNP}]+4[\text{LMN}]\\ &=z^2+4\cdot\frac{1}{2}z(x-y)\\ &=z(z+2(x-y))\\ &=(6\sqrt{3}-9)(6\sqrt{3}-9+2\left(\frac{9-3\sqrt{3}}{2}-\frac{3}{2}\right)\\ &=(6\sqrt{3}-9)(6\sqrt{3}-9+9-3\sqrt{3}-3)\\ &=(6\sqrt{3}-9)(3\sqrt{3}-3)\\ &=81-45\sqrt{3}. \end{aligned}$ $$ Thus, $a+b=81+(-45)=\boxed{36}$ . $$

Consider a coordinate system where the origin is the centre of the square. Find the equations of the following lines accordingly.

On solving them, you'll get O $(\frac{9-3 \sqrt{3}}{2 \sqrt{3}} , 0)$ , N $(\frac{3 \sqrt{3} (\sqrt{3} -1)}{2( \sqrt{3} +1)}, \frac{3 \sqrt{3} (1- \sqrt{3} )}{2( \sqrt{3} +1)})$ and M $(0, \frac{3 \sqrt{3}-9}{2 \sqrt{3}})$ . Due to symmetry, take the mirror image of N and M in the first quadrant as P $(\frac{3 \sqrt{3} (\sqrt{3} -1)}{2( \sqrt{3} +1)}, \frac{3 \sqrt{3} ( \sqrt{3} -1 )}{2( \sqrt{3} +1)})$ and I $(0, \frac{9-3 \sqrt{3}}{2 \sqrt{3}})$ .

Now, find the area of the shaded part for the first quadrant.

$\frac{1}{4} A= (ZR \times RP)+(\frac{1}{2} RP \times RO)+(\frac{1}{2} WI \times WP) =(ZR \times RP)+(RP \times RO) =RP(ZR+RO) = (RP \times ZO)$ $\implies \frac{1}{4} A = \frac{9 \sqrt{3}}{4} (3 \sqrt{3} -5) \implies A= 81- 45 \sqrt{3}$

So, $81-45= \boxed{36}$

The octagon is big square $s^2 = 27$ minus 4 times yellow quadrangle, see geogebra illustration .

illustration

Yellow quadrangle is sum of two similar blue triangles. Similarity coefficient of two blue triangles is easily calculated by dividing their black heights: $\frac{\frac s2 - \frac{s \sqrt 3}4}{\frac{s \sqrt 3}4} = \frac{2 - \sqrt 3}{\sqrt 3}.$ Bigger blue triangle is half of equilateral triangle: $\frac{s^2 \sqrt 3}8$ . Adding smaller one gives their sum $\frac{s^2 \sqrt 3}8 \bigg( 1 + \frac{(2-\sqrt 3)^2}3 \bigg) = \frac{s^2 \sqrt 3}4 \cdot \frac{5-2\sqrt 3}3.$ 4 times yellow quadrangle is then $s^2 \sqrt 3 \cdot \frac{5-2\sqrt 3}3 = s^2 \cdot \frac{5 \sqrt 3 - 6}3.$ The area of octagon is $s^2 - s^2 \cdot \frac{5 \sqrt 3 - 6}3 = 9 \cdot (9 - 5 \sqrt 3) = a + b \sqrt 3.$ The solution is $a+b = 9 \cdot (9-5) = 36$ .

$[IJKLMNOP]=\frac{[AHB]}{3}-[PHN]$ $\frac{HP}{HA}=\frac{1}{2+\sqrt3}=>\frac{[PHN]}{[AHB]}=(\frac{1}{2+\sqrt3})^2$ $=>[IJKLMNOP]=[PHN] \times (\frac{1}{3}-\frac{1}{7+4\sqrt3})$ $=>[IJKLMNOP]=\frac{9\sqrt3(1+\sqrt3)}{7+4\sqrt3}=a+b\sqrt3$ $=>7a+12b=27, 4a+7b=9 => a=81, b=-45 => a+b=\boxed{36}.$

Let X be the centre of the square ABCD.
$\triangle FXK \equiv \triangle GXI$ ( $\angle KFX=30^\circ$ )
Let $FX=x$ , then $KX=FX/\sqrt{3} =\frac{x}{\sqrt{3}}$
$FG=\frac{x}{\sqrt{2}}$
$JY=FY \cdot \tan \angle JFY = FG/2 \tan15^\circ = \frac{ x \sqrt{2} } { 2 ( 2 - \sqrt{3} ) }$ .

AREA(XKJI) = 2Area(FXK) + Area(FJG) - Area(FXG) = 2∙x^2/(2√3) + 1/2 ∙(x√2)/2(2-√3)∙ x√2 - x^2/2 =
=x^2 [√3/3+1/2(2-√3)-1/2]=27/4(4-2√3)(1/2-√3/6)=1/4(81-45√3)
Area(Octagon)= 4∙ AREA(XKJI)= 81-45√3→a+b=36