Drunk hexagon

Label the center $O$ . Due to the symmetry, the center of the hexagon will be the center of the square. If only 1 diameter of the hexagon is touching the sides of the square, then we could rotate it slightly and expand it, to get a larger hexagon. Hence, 2 diameters of the hexagon must be touching the square.

Split the square up into 4 squares, and the hexagon into 6 equilateral triangles. Consider one of the small squares which contains an equilateral triangle, which has 2 vertices on the perimeter of the original square, and the last vertex is $O$ .

The largest possible side length of this equilateral triangle occurs when it is symmetric about the diagonal of the square from vertex $O$ . This gives us that the side length of the triangle is $\frac{ 1}{2 \cos 15} = \frac{ 2} { \sqrt{6} + \sqrt{2} } = \frac{ \sqrt{6} - \sqrt{2} } { 2}$ .

Thus, the diameter of the hexagon is $\sqrt{6} - \sqrt{2}$ hence $a^2 + b^2 = 6^2 + 2^2 = 40$ .

$\text{Largest lines in the square and the hexagonal are their diagonals.} \\\text{So the largest hexagon is when the two diagonal coincide.} \\ \text{This is shown in one fourth of the sketch above. } \\ \text{Half diagonal 'df ' of the hexagon} \\ \text{coincides with half the diagonal 'cf' of the square.} \\ \text{Half hexagon side 'bg' =x/2 has 'b' on the square half} \\ \text{side 'ac'. 'bd'=x is the hexagon side. From the sketch:-}\\\text{Using the properties of the square and hexagon we have:-}\\p=xSin60^o=ySin45^o ~~~~~~~~~~~~\implies~ \dfrac {\sqrt 3 } 2 x =\dfrac 1 {\sqrt 2 }y\\\therefore~~bc=y=\sqrt{\dfrac 3 2 }x~~~~~~~~~~~~and~~~~~~~~~~~~ ab=\sqrt 2*\dfrac x 2.\\ac=\dfrac 1 2 =ab+bc=(\sqrt 2*\dfrac 1 2 +\sqrt{\dfrac 3 2 })*x.~~~~\therefore~x=\dfrac{\sqrt 2}{\sqrt 3 +1}.\\But~ the~ hexagon~diagonal~=2*df=2x=2*\dfrac{\sqrt 2}{\sqrt 3 +1}.\\=\sqrt 6- \sqrt 2=\sqrt a - \sqrt b. \therefore~a^2+b^2= ~~~~~~~~ \color{#D61F06}{\Large {40} }$