Rotate triangle ACE 90 degrees clockwise to get triangle DCE',connect the EE' and AD,extended AE intersects DE' at F.We can get the following information:

• The triangle CEE' is an isosceles right triangle.So EE' is going to be equal to $3\sqrt{2}$ .

• $DE'=AE=4$ .

• $AE\perp DE'$ (AE rotates 90 degrees through the triangle ACE to become DE', so they're perpendicular.).

The first solution:Crazy use of the Pythagorean theorem

Let's focus on the triangle DEE'.Let's say FD is $x$ ,so:

${EE'}^2-{E'F}^2={DE}^2-{DF}^2$

$18-(4-x)^2=25-x^2$

$x=\frac{23}{8}$

So DF is equal to $\frac{23}{8}$ , and then we get EF is equal to $\sqrt{{DE}^2-{DF}^2}$ ,which is $\frac{\sqrt{1071}}{8}$ .

So we can figure out that AD is equal to $\sqrt{{AF}^2+{DF}^2}$ , which is $\sqrt{\frac{2624+64\sqrt{1071}}{64}}$ (This number looks terrible).

And we also know that the sum of the squares of the two sides of the square is equal to the square of the diagonal, so the length of the sides of the square is equal to $\frac{1}{\sqrt{2}}$ of the diagonal,which is $AB=\frac{1}{\sqrt{2}}AD$ .

So the side length is

$\sqrt{\frac{2624+64\sqrt{1071}}{128}}\approx\boxed{6.0715}$

The second solution:Cosine formulas

Cosine formula:

$c^2=a^2+b^2-2ab\cos C$

$C=\cos^{-1}(\frac{a^2+b^2-c^2}{2ab})$

In triangle DEE', we get the degree of Angle DE'E by using the second formula:

$\angle DEE'=\cos^{-1}(\frac{2\times3^2+4^2-5^2}{2\times3\sqrt{2}\times4})\approx74.623^{\circ}$

And because the triangle CEE' is a right-angled isosceles triangle, the angle CE'E is equal to 45 degrees, so the angle CE'D is approximately equal to 119.623 degrees.

And then you apply the first formula:

${CD}^2=3^2+4^2-2\times3\times4\times\cos(119.623^{\circ})$

So you compute this, and you get:

$CD=AB\approx\boxed{6.0715}$

$\color{#D61F06}\rm Tip:If\ I\ have\ any\ mistakes\ in\ my\ process, please\ correct\ them\ in\ the\ comments\ section. Thank\ you!$

Let the side length of square $ABCD$ be $a$ , Let the coordinates be $C(0,0)$ , $A(0,a)$ , $D(a,0)$ , and $E(x,y)$ . By Pythagorean theorem , we have:

$\begin{cases} CE: & x^2 + y^2 = 9 & ...(1) \\ AE: & x^2 + (y-a)^2 = 16 & ...(2) \\ DE: & (x-a)^2 + y^2 = 25 & ...(3) \end{cases}$

From $(2)-(1): \ a^2 - 2ay = 7 \implies y = \dfrac {a^2-7}{2a}$ . Similarly, from $(3)-(1): \ \implies x = \dfrac {a^2 - 16}{2a}$ . As $(1): \ x^2 + y^2 = 9$ , we have:

$\begin{aligned} \left(\frac {a^2 - 7}{2a} \right)^2 + \left(\frac {a^2 - 16}{2a} \right)^2 & = 9 \\ 2a^4 - 46 a^2 + 305 & = 36a^2 \\ 2a^4 - 82a^2 + 305 & = 0 \end{aligned}$

$\begin{aligned} \implies a^2 & = \frac {82+\sqrt{82^2-4(2)(305)}}4 \\ \implies a & = \sqrt{\frac {41+3\sqrt{119}}2} \approx \boxed{6.0715} \end{aligned}$

Let $x = AC = CD$ , $\theta = \angle ACE$ , and $y = \cos \theta$ .

By the law of cosines on $\triangle ACE$ , $4^2 = 3^2 + x^2 - 2 \cdot 3 \cdot x \cdot \cos \theta = 9 + x^2 - 6xy$ , which rearranges to $36x^2y^2 = x^4 - 14x^2 + 49$ .

By the law of cosines on $\triangle ECD$ , $5^2 = 3^2 + x^2 - 2 \cdot 3 \cdot x \cdot \cos (90° - \theta) = 9 + x^2 - 6x \sin \theta = 9 + x^2 - 6x \sqrt{1 - \cos^2 \theta} = 9 + x^2 - 6x \sqrt{1 - y^2}$ , which rearranges to $36x^2y^2 = -x^4 + 68x^2 - 256$ .

Therefore, $36x^2y^2 = x^4 - 14x^2 + 49 = -x^4 + 68x^2 - 256$ , so $2x^4 - 82x^2 + 305 = 0$ , which solves to $x = \sqrt{\frac{41 + 3\sqrt{119}}{2}} \approx \boxed{6.0715}$ for $x > 0$ .

Let z be size of square and write the following three relations:

(1). x^2+y^2=3^2

(2). x^2+(z-y)^2=4^2

(3). y^2+(z-x)^2=5^2

Solve with WolframAlpha

Answer z~=6.0715