Rectangular pegs in a round hole .....

By symmetry the three rectangles to be found can be assumed to form a symmetric ‘cross’ as shown.

Imgur

It can be entirely defined by a single angle (theta) shown.

Instead of three non overlapping rectangles we can take them to be two identical but overlapping rectangles. Their dimensions and area can be seen to be $2R \sin(\theta) \times 2R \cos(\theta) = 2 R^{2} \sin(2 \theta)$ while their overlapping area would be $4 R^2 \sin^{2}(\theta)$ Thus the area of the cross = $2R^{2}sin(2\theta) - 4R^{2}sin^{2}(\theta)$ Differentiating and equating it to zero we get, $2 \cos(2\theta) = \sin(2\theta)$ that is, $\tan(2\theta) = 2$ Giving $\theta = 31.72°$ and the fraction of circle’s area covered by the cross = 0.787