Think you can Geometry?

Looking at all the connections from A to other vertices of the polygon, there are two types of lines.

(1) The diameter AB, which contributes 100 to the sum of squares.

(2) All the other lines, which appear in pairs. AC goes to a point some distance from A, AD goes to a point the same distance from point B. This second line can be imagined in location CB instead of AD. Triangle ACB is a right triangle and $\overline{AC}^2+\overline{CB}^2=\overline{AB}^2=100$

Since the sum of squares is $800$ , and $100$ is taken up by the diameter, there are $7$ of these pairs, that is 14 additional vertices beyond A and B.

There is a total of $\boxed{16}$ vertices.

Imgur

Taking (0,5) as the center of the inscribing circle, the vertices $V_{i} = A, B, C . . . C', B'$ with i = 1, 2, 3, . . . n will be given by - $(x_{i},y_{i}) = \left( 5 \sin \left( \frac{2\pi i}{n} \right), 5 - 5 \cos \left( \frac{2\pi i}{n} \right) \right)$ Now the square of the distance $d_{i}$ of $V_{i}$ from vertex A (which is also the origin) will be given by - $d_{i}^2 = 25 \sin^2 \left( \frac{2\pi i}{n} \right) + 25 - 50 \cos \left( \frac{2\pi i}{n} \right) + 25 \cos^2 \left( \frac{2\pi i}{n} \right)$ $= 50 - 50 \cos \left( \frac{2\pi i}{n} \right)$ When we take the sum, the cosine terms for B & B', C & C' etc. will pairwise cancel by symmetry and we will be left with $50 \times n = 800$ giving $n = \boxed{16}$

We note that the sum of squares of the distances from vertex $A$ to all other vertices of the regular $2n$ -sided polygon $S$ , where $n = 1,2,3...$ , is given as below:

$S=10^{2}+2\sum _{ i=1 }^{ n-1 }{ { \left( 5\times 2\sin { \frac { i\pi }{ 2n } } \right) }^{ 2 } }$ $\Rightarrow S=100+200\sum _{ i=1 }^{ n-1 }{ \sin ^{ 2 }{ \frac { i\pi }{ 2n } } }$

Using the formula, it is found that when:

$2n=4 \Rightarrow S=200$ $2n=6 \Rightarrow S=300$ $2n=8 \Rightarrow S=400$ $\Rightarrow S=100n$ $...$ $2n=\boxed{16} \Rightarrow S=800$

Let $x_k=5e^{\dfrac{2 \pi k i}{n}}$ the vertice $k$ of the n-gon. Then, we will find a formula for the distance from $x_0$ to any vertice $x_m$ : $d=|x_m-x_0|=5|\cos \left(\dfrac{2 \pi m}{n}\right) + i \sin\left(\dfrac{2 \pi m}{n}\right) - 1|$ $=5\sqrt{\left(\cos \left(\dfrac{2 \pi m}{n}\right)-1\right)^2+\sin^2\left(\dfrac{2 \pi m}{n}\right)}$ $=5\sqrt{2-2\cos\left(\dfrac{2 \pi m}{n}\right)}$ $d=10\sin\left(\dfrac{\pi m}{n}\right)$ Now, we need to solve: $\sum_{m=1}^{n-1} 100 \sin^2\left(\dfrac{\pi m}{n}\right)=800$ $\sum_{m=1}^{n-1} \sin^2\left(\dfrac{\pi m}{n}\right)=8$ We use a formula to delete that sum: $\dfrac{n}{2}=8 \Rightarrow n=\boxed{16}$