Relevant wiki: Dot Product - Problem Solving

Let $(x_1,y_1),\ (x_2, y_2),\ (x_3,y_3)$ be three successive points of such an $n$ -gon. The length and direction of the sides are described by the vectors $\vec v_1 = (x_2-x_1,y_2-y_1);\ \vec v_2 = (x_3-x_2,y_3-y_2).$ Take the dot product : $\vec v_1\cdot \vec v_2 = (x_2-x_1)(x_3-x_2)+(y_2-y_1)(y_3-y_2),$ which is an integer. This product is equal to $\ell^2\cos \theta$ , where $\theta = 2\pi/n$ is the angle over which one turns each vertex, and $\ell$ is the length of each side. Now $\cos \frac{2\pi}n = \frac{\vec v_1\cdot \vec v_2}{\ell^2} = \frac{(x_2-x_1)(x_3-x_2)+(y_2-y_1)(y_3-y_2)}{(x_2-x_1)^2+(y_2-y_1)^2}$ is a rational number . However, for $n > 4$ the cosine $\cos 2\pi/n$ is irrational . Hence $n \leq \boxed{4}$ .

This is the same as finding rational values for $Tan\left(\frac{2\pi}{n} \right)$ where $n$ is an integer. Only the square makes it.

"The acute angles in a right triangle with rational side lengths are never rational multiples of $\pi$ "

Edit: Okay, for the more general case of any regular polygon of $n$ sides having its vertices of integer coordinates anywhere in an integer lattice, we note that:

1) If all the vertices have integer or rational coordinates, then by the Shoelace Theorem, the area of the polygon is rational
2) If all the vertices of the regular n-polygon have rational coordinates, then its center has rational coordinates (which is the arithmetic average of all of the vertices)
3) Likewise, the midpoints between adjacent vertices have rational coordinates
4) The distances between the center and any of the vertices, between the center and any of the midpoints, and the between any vertex with the nearest midpoint can be expressed as $\sqrt{a}$ , where $a$ is some rational number
5) The area of the right triangle formed by the center, any one of the vertices, and any of its nearest midpoints can be expressed as follows

Area= $\dfrac { 1 }{ 2 } rSin\left( \dfrac { 1 }{ 2 } \dfrac { 2\pi }{ n } \right) rCos\left( \dfrac { 1 }{ 2 } \dfrac { 2\pi }{ n } \right) =\dfrac { 1 }{ 4 } { r }^{ 2 }Sin\left( \dfrac { 2\pi }{ n } \right)$

6) Since $r$ can be expressed as the square root of a rational number, $\;{r}^{2}$ is a rational number
7) Hence, in order for the area to be rational, the expression

$Sin \left( \dfrac { 2\pi }{ n } \right)$

has to be rational
8) The only non-trivial integer values for $n$ which yields such rational numbers is $n=4$ , which corresponds with a square

This solution is wrong as pointed out by Arjen below.

---

Assume that without the loss of generality the regular $n$ -gon is centered at the origin with radius $r$ , then it can be rotated such that it has a vertex at $(0,r)$ . Moving anticlockwise, the next vertex is at $r(\cos \theta, \sin \theta)$ , where $\theta = 2\pi /n$ . Taking just the $x$ components of these vertices, for the polygon to have integer components, both $r$ and $\cos \left( \frac{2\pi}{n} \right)$ must be integers.

For this to be true, $\cos\theta$ must be rational. Using Niven's theorem, the only rational values of $\cos(n\pi)$ are $0, \pm 1, \pm \frac12$ , and the corresponding $\theta$ values are $180^\circ, 90^\circ , 60^\circ$ . Since an internal angle of $60^\circ$ would give an irrational $y$ component (following the same argument as above, a regular $n$ -gon with integer vertices must have an internal angle of $180^\circ$ or $90^\circ$ , or a regular $\boxed{4}$ -gon or a trivial $2$ -gon.

Here's a neat and easy solution.

It's obviously not possible for pentagon or hexagon. (For example, it's easy to explicitly calculate 5th and 6th complex root of 1 and see that at least one component is irrational.)

Suppose it's possible for n-gon for n>6. Let's then take the smallest possible n-gon - the shortest possible side, length a, so that all vertices have integer coordinates. On each edge we define first and second point by going counterclockwise.

Now we just move all edges so that their first points coincide in point O (0,0). The second points of edges in new positions now form vertices of another n-gon, inscribed in a circle with centre O and radius a, and vertices still have integer coordinates. With hexagon the side would be the same as radius. But for bigger n the side is smaller.

So we found an n-gon with vertices with integer coordinates and the side <a. But that's a contradiction, as we initially choose an n-gon with the shortest possible side, the length of which we called a. QED