Left, middle and right are same!!!

Why $20$ socks won't work?

With $20$ socks, it is possible to have $5$ socks of each kind.

Why $21$ socks work?

With $21$ socks, there are always $6$ socks having one same color.

Hence, the smallest number of socks you must take out of the drawer in order to be certain that you have a pair that match is $\boxed{21}$ .

By pigeonhole principle if you'll select 21 socks, there will be atleast 6 socks of same colour.

Whereas if you'll select 20, there can be 5 red, 5 blue, 5 green and 5 yellow socks.

Taking extreme case or the worst possibility, we will have 5 socks of each colour. By that time, we would have drawn 20 socks. But the 21st pick will guarantee another coloured sock. Thus, you will have 6th sock of some colour