It happens if you have the guest additions installed.
What this thing does is, when you resize your Virtualbox window, it will resize the resolution of the guest, which is nice and what you want. Otherwise you'd have to manually change the guest resolution to make it match the window size, which would be extremely annoying.
Unfortunately, for some reason, on my linux PC it was constantly flickering and resizing itself becoming smaller over time.
Took me a while but I finally figured out why - it was because my host display was zoomed in at 115% whereas my virtualbox display setting was at 100%.
I changed my host display zoom back to 100% and that solved the issue.
