[all variants] How to stop linux-image-*-generic interfering with VirtualBox Guest Additions?

I have a VirtualBox VM ruing Xubuntu 16.04 LTS amd64 as the guest.
The keel packages for Ubuntu 16.04, for instance linux-image-4.4.0-28-generic and linux-image-extra-4.4.0-28-generic, include keel modules that are part of VirtualBox Guest Additions. These "stray" keel modules are interfering significantly with the ability to install and use Guest Additions.

Because they are a special Ubuntu version, installing Guest Additions from the VirtualBox ISO does not work properly (if at all - installing a Guest Additions version older than the keel modules version, such as 5.0.12, doesn't even allowed). This means I don't get to have 3D acceleration inside the VM. So I tried installing from the Ubuntu package repositories, which worked initially, but broke upon apt-get dist-upgrade due to dkms detecting the stray modules.

Apparently, after purging the Ubuntu package Guest Additions, the Guest Additions version 5.0.20 could be installed fine from the ISO, but it asked to confirm removing leftover parts of a 3rd-party install or something like that, and I said yes. (Looks like it deleted the modules from linux-image-*-generic packages.)
Guest Additions seems to work fine at this point, without all the glitching and application-crashing I saw before, but I'm not sure how stable this "good" setup is and I somehow doubt that it will stay good after the next dist-upgrade...

I'm not happy about just deleting the stray modules and reboot, because the next keel upgrade will include them again. And unless I'm misunderstanding something about the nature of updates to 16.04 LTS, I need to keep the system fully updated irregularly, so ignoring this and leaving the keel(s) as-is / not updated won't work either.

So how to "permanently" disable or get rid of these stray modules in such a way that Guest Additions installed from the ISO won't also be disabled or otherwise impaired?

Thanks