VirtualBox and VMware don't "support" virtualisation of OS X on non-OS X platforms. You either have to hack your VM, or the guest installer.

"hack" is a bit strong.

You can install MacOS on current virtual box on Windows with only a text file modification.

The issue is that the MacOS license only grants permission to use it in VMs on Apple hardware.

Can a vanilla OS X DMG / ISO be installed with this text file modification? (last time I looked, you had to replace one of VirtualBox's DLLs on Windows, for example)

Yes (though I misspoke - it's changing VirtualBox settings, rather than text file mod)

http://www.insanelymac.com/forum/topic/309654-run-vanilla-os...

Starts with an installer from the Mac App Store to make an install DVD, or a physical Snow Leopard DVD, no DLL replacement or kernel modifications.

I'd like to know how to do this too.

Sorry, but I can choose "Mac OS X" right from the "machine type" (It then lists a number of OS X versions from Snow Leopard up to El Capitan) when creating a new machine on VirtualBox 5.x on Linux, and I'm fairly sure it was the same on 4.x too. How is this "unsupported"? Don't know about VMware though.

Edit: I must add that there are no Guest Additions available for OS X, so if you go on and create the machine, it won't be able to benefit from decent graphics integration, shared clipboard, drag & drop support - but this is IMHO far from OS X not being supported.