At the present time the kernel will have the choice between panicking or shooting down tasks.
There's a paper by Jochen in a recent HotOS suggesting a way the kernel could reclaim memory from apps.
Cristan has been working on a proposal for non-persistant mappings. Ie when you run out of memory you dump some mappings and free the memory they use. You'd have to ask Cris about the state of his proposal but he's been exploring the idea. The idea is that the kernel would flush a mapping and then a standard page fault would occur and your pager would map it back if required. Of course we have been discusing pinning of mappings and non-policy bound mecanisms for doing this.
cheers Adam