Safepoint — let the whole world wait

To perform some operations (not just garbage collection) the JVM needs to completely stop application threads. Why does a virtual machine need stop-the-world pauses? How does a Java program stop and what side effects does this lead to? What safepoint related optimizations does HotSpot JVM do? How to find the reason for long stops? What developers should remember to avoid pauses?