From a scientific perspective, you don't go anywhere.
Normally, when you are asleep, your body stops feeling sensations from the outside world. This includes the sense of your own body. During an OBE, this happens while you are still awake. So instead of feeling your body, your brain creates a mental model of your body - like a dream self - and the area around you. This mental model and your real self are slightly out of whack so it feels like you have left your body.
You can actually cause people to have OBEs by stimulating a certain part of their brains.
For more information, see
The Science of Out of Body Experiences
Of course, that is only a physical explanation. Only you can know where you go in your mind (in your dream world). When you say that you only feel the re-entry, do you mean that you feel like you have been outside your body but you don't remember actually being outside your body?
Here is another perspective that might interest you:
Astral Projection
I think the advice here on how to relax into an OBE might make you feel less panicky when you come out of it and possibly more likely to remember what happened in your mind before.