Can a model be repositioned (e.g. by Nihilation Vortex) beyond the play area?
I would say no, because repositioning can only happen on the battle field (which is presumably synonymous to the play area?):
A model may be repositioned on the battle field […]
(The Thousand Worlds, p. 199)
How do you play it?
As far as I know you cannot intentionally or unintentionally leave the battlefield off the edges for any reason.
Well you can be slammed off the play area:
BEYOND THE PLAY AREA
If an effect would cause a model to move or be placed beyond the table edge (such as being slammed), the model is destroyed.
(The Thousand Worlds, p. 204)
I believe specifically reposition can’t happen off the board side, but slams (such as with velocity projector) or places (can’t recall an unfriendly placement) would destroy the model
Agree here, except I don’t think place exists in Warcaster
Reposition is the same thing, and can’t be used to put someone off the table. Slams can though. Nemesis can get both effects, use a displacer shot to set them up, then slam them off with the appropriately named obliterator 
REPOSITIONING MODELS A model may be repositioned on the battlefield as a result of squad movement or a special rule. Repositioning is not movement, and anything that affects or would be affected by a model’s movement does not apply. When you are repositioning a model, it must be repositioned to a location it can stand on or otherwise occupy. For example, you cannot place the model on an obstacle that is not wide enough for it to stand on or reposition it in midair over the table.
I am only aware of friendly places in the game, e.g. through Recurrent Apotheosis or Psychoclasm: A model replaces a different model, by first removing the replaced, and then placing the replacing model.
I don’t remember what the rule name is but there’s drags (e.g. the ISA Headsman). Ofc you can’t drag a model out of bounds so not an issue here.
I suppose you could use a friendly place as a means of resetting a squad