Question about retainers?

Alright so I havent worn my retainer for a while. I put it on and it was super tight, but after a few nights of wearing it, it fits again. I noticed my teeth have shifted. Am I screwed or if I keep wearing my retainer they will fix back to normal?