With melatonin, different people need different amounts. You might try starting out with just the 1 mg size and see if you still have the dreams. I also sometimes have very vivid dreams, and an occasional nightmare; but not any different than when i do not take melatonin.
Also, if you have not been sleeping well before you started taking the melatonin, your brain might not have been able to do the "de-fragging" that it is supposed to do at night; and if you keep taking the melatonin, then it should get all of those bad dreams over with, and you will sleep much better.
I also take a cal/mag/zinc at night when I take the melatonin. Both the calcium and the magnesium will help your body relax and sleep better; and try not to watch any scary shows on television before going to bed.
I like to relax with a good book, or just knit or crochet for a while before bedtime, and that seems to help my brain get settled down for the night.