Hi there!
It sounds to me like he hasnt had a lot of desensitizing training done on him, either that or he was abused... either way the bottom line is: he is scared.
First of all, get someone who is somewhat experienced to help you with this. Im sure you have heard of people tying bags to the end of a lunge whip and waving it in front of horses, etc. etc.
Great technique... but people usually go about it wrong.
First off, you need to find a rhythm with the whip and wave it in the same place at the same speed to the side of your horse until he relaxes. Have someone hold him while you stand next to them (in front, off to the right) and start gently waving the whip/flag. Up and down, nice and easy, a good couple feet away so he can see it. If he moves away keep going with him until he stops and relaxes. AS SOON AS HE STANDS STILL AND YOU SEE EVEN A GLIMMER OF RELAXATION GET THE FLAG AWAY FROM HIM. And pet him. A LOT.
Then repeat going a little faster and a little closer. Each time he stops and relaxes, GET THE FLAG AWAY AND PET HIM. That is his reward for relaxing. The "pressure" goes away. This is the most important part of the exercise.
This is the first step. Let me know how this goes and we can go from here.
Oh yeah, and dont talk to him while you are waving the flag. Thats a distraction and it can screw up voice commands. For example, dont sit there telling him to whoa if he decides to move away from the flag. It is ineffective and "untraining" the whoa.
Hope this is helpful!!! Good luck!!!