You didn't mention in your post whether your horse was a stallion or gelding. If a stallion then that may be your major problem. All stallions I have had any contact with are dominant unless there is a stronger dominant in the herd.

Otherwise I would have to agree with Urban Warrior that he is just bored and frustrated from lack of exercise and interaction. Another consideration here is rather you are feeding him enough?