The whole time I was growing up I can remember my grandpa's ranch having cats in the barn. They had a few that were tame and rubbed on your legs but most of them were wild. I guess they had about 10 cats or kittens running around at any given time. If my grandma found where they had the kittens then some of them would end up tame. But if they had kittens in the loft or something then noone knew about them till they grew and by then they were wild. I'm sure they acquired some strays from the around the country side that they other cats brought home or attracted.
So, if the cats were wild then you wouldn't get so attached to them and it wouldn't be so bad if they "went away".