Cat Losing Fur?

My cat is five years old and has lost fur on both of his back legs and hip area on both sides. There are no sores or irritated skin, just no fur. He had been eating Purina Naturals for three years and then we switched to Blue Brand. He gets monthly flea treatments and otherwise seems normal and active. I am assuming this is a skin problem or allergy? The vet said it was anxiety, which he has occasionally when my husband leaves, they are very attached. But the weird thing is we have never seen him bite that area.