Removing Rubber Stains on Refrigerator?

I used a magnet to hold up a calendar on my refrigerator. The magnet had rubber tips that have now left yellow stains on my refrigerator. How do I remove the stains?