I would try something like veggy oil on a terry cloth towel. Depending on how soft the rubber is, the paint should crack up a little if some pressure is applied. The oil may help break it down without eating at the rubber. Once you get it off, just use soap and water on the rubber, and hit with a light coat of dielectric grease.
Mineral spirits may be ok too.... not sure. It would probably work faster on the paint, I just don't know what it will do to the rubber. I would think it wouldn't be to harsh for it. Just use the soap and water afterwards to nuetralize the M spirits. Use something like the DE grease or something to condition the rubber afterwards though.