You don't need to quit color for fitness-in fact, whether you exercise or not, you can't overdo it. In fact, the relationship between the two is not significant unless there is a pathological change in neurological function.

However, color abstinence must be based on fitness or other intensive exercises. It's simple. Exercise consumes androgen (but whether it will permanently increase the level of stimulating hormone is another matter) and can balance the body's response to the outside world.