Breast fat is one of the reasons, but the influence will not be too great, because the most important structures in the breast are glands, ducts, connective tissue and fat, and the largest part of them is glands. Losing weight will not affect them, and the chest will not be completely lost because of losing weight. It is safe to lose weight.