It's normal and good to lose weight when you lose weight during fitness.
The reason why your muscles don't grow too much is because the purpose of your exercise may be to burn fat, and your weight or load is not so great that you can't fully stimulate muscle growth. Or you have a fitness plan now and your body has adapted. It's time to increase weight and strength-