The color sample of AI is different from that of PS, but all used patterns and colors will be added to the color sample. In fact, not only that, but also one more brush and symbol is used. Even if you delete the copied ones, they still exist.

You can open the corresponding palette, then clear the unused ones, and you can slim down your AI file!