They don't compete against men because of hormonal differences which mean that generally, men have more muscle mass and advantageous muscle composition. The science shows that those advantages dwindle pretty quickly once hormone replacement kicks in. I'm assuming here that we're talking about trans-women who have undergone a medical transition rather than all trans-women.
Male athletes like the ones I mentioned with the same build but less muscle/less efficient muscle would be at a significant competitive disadvantage in women's sport if they transitioned. An analogy I've heard used is like having a big car and putting an engine designed for a smaller car in. It won't go as fast or be as maneuveorable as a smaller car with the same engine would because it's lugging around a bunch more dead weight.
Obviously in some sports having a bigger frame is an advantage regardless but in many sports, especially low-contact, high mobility or endurance sports, lugging around a larger bone structure with smaller and less efficient muscles will slow you down more than it helps you. Ultimately I think blanket rules are daft either way, decisions should be on a sport by sport basis.
