And I'm not saying you're wrong. What I'm saying is, this is why that line of reasoning has, thus far, failed to produce harmony and peace between the varying factions. Because some of the factions believe it is not your right to do this thing they disagree with.
Of course, given their druthers, they likely don't believe it's your right to do a lot of other things they disagree with, like blaspheme or let gays get married or do business on a Sunday (another of the commandments that modern Christians seem to have mislaid--honoring the Sabbath and keeping it holy).