You must care about sex workers, because you are not doing work of an entirely different genus. Their interests are yours. Their rights are yours. And if you restrict their rights and speech, you better believe it'll impact you. Think about who rich people find obscene. It's everyone who isn't them.