He used sexual/anatomical imagery in his comment. That you are not offended is irrelevant to the issue, because Adria was offended. That's all that matters as far as that portion of the issue is concerned.
I'm perfectly comfortable with sex jokes made by any gendered person. But it was asked not to happen at this conference, and everyone who attended agreed to the CoC.
Grading the subjective offensiveness of the comment is beside the point.
They agreed not to do something, and to me and many others, we don't think it was in violation of the rules they agreed to.
Naturally this would vary from person-to-person. My opinion on the offensiveness (or lack of) is certainly relevant. That's what happens when you have vague, ambiguous terms of service that require emotions for a decision. Every person who heard the comment could have interpreted it differently.
Judging the words coming out of peoples mouths is hard, especially when you don't understand the jargon or the industry.
I do appreciate that, but I find direct adherence to a set of rules (for events of this nature) often completely bypasses the point of them. It's intended to infuse a sense of equanimity and mindfulness rather than be something against which you can be punished. Or at least that's my perspective of it.
The CoC was pretty clear to refrain from offensive language, swearing, and the like. Thus, it was inappropriate by the conference's standards.
[1]: https://us.pycon.org/2013/about/code-of-conduct/