As best as I can tell, and it’s a bit of a stretch, the implication may be at “my mans” stems from AVE, and therefore asking if people talk like that translates to being racist.

I don’t agree with the thought process or conclusion, but that’s the best I got