I haven’t read Theory of Moral Sentiments yet, but Freakonomics did a fantastic podcast series on Adam Smith. They spend a lot of time on the Theory of Moral Sentiments and how what he actually wrote doesn’t match the current “perception” of him.
I know it’s not the popular opinion, but I agree with your position in your first paragraph. I think the context does make a difference.
The chocolate pudding line is definitely weird, but also definitely racist.