No, Stephen, you are neither a demon, ghost, nor barbarian.
In reality, since we have a thread in H&S on it, is really a belittling racist term, used to nonchalantly imply that whites are uncivilized, strange, and yes, evil. It literally means, "his old man is a demon"!
Internally, I still bristle when I hear, especially those who act like know-it-alls or the hoity-toity, Chinese use the term.
Of course, to many, the full effect of the meaning is lost even if they understand it, and treat it as a joke or casual phrase, like "cuz", or "brotha".
A correct and proper, therefore alien sounding to most, would be "bak yun", or white man. I've only heard my wife use it normally. Other Cantonese speakers I know use "quilo" to mean the same and have forgotten the original offensive intent of the phrase.
I'll just call you Cattle Baron.