I can tell the difference, but I am not a true audiophile because if it sounds bad, I couldn't even begin to tell you whhat to do to make it sound better. The only thing I know is that someone once told me that if you've got two inputs (?) say you're playing a computer through a stereo, have the volume on the first one (the computer) turned up to maximum, and adjust the volume using the second one (the stereo). I don't know if this is true though.
I suspect it might be a lot like wine. It's not really worth spending loads of money you can't afford for everyday drinking and a lot of people can't tell the difference at first between a £7 bottle of wine and a £30 bottle. But some people can, and in the case of wine at least, £3 bottles almost invariably taste grim. And, like wine, what you get in pubs and clubs is often pretty shoddy.