I've spent a LOT of time studying user interfaces and it was all well a mystery until I met up with a few folks who where versed in the "science" of cognitive psychology. I put science in quotes because, as a self-appointed armchair mathematician it is only with much resistance that I can admit that any field of psychology is, in fact, a scientific discipline.
After a quite a few conversations with folks who are versed in this field, as well as many hours of listening to lectures on introductory psychology (and more to come) from MIT http://ocw.mit.edu/OcwWeb/Brain-and-Cognitive-Sciences/index.htm I now see with blinding clarity the disarray of the state of things in user interface design.
Problem number one is that most, if not all, people who design user interfaces have an opinion about "what the user will think" and frankly I've now gotten to the point where that actually IRRITATES me. I think this is largely because the transference of "me" to "the user" doesn't take into account the idea that developers/managers/Salespeople are not always the target users of a system.
This means that when "me" the "me" looks at a feature, I develop UIs that are easy to develop (if I'm a developer), easy to manage the development of (if I'm a manager), and easy to sell (if I'm a salesperson). This leaves those other folks... you know... the one's who are probably paying the bills by purchasing your product, at the mercy of a bunch of folks with wildly different objectives, none of which are necessarily in line with their own.