Thats a good start, no config file. Then cutting out modules from Xorg and adding them one by one could give you some additional ideas.
If you try another desktop and/or display manager (ex.gdm) it will eliminate any problems with KDE, kdm.
LXDE has its own display manager. You can try that.