What's happening is that right around 96 to 99 users the system dbus seems to no longer accept GDM connections and more users cannot log in. Very often during this threshold we also see dbus chewing lots of CPU as it seems to be receiving retries over and over again which all fail. /var/log/messages show gdm crashing and to my eyes it's happening when it's trying to talk to dbus. I have installed the debug symbols and should get better backtraces starting tomorrow. The shot below shows the crash
The technique that I used was to look at the source code and find where the defaults are set and then try and double them. There are more parameters, any ideas which ones might help? It would be wonderful if dbus-monitor showed you these types of failures, but it seems to only show you bus activity which really doesn't help. As soon as we drop below the 96 users, everything works correctly and users can log on and off with no problems. This leads me to strongly believe this is a parameter that is being reached.
Any tips? Drop a comment. Thanks!
6 comments:
The available limits are all listed in "man dbus-daemon" (unless someone added one without adding it there, but probably they are all there). session.conf sets most of them explicitly; but your problem is probably with the system bus (system.conf) which uses the hardcoded defaults in dbus/bus/config-parser.c. If you copy the very high limits from session.conf you would surely solve the problem, though you might create a theoretical local DOS security problem, that may not be a concern in your environment.
@Robert;
Perhaps our man page is old, but the limits are not listed in there. So I looked over the source again and found the system.conf that is in the build system but on OpenSuse was not carried into the dbus config directory. I brought over all the settings and we will reboot and test tomorrow. It would be really helpful if dbus had some spewage in dbus-monitor that indicated thresholds that are being hit. Fingers crossed for tomorrow.
We designed DBUS for a one user system not for such a big deployment, you are on your own buddy, or you can pay me to do the improvement..
@Anonymous this is a main problem. If anybody builds a Linux based terminal server, this conception is dies. I think you must review your point of view.
Anonymous, thats awesome. I will add into my fortune quotes.
@All:
Good new, it's a known issue and Vincent the rescue.
https://bugzilla.novell.com/show_bug.cgi?id=739743
What happens is that dbus on OpenSuse 11.4 runs out of FDs at 1024 which is about 96 users. Will test a new RPM when it's created.
Post a Comment