Server session disappearing due to signal?

Community Support
1

I have been looking at thinlinc to connect to a VM with XFCE and it works. It’s just that randomly after seemingly longer intervals (think days) the server side session just gets killed. Until that time, connecting and disconnecting repeatedly is always fine.

xinit.log for the disappearing session at the end just shows without any timestamp:

tl-xinit: Caught signal, terminating
tl-xinit: Terminating X client...
tl-xinit: Terminating X server...

[...]

Is there a way to make this log verbose to figure out which signal appears to be killing the session and when this happened? tl-xinit seems to be sensitive to something that did not ever affect x2go for example and I have a hard time narrowing this down. What could it be?

Thank you!

2

Hi @hwrobel,

Good question - the only time ThinLinc should be ending sessions unattended is if you’ve set, for example, MaxIdleTime or MaxDisconnectionTime. See:

If you haven’t set these, then presumably something at OS-level is ending the session for you.

Which distribution are you running on? Is there anything in the system logs to indicate what might be happening?

3

This is on RHEL 8.10. It appears that tlwebadm wants info every 5 mins and that something or someone is triggering tlwebadm at about the time ~03:10 of the last unexpected session kill with much more activity suddenly, according to journalctl.

Jul 19 03:00:01 [...].com systemd[1]: Starting Collection of license statistics for ThinLinc Web Administration...
Jul 19 03:00:01 [...].com systemd[1]: Starting system activity accounting tool...
Jul 19 03:00:01 [...].com systemd[1]: sysstat-collect.service: Succeeded.
Jul 19 03:00:01 [...].com systemd[1]: Started system activity accounting tool.
Jul 19 03:00:01 [...].com systemd[1]: tl-license-stats.service: Succeeded.
Jul 19 03:00:01 [...].com systemd[1]: Started Collection of license statistics for ThinLinc Web Administration.
[...]
Jul 19 03:05:04 [...].com systemd[1]: Starting Collection of license statistics for ThinLinc Web Administration...
Jul 19 03:05:04 [...].com systemd[1]: tl-license-stats.service: Succeeded.
Jul 19 03:05:04 [...].com systemd[1]: Started Collection of license statistics for ThinLinc Web Administration.
[...]
Jul 19 03:10:01 [...].com systemd[1]: Starting Collection of license statistics for ThinLinc Web Administration...
Jul 19 03:10:01 [...].com systemd[1]: Starting system activity accounting tool...
Jul 19 03:10:01 [...].com systemd[1]: Stopping ThinLinc Web Administration...
Jul 19 03:10:01 [...].com systemd[1]: sysstat-collect.service: Succeeded.
Jul 19 03:10:01 [...].com systemd[1]: Started system activity accounting tool.
Jul 19 03:10:01 [...].com systemd[1]: tl-license-stats.service: Succeeded.
Jul 19 03:10:01 [...].com systemd[1]: Started Collection of license statistics for ThinLinc Web Administration.
[...]
Jul 19 03:10:05 [...].com systemd[1]: tlwebadm.service: Succeeded.
Jul 19 03:10:05 [...].com systemd[1]: Stopped ThinLinc Web Administration.
Jul 19 03:10:05 [...].com systemd[1]: Starting ThinLinc Web Administration...
Jul 19 03:10:06 [...].com systemd[1]: Started ThinLinc Web Administration.
Jul 19 03:10:06 [...].com systemd[1]: Stopping ThinLinc VSM agent...
Jul 19 03:10:06 [...].com systemd[1]: home-[...]-.gvfs.mount: Succeeded.
Jul 19 03:10:06 [...].com systemd[1]: var-opt-thinlinc-sessions-[...]-10-drives-l.mount: Succeeded.
Jul 19 03:10:08 [...].com systemd[1]: vsmagent.service: Succeeded.
Jul 19 03:10:08 [...].com systemd[1]: Stopped ThinLinc VSM agent.
Jul 19 03:10:08 [...].com systemd[1]: Starting ThinLinc VSM agent...
Jul 19 03:10:08 [...].com systemd[1]: Started ThinLinc VSM agent.
Jul 19 03:10:08 [...].com systemd[1]: Stopping ThinLinc VSM server...
Jul 19 03:10:09 [...].com systemd[1]: vsmserver.service: Succeeded.
Jul 19 03:10:09 [...].com systemd[1]: Stopped ThinLinc VSM server.
Jul 19 03:10:09 [...].com systemd[1]: Starting ThinLinc VSM server...
Jul 19 03:10:09 [...].com systemd[1]: Started ThinLinc VSM server.
Jul 19 03:10:09 [...].com systemd[1]: Stopping ThinLinc Web Access...
Jul 19 03:10:12 [...].com systemd[1]: tlwebaccess.service: Succeeded.
Jul 19 03:10:12 [...].com systemd[1]: Stopped ThinLinc Web Access.
Jul 19 03:10:12 [...].com systemd[1]: Starting ThinLinc Web Access...
Jul 19 03:10:12 [...].com systemd[1]: Started ThinLinc Web Access.

Given that no one should have the password, I am not using that tool, and that 3AM is not a good time to maintain a system, in theory tlwebadm shouldn’t be able to kill the session. The tlwebadm.log didn’t show anything that would suggest any specific action or any login attempts.

Nevertheless, I have now used systemctl to stop and disable the tlwebadm service. Now I’ll wait for a number of days to see if the current sessions gets killed again.

Thanks for nudging me into this direction.

4

The web administration service will update the license count every 5 minutes, that’s normal behaviour. The other system services restarting like that is a bit strange, though. Is it possible that you have some cron job or automated script which is doing this every few days?

5

I may have found the topic with some help from our local IT guys.

Our systems to a weekly logrotate.

The vmsagent configuration states like the other thinlinc service related logfiles:

/var/log/vsmagent.log {
missingok

postrotate
   /bin/systemctl try-restart vsmagent >/dev/null 2>&1
endscript

}

Restarting vsmagent kills any currently active session.

The question now is: Do we really need to restart vmsagent or the other services or is there a benign way to have log rotation working?

6

You should be able to restart vsmagent safely without affecting existing sessions. You can test this manually if you like, by restarting vsmagent in the same way at the command line with a session running. Restarting vsmagent as part of a log rotation is correct, and required.

Are there any other automated scripts or anything which might be the culprit?

7

Sounds like the PAM configuration might have been damaged on this system, causing user sessions to no longer be independent of the vsmagent service.

I would have a look in /etc/pam.d and make sure that pam_systemd is still being used. Which file to look in depends on the distribution used on the system.

8

A “sudo systemctl try-restart vsmagent” will kill the current session reproducibly.

9

I can confirm that there is no XDG_SESSION_ID variable set currently, i.e., pam_systemd is not active as it seems. I will take this up with our IT as they manage that part.

Thank you for the pointer about pam_systemd being necessary for vsmagent to work as expected! This is very non-obvious, and I’d never have found it.

10

All, as it is not always easy to effect a change in configurations managed by corporate IT, I would like to describe a workaround that appears to resolve the issue well enough.

By default, thinlinc appears to create /etc/pam.d/thinlinc as link to sshd in the same directory, i.e., the thinlinc service should treated equivalent to sshd.

Given that in this case my system configuration is missing pam_systemd, I removed that “thinlinc” link and replaced it with a “thinlinc” file of the following content:

auth		include sshd
account		include sshd
password	include sshd
session		include sshd
-session     optional      pam_systemd.so

This appears to do the same thing with respect to sshd equivalence, but also ensures that pam_systemd is started if thinlinc is invoked.

Maybe this is a general idea useful for making the system more robust if the underlying config isn’t well known.