Hi,
I have trouble with one of my ThinLinc nodes after restarting our physical servers that host them. Before the reboot, everything worked fine and users were able to login to both nodes. After the reboot, users are only able to connect to the first node (that is also the master) but all users that are delegated to the second node (by the master due to load-balancing) will receive a timeout message in their ThinLinc client.
The setup is the following:
- two physical servers (head01 and head02)
- each runs a LXC container (desk01 and desk02)
- desk01 is the master, desk02 is an agent
- head01 and head02 have external IPs for clients to connect over the internet
- desk01 and desk02 both have IP 10.10.1.2 for the container network and their respective heads have 10.10.1.1 and are configured as the gateway in the /etc/network/interfaces of the desks
Whenever a user is delegated to desk01, everything works. If they are delegated to desk02, the following happens in the /var/log/vsmserver.log of the master:
2023-04-03 09:48:43 INFO vsmserver.session: User with uid 1337 (dummy.user) requested a new session
2023-04-03 09:48:45 INFO vsmserver.session: Session desk02.domain.de:12 created for user dummy.user
and in the master’s systemd journal:
Apr 03 09:48:43 desk01 sshd(pam_google_auth)[127001]: Accepted google_authenticator for dummy.user
Apr 03 09:48:43 desk01 sshd[126999]: Accepted keyboard-interactive/pam for dummy.user from 10.0.80.254 port 51039 ssh2
Apr 03 09:48:43 desk01 sshd[126999]: pam_unix(sshd:session): session opened for user dummy.user by (uid=0)
Apr 03 09:48:43 desk01 systemd[1]: Created slice User Slice of UID 1337.
Apr 03 09:48:43 desk01 systemd[1]: Starting User Runtime Directory /run/user/1337...
Apr 03 09:48:43 desk01 systemd-logind[119]: New session 5140 of user dummy.user.
Apr 03 09:48:43 desk01 udisksd[2028]: Error statting none: No such file or directory
Apr 03 09:48:43 desk01 systemd[1]: Started User Runtime Directory /run/user/1337.
Apr 03 09:48:43 desk01 systemd[1]: Starting User Manager for UID 1337...
Apr 03 09:48:43 desk01 systemd[127119]: pam_unix(systemd-user:session): session opened for user dummy.user by (uid=0)
Apr 03 09:48:43 desk01 systemd[127119]: Listening on GnuPG cryptographic agent (ssh-agent emulation).
Apr 03 09:48:43 desk01 systemd[127119]: Listening on GnuPG cryptographic agent and passphrase cache (access for web browsers).
Apr 03 09:48:43 desk01 systemd[127119]: Listening on GnuPG network certificate management daemon.
Apr 03 09:48:43 desk01 systemd[127119]: Reached target Timers.
Apr 03 09:48:43 desk01 systemd[127119]: Listening on Sound System.
Apr 03 09:48:43 desk01 systemd[127119]: Reached target Paths.
Apr 03 09:48:43 desk01 systemd[127119]: Listening on GnuPG cryptographic agent and passphrase cache.
Apr 03 09:48:43 desk01 systemd[127119]: Starting D-Bus User Message Bus Socket.
Apr 03 09:48:43 desk01 systemd[127119]: Listening on GnuPG cryptographic agent and passphrase cache (restricted).
Apr 03 09:48:43 desk01 systemd[127119]: Listening on D-Bus User Message Bus Socket.
Apr 03 09:48:43 desk01 systemd[127119]: Reached target Sockets.
Apr 03 09:48:43 desk01 systemd[127119]: Reached target Basic System.
Apr 03 09:48:43 desk01 systemd[127119]: Reached target Default.
Apr 03 09:48:43 desk01 systemd[127119]: Startup finished in 55ms.
Apr 03 09:48:43 desk01 systemd[1]: Started User Manager for UID 1337.
Apr 03 09:48:43 desk01 systemd[1]: Started Session 5140 of user dummy.user.
Apr 03 09:48:45 desk01 sshd[126999]: pam_unix(sshd:session): session closed for user dummy.user
Apr 03 09:48:45 desk01 systemd-logind[119]: Session 5140 logged out. Waiting for processes to exit.
Apr 03 09:48:45 desk01 systemd[1]: session-5140.scope: Succeeded.
Apr 03 09:48:45 desk01 systemd-logind[119]: Removed session 5140.
Apr 03 09:48:56 desk01 systemd[1]: Stopping User Manager for UID 1337...
Apr 03 09:48:56 desk01 systemd[127119]: Stopped target Default.
Apr 03 09:48:56 desk01 systemd[127119]: Stopped target Basic System.
Apr 03 09:48:56 desk01 systemd[127119]: Stopped target Timers.
Apr 03 09:48:56 desk01 systemd[127119]: Stopped target Paths.
Apr 03 09:48:56 desk01 systemd[127119]: Stopped target Sockets.
Apr 03 09:48:56 desk01 systemd[127119]: dbus.socket: Succeeded.
Apr 03 09:48:56 desk01 systemd[127119]: Closed D-Bus User Message Bus Socket.
Apr 03 09:48:56 desk01 systemd[127119]: gpg-agent-browser.socket: Succeeded.
Apr 03 09:48:56 desk01 systemd[127119]: Closed GnuPG cryptographic agent and passphrase cache (access for web browsers).
Apr 03 09:48:56 desk01 systemd[127119]: gpg-agent-ssh.socket: Succeeded.
Apr 03 09:48:56 desk01 systemd[127119]: Closed GnuPG cryptographic agent (ssh-agent emulation).
Apr 03 09:48:56 desk01 systemd[127119]: pulseaudio.socket: Succeeded.
Apr 03 09:48:56 desk01 systemd[127119]: Closed Sound System.
Apr 03 09:48:56 desk01 systemd[127119]: gpg-agent.socket: Succeeded.
Apr 03 09:48:56 desk01 systemd[127119]: Closed GnuPG cryptographic agent and passphrase cache.
Apr 03 09:48:56 desk01 systemd[127119]: dirmngr.socket: Succeeded.
Apr 03 09:48:56 desk01 systemd[127119]: Closed GnuPG network certificate management daemon.
Apr 03 09:48:56 desk01 systemd[127119]: gpg-agent-extra.socket: Succeeded.
Apr 03 09:48:56 desk01 systemd[127119]: Closed GnuPG cryptographic agent and passphrase cache (restricted).
Apr 03 09:48:56 desk01 systemd[127119]: Reached target Shutdown.
Apr 03 09:48:56 desk01 systemd[127119]: systemd-exit.service: Succeeded.
Apr 03 09:48:56 desk01 systemd[127119]: Started Exit the Session.
Apr 03 09:48:56 desk01 systemd[127119]: Reached target Exit the Session.
Apr 03 09:48:56 desk01 systemd[127120]: pam_unix(systemd-user:session): session closed for user dummy.user
Apr 03 09:48:56 desk01 systemd[1]: user@1337.service: Succeeded.
Apr 03 09:48:56 desk01 systemd[1]: Stopped User Manager for UID 1337.
Apr 03 09:48:56 desk01 systemd[1]: Stopping User Runtime Directory /run/user/1337...
Apr 03 09:48:56 desk01 systemd[64381]: run-user-1337.mount: Succeeded.
Apr 03 09:48:56 desk01 systemd[36782]: run-user-1337.mount: Succeeded.
Apr 03 09:48:56 desk01 udisksd[2028]: Error statting none: No such file or directory
Apr 03 09:48:56 desk01 systemd[2406]: run-user-1337.mount: Succeeded.
Apr 03 09:48:56 desk01 systemd[5866]: run-user-1337.mount: Succeeded.
Apr 03 09:48:56 desk01 systemd[1409]: run-user-1337.mount: Succeeded.
Apr 03 09:48:56 desk01 systemd[3508]: run-user-1337.mount: Succeeded.
Apr 03 09:48:56 desk01 systemd[130168]: run-user-1337.mount: Succeeded.
Apr 03 09:48:56 desk01 systemd[102376]: run-user-1337.mount: Succeeded.
Apr 03 09:48:56 desk01 systemd[4421]: run-user-1337.mount: Succeeded.
Apr 03 09:48:56 desk01 systemd[1]: run-user-1337.mount: Succeeded.
Apr 03 09:48:56 desk01 systemd[763]: run-user-1337.mount: Succeeded.
Apr 03 09:48:56 desk01 systemd[76771]: run-user-1337.mount: Succeeded.
Apr 03 09:48:56 desk01 systemd[27814]: run-user-1337.mount: Succeeded.
Apr 03 09:48:56 desk01 systemd[18704]: run-user-1337.mount: Succeeded.
Apr 03 09:48:56 desk01 systemd[69966]: run-user-1337.mount: Succeeded.
Apr 03 09:48:56 desk01 systemd[1]: user-runtime-dir@1337.service: Succeeded.
Apr 03 09:48:56 desk01 systemd[1]: Stopped User Runtime Directory /run/user/1337.
Apr 03 09:48:56 desk01 systemd[1]: Removed slice User Slice of UID 1337.
Looks like everything is okay here. At the same time on desk02 (the agent not working) the following happens in the systemd journal:
Apr 03 09:48:43 desk02 tl-session[37245]: pam_unix(thinlinc:session): session opened for user dummy.user by (uid=0)
Apr 03 09:48:43 desk02 systemd-logind[103]: New session 228 of user dummy.user.
Apr 03 09:48:43 desk02 systemd[1]: Started Session 228 of user dummy.user.
Apr 03 09:48:50 desk02 dbus-daemon[5193]: [session uid=1337 pid=5193] Activating service name='org.kde.kglobalaccel' requested by ':1.86' (uid=1337 pid=37390 comm="kded5 [kdeinit5] ")
Apr 03 09:48:50 desk02 dbus-daemon[5193]: [session uid=1337 pid=5193] Successfully activated service 'org.kde.kglobalaccel'
Apr 03 09:48:50 desk02 org.kde.kglobalaccel[5193]: qt.qpa.xcb: QXcbConnection: XCB error: 3 (BadWindow), sequence: 176, resource id: 4194443, major code: 20 (GetProperty), minor code: 0
Apr 03 09:48:50 desk02 dbus-daemon[5193]: [session uid=1337 pid=5193] Activating service name='org.kde.KScreen' requested by ':1.86' (uid=1337 pid=37390 comm="kded5 [kdeinit5] ")
Apr 03 09:48:50 desk02 dbus-daemon[5193]: [session uid=1337 pid=5193] Successfully activated service 'org.kde.KScreen'
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: qt.qpa.xcb: QXcbConnection: XCB error: 3 (BadWindow), sequence: 176, resource id: 4194443, major code: 20 (GetProperty), minor code: 0
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xrandr: Connected output 60 to CRTC 59
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xcb.helper: Detected XRandR 1.6
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xcb.helper: Event Base: 92
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xcb.helper: Event Error: 145
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xrandr: XRandR::setConfig
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xrandr: Requested screen size is QSize(1920, 1080)
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xrandr: Needed CRTCs: 1
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xrandr: Actions to perform:
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xrandr: Primary Output: true
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xrandr: Old: 0
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xrandr: New: 60
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xrandr: Change Screen Size: false
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xrandr: Disable outputs: false
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xrandr: Change outputs: false
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xrandr: Enable outputs: false
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xrandr: RRSetOutputPrimary
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xrandr: New primary: 60
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xrandr: XRandR::setConfig done!
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xcb.helper: RRNotify_OutputChange
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xcb.helper: Output: 60
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xcb.helper: CRTC: 59
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xcb.helper: Mode: 62
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xcb.helper: Rotation: "Rotate_0"
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xcb.helper: Connection: "Connected"
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xcb.helper: Subpixel Order: 0
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xcb.helper: RRScreenChangeNotify
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xcb.helper: Window: 29360133
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xcb.helper: Root: 252
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xcb.helper: Rotation: "Rotate_0"
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xcb.helper: Size ID: 0
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xcb.helper: Size: 1920 1080
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xcb.helper: SizeMM: 508 286
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xcb.helper: RRNotify_OutputChange
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xcb.helper: Output: 60
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xcb.helper: CRTC: 59
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xcb.helper: Mode: 62
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xcb.helper: Rotation: "Rotate_0"
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xcb.helper: Connection: "Connected"
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xcb.helper: Subpixel Order: 0
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xrandr: XRandROutput 60 update
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xrandr: m_connected: 0
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xrandr: m_crtc XRandRCrtc(0x55f6c6063850)
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xrandr: CRTC: 59
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xrandr: MODE: 62
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xrandr: Connection: 0
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xrandr: Primary: true
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xrandr: Output 60 : connected = true , enabled = true
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xrandr: XRandROutput 60 update
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xrandr: m_connected: 0
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xrandr: m_crtc XRandRCrtc(0x55f6c6063850)
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xrandr: CRTC: 59
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xrandr: MODE: 62
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xrandr: Connection: 0
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xrandr: Primary: true
Apr 03 09:48:50 desk02 org.kde.KScreen[5193]: kscreen.xrandr: Output 60 : connected = true , enabled = true
Apr 03 09:48:50 desk02 dbus-daemon[5193]: [session uid=1337 pid=5193] Activating service name='org.kde.ActivityManager' requested by ':1.99' (uid=1337 pid=37432 comm="/usr/bin/plasmashell ")
Apr 03 09:48:50 desk02 polkitd(authority=local)[192]: Registered Authentication Agent for unix-session:228 (system bus name :1.337 [/usr/lib/x86_64-linux-gnu/libexec/polkit-kde-authentication-agent-1], object path /org/kde/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8)
Apr 03 09:48:50 desk02 org.kde.ActivityManager[5193]: qt.qpa.xcb: QXcbConnection: XCB error: 3 (BadWindow), sequence: 176, resource id: 4194443, major code: 20 (GetProperty), minor code: 0
Apr 03 09:48:50 desk02 dbus-daemon[5193]: [session uid=1337 pid=5193] Successfully activated service 'org.kde.ActivityManager'
Apr 03 09:48:50 desk02 org.kde.ActivityManager[5193]: qt.qpa.xcb: QXcbConnection: XCB error: 3 (BadWindow), sequence: 176, resource id: 4194443, major code: 20 (GetProperty), minor code: 0
Apr 03 09:48:50 desk02 org.kde.ActivityManager[5193]: Service started, version: 7.0.0
Apr 03 09:48:51 desk02 dbus-daemon[85]: [system] Activating service name='org.kde.powerdevil.discretegpuhelper' requested by ':1.339' (uid=1337 pid=37466 comm="/usr/lib/x86_64-linux-gnu/libexec/org_kde_powerdev") (using servicehelper)
Apr 03 09:48:51 desk02 org.kde.powerdevil.discretegpuhelper[37662]: QDBusArgument: read from a write-only object
Apr 03 09:48:51 desk02 org.kde.powerdevil.discretegpuhelper[37662]: QDBusArgument: read from a write-only object
Apr 03 09:48:51 desk02 org.kde.powerdevil.discretegpuhelper[37662]: QDBusArgument: read from a write-only object
Apr 03 09:48:51 desk02 dbus-daemon[85]: [system] Successfully activated service 'org.kde.powerdevil.discretegpuhelper'
Apr 03 09:48:51 desk02 dbus-daemon[85]: [system] Activating service name='org.kde.powerdevil.backlighthelper' requested by ':1.339' (uid=1337 pid=37466 comm="/usr/lib/x86_64-linux-gnu/libexec/org_kde_powerdev") (using servicehelper)
Apr 03 09:48:51 desk02 org.kde.powerdevil.backlighthelper[85]: powerdevil: no kernel backlight interface found
Apr 03 09:48:51 desk02 org.kde.powerdevil.backlighthelper[37667]: QDBusArgument: read from a write-only object
Apr 03 09:48:51 desk02 org.kde.powerdevil.backlighthelper[37667]: QDBusArgument: read from a write-only object
Apr 03 09:48:51 desk02 org.kde.powerdevil.backlighthelper[37667]: QDBusArgument: read from a write-only object
Apr 03 09:48:51 desk02 dbus-daemon[85]: [system] Successfully activated service 'org.kde.powerdevil.backlighthelper'
Apr 03 09:48:51 desk02 org.kde.KScreen[5193]: kscreen.xrandr: Emitting configChanged()
Apr 03 09:48:51 desk02 dbus-daemon[85]: [system] Activating via systemd: service name='org.bluez' unit='dbus-org.bluez.service' requested by ':1.334' (uid=1337 pid=37390 comm="kded5 [kdeinit5] ")
Apr 03 09:48:51 desk02 systemd[1]: Condition check resulted in Bluetooth service being skipped.
Apr 03 09:48:52 desk02 dbus-daemon[5193]: [session uid=1337 pid=5193] Activating service name='org.kde.kuiserver' requested by ':1.99' (uid=1337 pid=37432 comm="/usr/bin/plasmashell ")
Apr 03 09:48:52 desk02 dbus-daemon[5193]: [session uid=1337 pid=5193] Successfully activated service 'org.kde.kuiserver'
Apr 03 09:49:16 desk02 dbus-daemon[85]: [system] Failed to activate service 'org.bluez': timed out (service_start_timeout=25000ms)
Also looks okay to me. The /var/log/vsmagent.log on desk02 also looks promising:
2023-04-03 09:48:43 DEBUG vsmagent: Got encrypted vnc password for dummy.user
2023-04-03 09:48:43 DEBUG vsmagent: Got encrypted shadower vnc password for dummy.user
2023-04-03 09:48:43 DEBUG vsmagent: Cannot find font path element /usr/share/X11/fonts/misc, skipping
2023-04-03 09:48:43 DEBUG vsmagent: Cannot find font path element /usr/share/X11/fonts/75dpi, skipping
2023-04-03 09:48:43 DEBUG vsmagent: Cannot find font path element /usr/share/X11/fonts/100dpi, skipping
2023-04-03 09:48:43 DEBUG vsmagent: Cannot find font path element /usr/share/X11/fonts/Type1, skipping
2023-04-03 09:48:43 DEBUG vsmagent: Cannot find font path element /usr/share/X11/fonts/TTF, skipping
2023-04-03 09:48:43 DEBUG vsmagent: Cannot find font path element /usr/share/fonts/default/Type1, skipping
2023-04-03 09:48:43 DEBUG vsmagent: Cannot find font path element /usr/share/fonts/misc, skipping
2023-04-03 09:48:43 DEBUG vsmagent: Cannot find font path element /usr/share/fonts/75dpi, skipping
2023-04-03 09:48:43 DEBUG vsmagent: Cannot find font path element /usr/share/fonts/100dpi, skipping
2023-04-03 09:48:43 DEBUG vsmagent: Cannot find font path element /usr/share/fonts/Type1, skipping
2023-04-03 09:48:43 DEBUG vsmagent: Cannot find font path element /usr/share/fonts/URW, skipping
2023-04-03 09:48:43 DEBUG vsmagent: Cannot find font path element /usr/share/fonts/Speedo, skipping
2023-04-03 09:48:43 DEBUG vsmagent: Cannot find font path element /usr/share/fonts/cyrillic, skipping
2023-04-03 09:48:43 DEBUG vsmagent: Cannot find font path element /usr/share/fonts/misc, skipping
2023-04-03 09:48:45 INFO vsmagent.session: Verified connectivity to newly started Xvnc for dummy.user
2023-04-03 09:48:45 DEBUG vsmagent: Handling connection from ('10.0.81.11', 1023)
2023-04-03 09:48:45 DEBUG vsmagent.extproc: Starting process ['ss', '-n', '-t', '-a', '-p', '(', 'sport', '>=', ':5020', 'and', 'sport', '<', ':5026', ')', 'or', '(', 'dport', '>=', ':5020', 'and', 'dport', '<', ':5026', ')']
2023-04-03 09:48:45 DEBUG vsmagent.extproc: ['ss', '-n', '-t', '-a', '-p', '(', 'sport', '>=', ':5020', 'and', 'sport', '<', ':5026', ')', 'or', '(', 'dport', '>=', ':5020', 'and', 'dport', '<', ':5026', ')'] exited with exit code 0
continued in next post…