Challenges with Smart Card Setup on New ThinLinc Install

[Bill]

Preface: We studied and used the technique on this other post: How to set up authentication in ThinLinc with PKCS #15 smart cards

The presumption there is that the smart card has a file system on it. When we go to view the certificates, we do not see any. So, the process breaks.

Note: the post’s length is to respect your time by including pertinent details, due to the time we’ve spent on this issue.

Problem Summary
We are missing a key step in getting a filesystem on the smart cards. The smart card vendor tool does not provide a filesystem that works pkcs15 apparently. We’ve locked two smart cards in the process of creating useful certificates for ThinLinc.

Ask
If you worked with smart card authentication and could provide the missing step(s) or some insight, I think we can get this solution promoted to a POC. We are very close, but have not been able to get the smart card functionality to work (particulars in the Background section below).

Environment

• New VMware RHEL9 image with the ThinLinc server
• HP thin clients with the ThinLinc client installed for front-end access
• User ID and PW authentication works fine (from the HP thin client units)
• The customer requires smart card reader and smart card authentication access to handle authentication (very busy medical practice)
• Aventra smart cards
  The vendor claims their cards work with Cendio’s solution. Aventra provided input, but their software is not providing a filesystem that we can use, or we are unknowingly making a mistake. Aventra does not offer pre-configured smart cards.

Background
The excellent documentation offered on the Cendio site and forum plus even with third-party sites create the perception that using Smart Cards with SSH keys all appear to indicate that the keys are already stored on the card.

• We have not seen any reference to how to create and store a key on a Smart Card
• Performing a pkcs15-tool --list-certificates command, we see the card reader
  and the card when inserted, but there are no certificates and no ID value preset
• Even if we create a pin on the card, we are still not seeing a certificate (which seems correct, but offering it to be complete)
• Result: we are missing a key step in the process that others seem to know well

I’m a server and network guy, so some of these desktop issues make little sense to.
Appreciate any insight you can offer. We want to get our first Cendio installation successfully deployed.

[martin]

Hello @Bill

If those cards from Aventra are blank cards, you’d have to initialize them yourself (Aventra MyEID PKI card · OpenSC/OpenSC Wiki · GitHub) and generate a key-pair on the card. Then you can create a certificate request and sign it, using the private key stored on the card.

I’ve found a few results for this on Google,
http://cedric.dufour.name/blah/IT/SmartCardsHowto.html

I am by far no expert in this field either, but I believe the sections named
“PKCS#15”, “Key pairs and (X.509) certificates”, “Smart card preparation finalization”, and “OpenSSL” are of interest for you.

I’ve also found this script that should format the card and load it with a single key and certificate.

#!/bin/bash
#
# Initialize PKCS#15 card with self signed certificate
#

# XXX: set these!
PIN=1234
PUK=5678
BITS=2048

set -e
set -x

# Clear out any existing contents on the card
pkcs15-init --erase-card
# Create the basic structure (don't bother setting a SO PIN/PUK)
pkcs15-init --create-pkcs15 --so-pin "" --so-puk ""
# Create a user PIN/PUK
pkcs15-init --store-pin --auth-id 01 --label "authentication" --pin ${PIN} --puk ${PUK}
# Generate a key pair, and protect it using the user PIN
pkcs15-init --generate-key rsa/${BITS} --auth-id 01 --id 45

# Now we need a certificate, which first mean a certificate request

workdir=`mktemp -d` 
remove_workdir ()
{
    rm -rf "${workdir}"
}
trap remove_workdir 1 2 3 9 15 EXIT

config="${workdir}/openssl.cnf"
cat <<"EOF" > "${config}"
    openssl_conf = openssl_def

    [openssl_def]
    engines = engine_section

    [engine_section]
    pkcs11 = pkcs11_section

    [pkcs11_section]
    MODULE_PATH = opensc-pkcs11.so
    PIN = @PIN@

    [ req ]
    prompt = no
    distinguished_name = req_dn
    x509_extensions     = cert_exts
    string_mask = utf8only
    utf8 = yes

    [ req_dn ]
    C = SE
    commonName = Björn Testsson
    SN = Testsson
    givenName = Björn
    serialNumber = 739423794532

    [ cert_exts ]
    subjectKeyIdentifier = hash
    basicConstraints = CA:false
    keyUsage = keyCertSign,digitalSignature,keyEncipherment
EOF
sed -i "s/@PIN@/${PIN}/" "${config}"

# Generate a certificate request, and a certificate from card's key
cert="${workdir}/cert.pem"
openssl req -engine pkcs11 -new -key slot_0 -keyform engine -x509 -out "${cert}" -text -config "${config}"

# Store the certificate we just created
pkcs15-init --store-certificate "${cert}" --auth-id 01 --id 45 --format pem

# When done creating PIN codes, finalize (activate) the card. After this all access conditions (PINs) are in effect
pkcs15-init -F

Change values accordingly and see if this is a good fit for your use-case.

Kind regards,
Martin

[Bill]

@martin

#!/bin/bash
#

Hello, Martin. The information is clear and we will try these suggestions. I am flying to meet the client this weekend and will be on-site Tuesday. I’ll update you accordingly. Thank you.

[Bill]

Hi @martin Thank you again for the suggestions. My previous trip proved unsuccessful, but today, I am back at the client’s and now have a proper key structure on the card. Next is to fine-tune the process, then configure the ThinLinc Server side for the key exchange.

[martin]

Hello again @Bill

Glad to hear from you again!
Did the information about generate the key-pair in my previous post work for you, or did you manage to solve it some other way? Please enlighten us!

Kind regards,
Martin

[Bill]

Hi @martin - Sure! I’d be happy to write up the lessons learned, some tips plus the commands used to get it working. Happy to make a contribution here. Yes, the sections including key pairs were helpful.

As Aaron is away this week, he suggested posting my query here. Would you (or someone else in the community) be able to help with two comprehension questions? This seems like the last part of the process.

First, can you clarify what Cendio implies by the “subject name on the card”? The RSA key’s CSR omits a subject value, so I am uncertain to what you are referring in the Automatic Connection section?
https://www.cendio.com/resources/docs/tag/authentication_scpublickey.html?highlight=smart%20card#smartcard-auto

Second, on the LDAP Automatic Update (tl-ldap-certalias) piece, I do not understand how LDAP fits into the process, if I have to collect manually the RSA keys from the card and place them in each user’s home directory?
https://www.cendio.com/resources/docs/tag/authentication_scpublickey.html?highlight=smart%20card#smartcard-auto

We’re all excited to start testing this and appreciate the last part of guidance as we are so close.

Kindest regards,
–Bill

[martin]

Hi @Bill

This is not something I work with regularly, so my knowledge is somewhat limited, but I’ll try my best

Regarding your first question:
The X.509 Certificates on the card (for example, slot 45) contains the user certificate stored on the card. From this slot we can read out the ssh public key, and also the subject name. This is an example from a demo card I use:

maros: ~ $ pkcs15-tool --read-certificate  45 | openssl x509 -text -noout | grep "Subject:"
Using reader with a card: OMNIKEY AG 3121 USB 00 00
        Subject: C = SE, CN = Fredrika Testsson21, SN = Testsson21, GN = Fredrika, serialNumber = 189104124129

From this output, we can see that the Subject Name (CN) is Fredrika Testsson21. To have the ThinLinc client to automatically connect when the smartcard is inserted, one must enable
Options → Security → Hit “Details” for Smart Card and check “Use certificate subject as login name”

Which will then use this CN as login user name:

Since this name is rarely a valid user name, one must use nss-passwdalias module to map the CN’s to valid posix usernames.

This is required if one wants to implement automatic login when the smartcard is inserted.

Secondly:
tl-ldap-certalias/LDAP is not a requirement at all. In modern systems, looking up users from different backends (ldap, AD etc.) is usually done with sssd. Of course, you will still need some method of collecting the public keys from the smart cards (i.e. pkcs15-tool --read-ssh-key ) and store it on server, in ~/user/.ssh/authorized_keys

I think a common use-case for tl-ldap-certalias is when the user certificate from the smart card is stored in a central ldap database, and you want to automatically fetch the ssh key from that ldap server and store each key in ~/user/.ssh/authorized_keys.

Kind regards,

[Bill]

Hi Martin - Very helpful. Quick question - on the passwdaliases file, does it matter which components of the smart card’s subject I use, as long as they match and the correct UID is appended? If it does matter, then let me know what it is parsing, so I can ensure that data is in the file. Seeing different examples in the forum and wanted to rule out an error I received (posted separately this morning).

Kindest regards,

–Bill

[martin]

Hello @Bill,

Yes, you need to include the full subject. To illustrate, my journal log reports:

sshd[30762]: Connection closed by invalid user serialNumber=189104124129,givenName=Fredrika,sn=Testsson21,cn=Fredrika Testsson21,c=SE 10.47.1.40 port 38630 [preauth]

I then edited /etc/passwdaliases and added:

serialNumber=189104124129,givenName=Fredrika,sn=Testsson21,cn=Fredrika Testsson21,c=SE:1000

After that, together with adjusted pam configuration, login works as expected.

[Bill]

Thank you, Martin. My subject looks very different than yours, but attempted multiple times to work with it (Example below). In the end, still seeing the same error.

(1) When I inserted the smart card into the reader and monitored the ssh journalctl entries, I see on the thin client side, “You are not authorized to connect to this server” error dialog box every test, with a different error in the server’s sshd log entries in the journal, based upon the changes made.

I’ve spent the past seven hours researching online for different cures to the various error messages received, but nothing seems to resolve the error. I checked and changed the permissions and ownership correct on ~, ~/.ssh, ~/.ssh/*
different times and monitored the error messages. I’m back to your site’s suggestions: chmod 700 on .ssh and 600 on ~/.ssh/authorized_keys

My current server-side log error is this:
Connection closed by authenticating user 1.2.840.113549.1.9.1=#16246b656e6e6574682e6275726b654077696c64776f6f6470656469617472693732e636f6d,cn=Test User,o=Test Company,L=Fairfield,st=Maryland,c=US 172.20.1.166 port 52154 [preauth]

Again, when I add the full subject, the log error remains the same. I even created a DER file and extracted the subject from there. That subject is identical to your method above. That is good that they are the same, but tried another way to see if there was any difference. More troubleshooting steps.

Uncertain if my log entries beginning with a numerical string matters in the /etc/passwdaliases file?

If I preface the subject’s beginning numerical string with “serialNumber=” in the /etc/passwdaliases file, it generates an “invalid user” error which makes sense, as it’s not a precise match. Part of the troubleshooting process.

My concern is that all of the examples in the tl docs and community posts (that I read) have their full subject beginning with a key value, rather than a numerical string. It may not matter, but I do not know what that string represents.
The string 1.2.840.113549.1.9.1= is the same on both cards, with different CN values. If I remove this string plus the = and # characters from the /etc/passwdaliases file entry, instead starting with 6246b656 ... I get the same journal error message log entry "Connection closed by invalid user 1.2.840.113549.1.9.1=# ..."

The root cause is probably something simple, but I am uncertain where to go now. Any thoughts are appreciated.

(2) Second item, observation, oddly, on the thin client’s dock bar, there are four tl client login prompts listed. Closing one generates a replacement–so the quantity stays at four. When I use the smart card, I get the “You are not authorized to connect to this server.” tl client error dialog box - twice. There are four error messages in the server’s journalctl for each smart card login attempt. Seems like the repetition is delaying access by a few seconds. Wanted to mention this as it seems odd that there are so many tl client prompts that are perpetual. Perhaps related, perhaps not.

[Bill]

Hi @martin - Hope you enjoyed your weekend. I know the above is a lot. To assist further, I worked the issue more on Friday and over the weekend, making simple changes in a few documents, re-creating others and testing further. No luck.

I am at an impasse. I do not have enough TL product knowledge to troubleshoot any further. I opened a Demo Support ticket to see if I can talk to someone in support so we can save the project, buy the premium support license, and keep the customer from buying laptops later this week.

Any input is appreciated, but recognize the question above may be beyond the scope for now. Once working, I can finish the requested deliverable here, as I’ll then know what I offer is technically accurate end-to-end.

Kindest regards,

–Bill

[martin]

@Bill

Connection closed by authenticating user 1.2.840.113549.1.9.1=#16246b656e6e6574682e6275726b654077696c64776f6f6470656469617472693732e636f6d,cn=Test User,o=Test Company,L=Fairfield,st=Maryland,c=US 172.20.1.166 port 52154 [preauth]

This looks like you had it working with /etc/passwdaliases, but something else was preventing sucessfull authentication. Could’ve been bad permissions/owner of ~/.ssh/authorized_keys, and the path leading up to that file, or an invalid ssh key.

The defaul LogLevel of sshd sometimes does not yield enough information about these types of problems.

Please set the LogLevel to DEBUG, reproduce the scenario above and include the full log here and we’ll see if tells something about whats going on.

Regarding (2), are you running the tlclient with --loop perhaps? This argument will make the client start again after being closed. Make sure you’re only running one instance of the client with this argument supplied.

Regards,
Martin

[Bill]

Hi Martin - I called over to the office and had a team member insert the card for me. The log shows my ssh session initially. The card entry starts at 172.20.1.166. Also, you’ll notice that the single card entry results in duplication in the log file. I’ll check the --loop option to see if it helps, once I get on-site. Here is the log file:

[root@sessionmgr ~]# !less
less ~/journctl.ssh.log.2023.06.26.morning.test
,cn=Test User,o=Test Compan,L=Stamford,st=Connecticut,c=US 172.20.1.166 port 52180 [preauth]
Jun 26 07:52:16 sessionmgr.wwp.com sshd[760266]: debug1: do_cleanup [preauth]
Jun 26 07:52:16 sessionmgr.wwp.com sshd[760266]: debug1: monitor_read_log: child log fd closed
Jun 26 07:52:16 sessionmgr.wwp.com sshd[760266]: debug1: do_cleanup
Jun 26 07:52:16 sessionmgr.wwp.com sshd[760266]: debug1: PAM: cleanup
Jun 26 07:52:16 sessionmgr.wwp.com sshd[760266]: debug1: Killing privsep child 760267
Jun 26 07:52:19 sessionmgr.wwp.com sshd[760015]: debug1: Forked child 760268.
Jun 26 07:52:19 sessionmgr.wwp.com sshd[760268]: debug1: Set /proc/self/oom_score_adj to 0
Jun 26 07:52:19 sessionmgr.wwp.com sshd[760268]: debug1: rexec start in 5 out 5 newsock 5 pipe 7 sock 8
Jun 26 07:52:19 sessionmgr.wwp.com sshd[760268]: debug1: inetd sockets after dupping: 4, 4
Jun 26 07:52:19 sessionmgr.wwp.com sshd[760268]: Connection from 172.20.1.166 port 52182 on 172.20.1.36 port 22
Jun 26 07:52:19 sessionmgr.wwp.com sshd[760268]: debug1: Local version string SSH-2.0-OpenSSH_8.0
Jun 26 07:52:19 sessionmgr.wwp.com sshd[760268]: debug1: Remote protocol version 2.0, remote software version OpenSSH_8.7
Jun 26 07:52:19 sessionmgr.wwp.com sshd[760268]: debug1: match: OpenSSH_8.7 pat OpenSSH* compat 0x04000000
Jun 26 07:52:19 sessionmgr.wwp.com sshd[760268]: debug1: SELinux support enabled [preauth]
Jun 26 07:52:19 sessionmgr.wwp.com sshd[760268]: debug1: ssh_selinux_change_context: setting context from ‘system_u:system_r:sshd_t:s0-s0:c0.c1023’ to ‘system_u:system_r:sshd_net_t:s0-s0:c0.c1023’ [preauth]
Jun 26 07:52:19 sessionmgr.wwp.com sshd[760268]: debug1: permanently_set_uid: 74/74 [preauth]
Jun 26 07:52:19 sessionmgr.wwp.com sshd[760268]: debug1: list_hostkey_types: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
Jun 26 07:52:19 sessionmgr.wwp.com sshd[760268]: debug1: SSH2_MSG_KEXINIT sent [preauth]
Jun 26 07:52:19 sessionmgr.wwp.com sshd[760268]: debug1: SSH2_MSG_KEXINIT received [preauth]
Jun 26 07:52:19 sessionmgr.wwp.com sshd[760268]: debug1: kex: algorithm: curve25519-sha256 [preauth]
Jun 26 07:52:19 sessionmgr.wwp.com sshd[760268]: debug1: kex: host key algorithm: ssh-ed25519 [preauth]
Jun 26 07:52:19 sessionmgr.wwp.com sshd[760268]: debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: compression: none [preauth]
Jun 26 07:52:19 sessionmgr.wwp.com sshd[760268]: debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: compression: none [preauth]
Jun 26 07:52:19 sessionmgr.wwp.com sshd[760268]: debug1: kex: curve25519-sha256 need=64 dh_need=64 [preauth]
Jun 26 07:52:19 sessionmgr.wwp.com sshd[760268]: debug1: kex: curve25519-sha256 need=64 dh_need=64 [preauth]
Jun 26 07:52:19 sessionmgr.wwp.com sshd[760268]: debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
Jun 26 07:52:19 sessionmgr.wwp.com sshd[760268]: debug1: rekey out after 134217728 blocks [preauth]
Jun 26 07:52:19 sessionmgr.wwp.com sshd[760268]: debug1: SSH2_MSG_NEWKEYS sent [preauth]
Jun 26 07:52:19 sessionmgr.wwp.com sshd[760268]: debug1: expecting SSH2_MSG_NEWKEYS [preauth]
Jun 26 07:52:19 sessionmgr.wwp.com sshd[760268]: debug1: SSH2_MSG_NEWKEYS received [preauth]
Jun 26 07:52:19 sessionmgr.wwp.com sshd[760268]: debug1: rekey in after 134217728 blocks [preauth]
Jun 26 07:52:19 sessionmgr.wwp.com sshd[760268]: debug1: KEX done [preauth]
Jun 26 07:52:19 sessionmgr.wwp.com sshd[760268]: debug1: userauth-request for user 1.2.840.113549.1.9.1=#16246b656e6e6574682e6275726b654077696c64776f6f64706564696174726963732e636f6d,cn=Test User,o=Test Company,L=Stamford,st=Connecticut,c=US service ssh-connection method none [preauth]
Jun 26 07:52:19 sessionmgr.wwp.com sshd[760268]: debug1: attempt 0 failures 0 [preauth]
Jun 26 07:52:19 sessionmgr.wwp.com sshd[760268]: Invalid user 1.2.840.113549.1.9.1=#16246b656e6e6574682e6275726b654077696c64776f6f64706564696174726963732e636f6d,c from 172.20.1.166 port 52182
Jun 26 07:52:19 sessionmgr.wwp.com sshd[760268]: debug1: PAM: initializing for “1.2.840.113549.1.9.1=#16246b656e6e6574682e6275726b654077696c64776f6f64706564696174726963732e636f6d,cn=Test User,o=Test Compan,L=Stamford,st=Connecticut,c=US”
Jun 26 07:52:19 sessionmgr.wwp.com sshd[760268]: debug1: PAM: setting PAM_RHOST to “172.20.1.166”
Jun 26 07:52:19 sessionmgr.wwp.com sshd[760268]: debug1: PAM: setting PAM_TTY to “ssh”
Jun 26 07:52:19 sessionmgr.wwp.com sshd[760268]: debug1: userauth-request for user 1.2.840.113549.1.9.1=#16246b656e6e6574682e6275726b654077696c64776f6f64706564696174726963732e636f6d,cn=Test User,o=Test Company,L=Stamford,st=Connecticut,c=US service ssh-connection method publickey [preauth]
Jun 26 07:52:19 sessionmgr.wwp.com sshd[760268]: debug1: attempt 1 failures 0 [preauth]
Jun 26 07:52:19 sessionmgr.wwp.com sshd[760268]: debug1: userauth_pubkey: test pkalg rsa-sha2-512 pkblob RSA SHA256:E8g+5pVLWnAwQU0ar2TCNcQIj3v/gXfLDFxK0JK9jqY [preauth]
Jun 26 07:52:19 sessionmgr.wwp.com sshd[760268]: Connection closed by invalid user 1.2.840.113549.1.9.1=#16246b656e6e6574682e6275726b654077696c64776f6f64706564696174726963732e636f6d,cn=Test User,o=Test Company,L=Stamford,st=Connecticut,c=US 172.20.1.166 port 52182 [preauth]
Jun 26 07:52:19 sessionmgr.wwp.com sshd[760268]: debug1: do_cleanup [preauth]
Jun 26 07:52:19 sessionmgr.wwp.com sshd[760268]: debug1: monitor_read_log: child log fd closed
Jun 26 07:52:19 sessionmgr.wwp.com sshd[760268]: debug1: do_cleanup
Jun 26 07:52:19 sessionmgr.wwp.com sshd[760268]: debug1: PAM: cleanup
Jun 26 07:52:19 sessionmgr.wwp.com sshd[760268]: debug1: Killing privsep child 760269
(END)

[martin]

Hello @Bill

It looks like you weren’t able to replicate the scenario where it was able to identify the user from passwdaliases. The attached log still says

Invalid user 1.2.840.113549.1.9.1=#16246b656e6e6574682e6275726b654077696c64776f6f64706564696174726963732e636f6d,c from 172.20.1.166 port 52182

At one point you had the server report:

Connection closed by authenticating user 1.2.840.113549.1.9.1=#16246b656e6e6574682e6275726b654077696c64776f6f6470656469617472693732e636f6d,cn=Test User,o=Test Company,L=Fairfield,st=Maryland,c=US 172.20.1.166 port 52154 [preauth]

You need to get back to this, somewhat working, situation and then attach the journal log with LogLevel=DEBUG in sshd_config.

Regards,
Martin

[Bill]

Thanks, Martin. Long 22 hour shift, but we’re almost ready. A few logistical issues arose extending the day, but all six TC units are updated, tested and ready to go for an afternoon demo and POC launch. The second card kept generating errors, so I erased it and started over. Will test it later today and keep you updated as appropriate. Thanks again to you and Pierre for the input.

[Bill]

Hi Martin - Hope you enjoyed your weekend. Two more 24 hour shifts, but we got everything done that they needed for the POC on-time. Long week. Results? Success. The Partners are happy.

You were correct on the remote office’s performance over the VPN. Impressive in that it is about only 0.5 seconds longer to authenticate than when in the main office with almost identical performance. Excellent job to your team on the coding. Interesting to see how things hold-up when we have all seven in that office on the pipe.

For this week, we are putting a quote together for all the parts and a project timeline. Thank you! Have a great week.

Kindest regards,

–Bill