Background
In Java 8u261 was MSCAPI completely rewritten [1] and also was support for MS Cryptography next generation (CNG) added [2].
[1] JDK-8213009 Refactoring existing SunMSCAPI classes
[2] JDK-8026953 Add support for MS Cryptography next generation (CNG)
"The CNG API integrates with the smart card subsystem by including a Base Smart Card Cryptographic Service Provider (Base CSP) module which encapsulates the smart card API. Smart card manufacturers just have to make their devices compatible with this, rather than provide a from-scratch solution." https://en.wikipedia.org/wiki/Microsoft_CryptoAPI#Cryptography_API:_Next_Generation
See source code for Java 8 http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/687fd7c7986d/src/windows/classes/sun/security/mscapi/SunMSCAPI.java
And especially sun.security.mscapi.RSASignature http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/687fd7c7986d/src/windows/classes/sun/security/mscapi/RSASignature.java
See source code for Java 11 https://github.com/openjdk/jdk11u/tree/master/src/jdk.crypto.mscapi/windows/classes/sun/security/mscapi
And especially the new class sun.security.mscapi.CSignature that has replaced sun.security.mscapi.RSASignature https://github.com/openjdk/jdk11u/blob/master/src/jdk.crypto.mscapi/windows/classes/sun/security/mscapi/CSignature.java
See release notes Java 8u261 https://www.oracle.com/java/technologies/javase/8all-relnotes.html#R180_261
security-libs/javax.net.ssl TLS Support for RSASSA-PSS Signature Algorithms
security-libs/javax.net.ssl JEP 332: Transport Layer Security (TLS) 1.3
For Consolidated Release Notes for JDK 8 and JDK 8 Update Releases, see https://www.oracle.com/java/technologies/javase/8all-relnotes.html
Problem
The problem is that most smart card does not support RSASSA-PSS.
See "We are trying to disable RSASSA-PSS, because it is not supported in the JCE PKCS11 wrapper, and causes errors when setting up TLS1.2 errors." JDK-8226374 Restrict TLS signature schemes and named groups
And also "was unsupported by the open source smart card driver OpenSC, as well as an overall industry-wide problematic treatment of certificates with RSASSA-PSS. The issue with RSASSA-PSS in certificates was quite fundamental due to their unique and complex design, and was ultimately addressed by the TLS working group by making the RSASSA-PSS in certificates optional." RED HAT BLOG Transport Layer Security version 1.3 in Red Hat Enterprise Linux 8
The stacktrace from a Java client connecting with SunMSCAPI
$ java -Djavax.net.ssl.keyStore=NONE \
-Djavax.net.ssl.keyStoreType=Windows-MY \
-Djavax.net.ssl.keyStoreProvider=SunMSCAPI \
-Djavax.net.ssl.trustStore=NONE \
-Djavax.net.ssl.trustStoreType=Windows-ROOT \
-Djavax.net.ssl.trustStoreProvider=SunMSCAPI \
se.mkk.Main
...
Exception in thread "main" javax.net.ssl.SSLHandshakeException: Cannot produce CertificateVerify signature
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:353)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:296)
at java.base/sun.security.ssl.CertificateVerify$T12CertificateVerifyMessage.<init>(CertificateVerify.java:611)
at java.base/sun.security.ssl.CertificateVerify$T12CertificateVerifyProducer.produce(CertificateVerify.java:761)
at java.base/sun.security.ssl.SSLHandshake.produce(SSLHandshake.java:436)
at java.base/sun.security.ssl.ServerHelloDone$ServerHelloDoneConsumer.consume(ServerHelloDone.java:182)
at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421)
at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:183)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1506)
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1416)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:456)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:427)
at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:572)
at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:201)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1592)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1520)
at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:527)
at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:334)
at se.mkk.HttpURLConnectionBuilder.send(HttpURLConnectionBuilder.java:116)
at se.mkk.Main.getHttpURLConnection(Main.java:54)
at se.mkk.Main.httpURLConnection(Main.java:44)
at se.mkk.Main.main(Main.java:18)
Caused by: java.security.SignatureException: Unknown error
at jdk.crypto.mscapi/sun.security.mscapi.CSignature.signCngHash(Native Method)
at jdk.crypto.mscapi/sun.security.mscapi.CSignature$PSS.engineSign(CSignature.java:607)
at java.base/java.security.Signature$Delegate.engineSign(Signature.java:1404)
at java.base/java.security.Signature.sign(Signature.java:713)
at java.base/sun.security.ssl.CertificateVerify$T12CertificateVerifyMessage.<init>(CertificateVerify.java:609)
One way to get around this is to disable RSASSA-PSS, but this only works for > Java 11, due to that above bug JDK-8226374 is not backpoarted to Java 8.
%JAVA_HOME%\conf\security\java.security
…
jdk.tls.disabledAlgorithms=…, RSASSA-PSS
…
Other References
Sean Mullan Technical Lead of the Java Security Libraries Team at Oracle
Additional information on Oracle's JDK and JRE Cryptographic Algorithms
