OpenShift 4.6 Automation and Integration: Kubernetes vs OpenShift, Kustomize and Image Streams

Kubernetes vs OpenShift

https://access.redhat.com/documentation/en-us/openshift_container_platform/4.6/html-single/applications/index#what-deployments-are

Kubernetes	OpenShift
Namespace	Project
Ingress	Route
Deployment Emphasizes availability over consistency. Uses ReplicaSets that support set-based match selectors. Red Hat recommends using Deployments unless you need specific DeploymentConfigs feature.	DeploymentConfig Emphasizes consistency over availability.
Kustomize	Template

$ kubectl create -f hello.yml

$ kubectl apply -f hello.yml

$ kubectl get ingresses.v1.networking.k8s.io

Kustomize

A kustomization is a directory containing a kustomization.yml file.

https://kubernetes.io/docs/tasks/manage-kubernetes-objects/kustomization/

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - mydeployment.yml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - mydeployment.yaml
images:
  - name: image
    newName: new-image
    newTag: new-tag

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - mydeployment.yaml
secretGenerator:
  - name: mycert
    namespace: openshift-config
    files:
      - tls.crt=my-priv-cert.crt
      - tls.key=my-priv-cert.key
generatorOptions:
  disableNameSuffixHash: true

A kustomization without a bases field is a base.

An overlay includes all resources in its bases.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
  - path-to-kustomization

Validate your Kustomize configurations.

$ oc kustomize kustomize_folder

$ oc apply --dry-run -k config

Apply your Kustomize configurations.

$ kubectl apply -k directory_name

Image Streams

Image streams use a unique SHA256 identifier instead of a mutable image tag. This is more robust since image tags (:latest or :v1.1) can change without further notice.

https://access.redhat.com/documentation/en-us/openshift_container_platform/4.6/html-single/images/index#managing-image-streams

Annotating Deployments with Image Stream Triggers

Key: image.openshift.io/triggers
Value:
[
 {
   "from": {
     "kind": "ImageStreamTag",
     "name": "example:latest",
     "namespace": "myapp"
   },
   "fieldPath": "spec.template.spec.containers[?(@.name==\"web\")].image",
   "paused": false
 },
 ...
]

$ skopeo copy \
  docker://quay.io/redhattraining/versioned-hello:v1.0 \
  docker://quay.io/your_account/versioned-hello:latest
  
$ oc get imagestreams

Import image and create image streams and set periodically scheduled (--scheduled) imports to get latest updates.

$ oc import-image quay.io/your_account/versioned-hello:latest --confirm --scheduled

$ oc set triggers deployment/hello --from-image versioned-hello:latest -c hello

RPM In Which Order is Scripts Executed

See 

How to execute a script at %pre, %post, %preun or %postun stage (spec file) while installing/upgrading an rpm

https://www.golinuxhub.com/2018/05/how-to-execute-script-at-pre-post-preun-postun-spec-file-rpm/

Java Smart Card Authentication Fails on RHEL 8 with Java 8u261 and SunMSCAPI

Background

In Java 8u261 was MSCAPI completely rewritten [1] and also was support for MS Cryptography next generation (CNG) added [2].

[1] JDK-8213009 Refactoring existing SunMSCAPI classes

[2] JDK-8026953 Add support for MS Cryptography next generation (CNG)

"The CNG API integrates with the smart card subsystem by including a Base Smart Card Cryptographic Service Provider (Base CSP) module which encapsulates the smart card API. Smart card manufacturers just have to make their devices compatible with this, rather than provide a from-scratch solution." https://en.wikipedia.org/wiki/Microsoft_CryptoAPI#Cryptography_API:_Next_Generation

See source code for Java 8 http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/687fd7c7986d/src/windows/classes/sun/security/mscapi/SunMSCAPI.java

And especially sun.security.mscapi.RSASignature http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/687fd7c7986d/src/windows/classes/sun/security/mscapi/RSASignature.java

See source code for Java 11 https://github.com/openjdk/jdk11u/tree/master/src/jdk.crypto.mscapi/windows/classes/sun/security/mscapi

And especially the new class sun.security.mscapi.CSignature that has replaced sun.security.mscapi.RSASignature https://github.com/openjdk/jdk11u/blob/master/src/jdk.crypto.mscapi/windows/classes/sun/security/mscapi/CSignature.java

See release notes Java 8u261 https://www.oracle.com/java/technologies/javase/8all-relnotes.html#R180_261

security-libs/javax.net.ssl TLS Support for RSASSA-PSS Signature Algorithms

security-libs/javax.net.ssl JEP 332: Transport Layer Security (TLS) 1.3

For Consolidated Release Notes for JDK 8 and JDK 8 Update Releases, see https://www.oracle.com/java/technologies/javase/8all-relnotes.html

Problem

The problem is that most smart card does not support RSASSA-PSS.

See "We are trying to disable RSASSA-PSS, because it is not supported in the JCE PKCS11 wrapper, and causes errors when setting up TLS1.2 errors." JDK-8226374 Restrict TLS signature schemes and named groups

And also "was unsupported by the open source smart card driver OpenSC, as well as an overall industry-wide problematic treatment of certificates with RSASSA-PSS. The issue with RSASSA-PSS in certificates was quite fundamental due to their unique and complex design, and was ultimately addressed by the TLS working group by making the RSASSA-PSS in certificates optional." RED HAT BLOG Transport Layer Security version 1.3 in Red Hat Enterprise Linux 8

The stacktrace from a Java client connecting with SunMSCAPI

$ java -Djavax.net.ssl.keyStore=NONE \
-Djavax.net.ssl.keyStoreType=Windows-MY \
-Djavax.net.ssl.keyStoreProvider=SunMSCAPI \
-Djavax.net.ssl.trustStore=NONE \
-Djavax.net.ssl.trustStoreType=Windows-ROOT \
-Djavax.net.ssl.trustStoreProvider=SunMSCAPI \
se.mkk.Main
...
Exception in thread "main" javax.net.ssl.SSLHandshakeException: Cannot produce CertificateVerify signature
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:353)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:296)
        at java.base/sun.security.ssl.CertificateVerify$T12CertificateVerifyMessage.<init>(CertificateVerify.java:611)
        at java.base/sun.security.ssl.CertificateVerify$T12CertificateVerifyProducer.produce(CertificateVerify.java:761)
        at java.base/sun.security.ssl.SSLHandshake.produce(SSLHandshake.java:436)
        at java.base/sun.security.ssl.ServerHelloDone$ServerHelloDoneConsumer.consume(ServerHelloDone.java:182)
        at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421)
        at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:183)
        at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)
        at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1506)
        at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1416)
        at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:456)
        at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:427)
        at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:572)
        at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:201)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1592)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1520)
        at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:527)
        at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:334)
        at se.mkk.HttpURLConnectionBuilder.send(HttpURLConnectionBuilder.java:116)
        at se.mkk.Main.getHttpURLConnection(Main.java:54)
        at se.mkk.Main.httpURLConnection(Main.java:44)
        at se.mkk.Main.main(Main.java:18)
Caused by: java.security.SignatureException: Unknown error
        at jdk.crypto.mscapi/sun.security.mscapi.CSignature.signCngHash(Native Method)
        at jdk.crypto.mscapi/sun.security.mscapi.CSignature$PSS.engineSign(CSignature.java:607)
        at java.base/java.security.Signature$Delegate.engineSign(Signature.java:1404)
        at java.base/java.security.Signature.sign(Signature.java:713)
        at java.base/sun.security.ssl.CertificateVerify$T12CertificateVerifyMessage.<init>(CertificateVerify.java:609)

One way to get around this is to disable RSASSA-PSS, but this only works for > Java 11, due to that above bug JDK-8226374 is not backpoarted to Java 8.

%JAVA_HOME%\conf\security\java.security
…
jdk.tls.disabledAlgorithms=…, RSASSA-PSS
…

Other References

Sean Mullan Technical Lead of the Java Security Libraries Team at Oracle

Additional information on Oracle's JDK and JRE Cryptographic Algorithms

OpenShift 4.10 I: S2I, start-build, BuildConfig and Deployment

$ oc new-app --name=php-helloworld --image-stream=php:7.3 https://github.com/magnuskkarlsson/DO180-apps#s2i --context-dir=php-helloworld

$ oc start-build buildconfig.build.openshift.io/nodejs-dev

$ oc logs -f buildconfig.build.openshift.io/nodejs-dev
...
Push successful

$ oc logs -f deployment.apps/nodejs-dev

OpenShift 4.10 I: Create an OCP Application from Image, S2I, Template

From Image

$ oc new-project myproj01

$ oc new-app --name=httpd-24 --image=registry.access.redhat.com/ubi8/httpd-24 --labels app=httpd-24

$ oc get all

$ oc logs pod/httpd-24-9fb54567d-n9slj

$ oc expose service/httpd-24

$ oc get all

$ curl http://httpd-24-ch06s03.apps-crc.testing/

$ oc exec pod/httpd-24-9fb54567d-n9slj -- ps -aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
1000650+       1  0.0  0.2 391296 21980 ?        Ss   19:54   0:00 httpd -D FOREGROUND

$ oc describe pod/httpd-24-9fb54567d-n9slj

Source-to-Image (S2I)

$ oc new-project myproj03

$ oc get is -n openshift

$ oc new-app --name=ruby-hello-world --labels app=myapp --image-stream=ruby https://github.com/openshift/ruby-hello-world

$ oc get all

$ oc logs -f pod/ruby-hello-world-1-build

$ oc describe pod/ruby-hello-world-1-build

$ oc describe service/ruby-hello-world

$ oc expose service/ruby-hello-world

$ curl http://ruby-hello-world-myproj03.apps-crc.testing/

$ oc get buildconfig
NAME               TYPE     FROM   LATEST
ruby-hello-world   Source   Git    1

$ oc start-build ruby-hello-world

From Template

$ oc new-project myproj02

$ oc get templates -n openshift

$ oc get templates -n openshift | grep mysql
mysql-ephemeral                                 MySQL database service, without persistent storage. For more information abou...   8 (3 generated)   3
mysql-persistent                                MySQL database service, with persistent storage. For more information about u...   9 (3 generated)   4

$ oc describe template mysql-persistent -n openshift

$ oc new-app --name=app-db --template=mysql-persistent \
  --param MYSQL_USER=myuser \
  --param MYSQL_PASSWORD=redhat123 \
  --param MYSQL_ROOT_PASSWORD=redhat123 \
  --param MYSQL_DATABASE=items \
  --labels app=app-db

$ oc get events

$ oc describe service/mysql
Name:              mysql
Namespace:         myproj02
Labels:            app=app-db
                   template=mysql-persistent-template
Annotations:       openshift.io/generated-by: OpenShiftNewApp
                   template.openshift.io/expose-uri: mysql://{.spec.clusterIP}:{.spec.ports[?(.name=="mysql")].port}
Selector:          name=mysql
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.217.4.67
IPs:               10.217.4.67
Port:              mysql  3306/TCP
TargetPort:        3306/TCP
Endpoints:         10.217.0.106:3306
Session Affinity:  None
Events:            <none>

$ oc get pvc
NAME    STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
mysql   Bound    pv0023   100Gi      RWO,ROX,RWX                   2m59s

$ oc describe pvc mysql
Name:          mysql
Namespace:     myproj02
StorageClass:  
Status:        Bound
Volume:        pv0023
Labels:        app=app-db
               template=mysql-persistent-template
Annotations:   openshift.io/generated-by: OpenShiftNewApp
               pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      100Gi
Access Modes:  RWO,ROX,RWX
VolumeMode:    Filesystem
Used By:       mysql-1-qncwj
Events:        <none>

$ oc port-forward pod/mysql-1-qncwj 3306:3306

$ mysql --host=127.0.0.1 --port=3306 --user=myuser --password=redhat123 --database=items --execute="show databases;"

OpenShift 4.10 I: OCP Manifest/Custom Resources Documentation

$ oc api-resources 
NAME                                  SHORTNAMES       APIVERSION                                    NAMESPACED   KIND
bindings                                               v1                                            true         Binding
componentstatuses                     cs               v1                                            false        ComponentStatus
configmaps                            cm               v1                                            true         ConfigMap
endpoints                             ep               v1                                            true         Endpoints
...

$ oc explain pod
KIND:     Pod
VERSION:  v1

DESCRIPTION:
     Pod is a collection of containers that can run on a host. This resource is
     created by clients and scheduled onto hosts.

FIELDS:
   apiVersion	<string>
     APIVersion defines the versioned schema of this representation of an
     object. Servers should convert recognized schemas to the latest internal
     value, and may reject unrecognized values. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources

   kind	<string>
     Kind is a string value representing the REST resource this object
     represents. Servers may infer this from the endpoint the client submits
     requests to. Cannot be updated. In CamelCase. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds

   metadata	<Object>
     Standard object's metadata. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata

   spec	<Object>
     Specification of the desired behavior of the pod. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status

   status	<Object>
     Most recently observed status of the pod. This data may not be up to date.
     Populated by the system. Read-only. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status

$ oc explain pod.spec 
KIND:     Pod
VERSION:  v1

RESOURCE: spec <Object>

DESCRIPTION:
     Specification of the desired behavior of the pod. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status

     PodSpec is a description of a pod.

FIELDS:
   activeDeadlineSeconds	<integer>
     Optional duration in seconds the pod may be active on the node relative to
     StartTime before the system will actively try to mark it failed and kill
     associated containers. Value must be a positive integer.

   affinity	<Object>
     If specified, the pod's scheduling constraints
...

$ oc explain pod.spec --recursive 
KIND:     Pod
VERSION:  v1

RESOURCE: spec <Object>

DESCRIPTION:
     Specification of the desired behavior of the pod. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status

     PodSpec is a description of a pod.

FIELDS:
   activeDeadlineSeconds	<integer>
   affinity	<Object>
      nodeAffinity	<Object>
         preferredDuringSchedulingIgnoredDuringExecution	<[]Object>
            preference	<Object>
               matchExpressions	<[]Object>
                  key	<string>
                  operator	<string>
                  values	<[]string>
               matchFields	<[]Object>
                  key	<string>
                  operator	<string>
                  values	<[]string>
            weight	<integer>
         requiredDuringSchedulingIgnoredDuringExecution	<Object>
...

OpenShift 4.10 I: Create an OCP Application

First create a new OCP project.

$ oc new-project myapp

Create an application from an image.

$ oc new-app --name=todonodejs \
  --image=quay.io/redhattraining/do180-todonodejs-12 \
  --env MYSQL_ENV_MYSQL_DATABASE=tododb \
  --env MYSQL_ENV_MYSQL_USER=user1 \
  --env MYSQL_ENV_MYSQL_PASSWORD=redhat123 \
  --labels app=todonodejs

Create an application based on source code in a git repository - Source-to-Image (S2I).

$ oc new-app --name=nodejs-dev \
  --image-stream=nodejs:16-ubi8 \
  https://github.com/magnuskkarlsson/DO180-apps#troubleshoot-review \
  --context-dir=nodejs-app

Create an application from an existing template.

$ oc new-app --name=mysql --template=mysql-persistent \
  --param MYSQL_USER=user1 \
  --param MYSQL_PASSWORD=redhat123 \
  --param MYSQL_ROOT_PASSWORD=redhat123 \
  --param MYSQL_DATABASE=tododb \
  --labels app=todonodejs