OpenShift 4.6 Automation and Integration: Kibana

Filtering Queries

12.3. Kubernetes exported fields
https://access.redhat.com/documentation/en-us/openshift_container_platform/4.6/html-single/logging/index#cluster-logging-exported-fields-kubernetes_cluster-logging-exported-fields

These are the Kubernetes fields exported by the OpenShift Container Platform cluster logging available for searching from Elasticsearch and Kibana.

hostname	The hostname of OpenShift node that generated the message.
kubernetes.flat_labels	The label for the pod that generated the message. Format: key=value
kubernetes.container_name	The name of the container in Kubernetes.
kubernetes.namespace_name	The name of the namespace in Kubernetes.
kubernetes.pod_name	The name of the pod that generated the log message.
level	The log level of the message.
message	The actual log message.

Example Lucene query:

+kubernetes.namespace_name:"openshift-etcd" +message:elected

Finding OpenShift Event Logs

kubernetes.event
kubernetes.event.involvedObject.name	Resource name invloved in event.
kubernetes.event.involvedObject.namespace	Namespace of the resource name invloved in event.
kubernetes.event.reason	The reason for the event. Correspond to the values in the REASON column that displays in the output of the oc get events command.
kubernetes.event.type	The type of message, e.g. kubernetes.event.type:warning

Visualizing Time Series with Timelion

Timelion Tutorial – From Zero to Hero
https://www.elastic.co/blog/timelion-tutorial-from-zero-to-hero

.es('+kubernetes.namespace_name:logging-query +message:200'),
.es('+kubernetes.namespace_name:logging-query +message:404'),
.es('+kubernetes.namespace_name:logging-query +message:500')

.es('+kubernetes.container_name:logger +message:500')
.divide(.es('+kubernetes.container_name:logger +message:*'))
.multiply(100)

.es('+kubernetes.container_name:logger +message:500').label(current),
.es(q='+kubernetes.container_name:logger +message:500', offset=-5m).label(previous)

Troubleshooting cluster logging

Chapter 10. Troubleshooting cluster logging
https://access.redhat.com/documentation/en-us/openshift_container_platform/4.6/html-single/logging/index#troubleshooting-cluster-logging

$ oc get -n openshift-logging clusterlogging instance -o yaml

apiVersion: logging.openshift.io/v1
kind: ClusterLogging
....
status:  
...
  logstore:
    elasticsearchStatus:
    - ShardAllocationEnabled:  all
      cluster:
        activePrimaryShards:    5
        activeShards:           5
        initializingShards:     0
        numDataNodes:           1
        numNodes:               1
        pendingTasks:           0
        relocatingShards:       0
        status:                 green
        unassignedShards:       0
      clusterName:             elasticsearch
...

Using Grafana

Monitoring -> Dashboards:

Dashboards: Kubernetes / Compute Resources / Node (Pods)
Namespace: openshift-logging

Using Kibana

Infra index
+kubernetes.namespace_name:openshift-logging +kubernetes.container_name: