Introduction Elasticsearch
Why Elasticsearch? "Elasticsearch is a Near Realtime (NRT) search platform." [1]
Elasticsearch stores only JSON Object (Document). A JSON Object is built up by name and value pairs. Value can be one of the following data types:
- a string
- a number
- an object (JSON object)
- an array
- a boolean
- null
Elasticsearch supports more data types, than standard JSON, such as date and geo-point. [2]
Elasticsearch is accessed primarly with JSON over HTTP.
Several of the same documents type are mapped to Types. Which can be thought of as Tables in a RDBM. Different Types are stored in an Indices (plural of Index) which can be thought of a Database.
| MySQL | => Databases | => Tables | => Columns/Rows |
| Elasticsearch | => Indices | => Types | => Documents with Properties |
Searching and querying takes the format of: http://localhost:9200/[index]/[type]/[id], i.e. GET localhost:9200/accounts/person/1
An Index can potentially grow and be very large, to help that, Elasticsearch splits Index to multiple Shards. Both Index and Shard are stored in a server Node.
"Sharding is important for two primary reasons:
- It allows you to horizontally split/scale your content volume
- It allows you to distribute and parallelize operations across shards (potentially on multiple nodes) thus increasing performance/throughput"
"... Elasticsearch allows you to make one or more copies of your index’s shards into what are called replica shards, or replicas for short."
"Replication is important for two primary reasons:
- It provides high availability in case a shard/node fails. For this reason, it is important to note that a replica shard is never allocated on the same node as the original/primary shard that it was copied from.
- It allows you to scale out your search volume/throughput since searches can be executed on all replicas in parallel."
Multiple Nodes forms a Cluster. When you first start Elasticsearch, you start a Node in a Cluster called "elasticsearch", i.e. you have created a Cluster with one Node.
Reference
- [1] https://www.elastic.co/guide/en/elasticsearch/reference/current/_basic_concepts.html
- [2] https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html
Elasticsearch
Download
$ wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.4.tar.gz
$ wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.4.tar.gz.sha512
$ shasum -a 512 -c elasticsearch-6.2.4.tar.gz.sha512
elasticsearch-6.2.4.tar.gz: OK
ZIP Installation
$ tar -xzf elasticsearch-6.2.4.tar.gz
$ cd elasticsearch-6.2.4/
Run Interactively
$ bin/elasticsearch
Test
$ curl http://localhost:9200/
{
"name" : "hCDxdzG",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "npmGFI2nRcmXke0YLW9tXQ",
"version" : {
"number" : "6.2.4",
"build_hash" : "ccec39f",
"build_date" : "2018-04-12T20:37:28.497551Z",
"build_snapshot" : false,
"lucene_version" : "7.2.1",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}
Or open browser 'http://localhost:9200/'.
Reference
Kibana
Download
$ wget https://artifacts.elastic.co/downloads/kibana/kibana-6.2.4-linux-x86_64.tar.gz
$ wget https://artifacts.elastic.co/downloads/kibana/kibana-6.2.4-linux-x86_64.tar.gz.sha512
$ shasum -a 512 -c kibana-6.2.4-linux-x86_64.tar.gz.sha512
kibana-6.2.4-linux-x86_64.tar.gz: OK
ZIP Installation
$ tar -xzf kibana-6.2.4-linux-x86_64.tar.gz
$ cd kibana-6.2.4-linux-x86_64/
Run Interactively
$ bin/kibana
Test
Or open browser 'http://localhost:5601/'.
Reference
X-Pack
"X-Pack is an Elastic Stack extension that bundles security (*), alerting, monitoring, reporting, and graph capabilities into one easy-to-install package."
(*) Authentication, Authorization and Audit Log.
Without X-Pack, Elasticsearch is wide open and everyone can read everything.
Download
$ wget https://artifacts.elastic.co/downloads/packs/x-pack/x-pack-6.2.4.zip
$ wget https://artifacts.elastic.co/downloads/packs/x-pack/x-pack-6.2.4.zip.sha512
$ shasum -a 512 -c x-pack-6.2.4.zip.sha512
x-pack-6.2.4.zip: OK
X-Pack Elasticsearch
Installation
$ bin/elasticsearch-plugin install [x-pack|file:///path/to/file/x-pack-6.2.4.zip]
$ bin/elasticsearch-plugin install file:///home/magnus/bin/x-pack-6.2.4.zip
-> Downloading file:///home/magnus/bin/x-pack-6.2.4.zip
[=================================================] 100%
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: plugin requires additional permissions @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.io.FilePermission \\.\pipe\* read,write
* java.lang.RuntimePermission accessClassInPackage.com.sun.activation.registries
* java.lang.RuntimePermission getClassLoader
* java.lang.RuntimePermission setContextClassLoader
* java.lang.RuntimePermission setFactory
* java.net.SocketPermission * connect,accept,resolve
* java.security.SecurityPermission createPolicy.JavaPolicy
* java.security.SecurityPermission getPolicy
* java.security.SecurityPermission putProviderProperty.BC
* java.security.SecurityPermission setPolicy
* java.util.PropertyPermission * read,write
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.
Continue with installation? [y/N]y
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: plugin forks a native controller @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
This plugin launches a native controller that is not subject to the Java
security manager nor to system call filters.
Continue with installation? [y/N]y
Elasticsearch keystore is required by plugin [x-pack-security], creating...
-> Installed x-pack with: x-pack-monitoring,x-pack-ml,x-pack-logstash,x-pack-graph,x-pack-core,x-pack-upgrade,x-pack-deprecation,x-pack-security,x-pack-watcher
Then restart elasticsearch and continue with changing admin passwords.
$ bin/x-pack/setup-passwords [auto|interactive]
$ bin/x-pack/setup-passwords auto
Initiating the setup of passwords for reserved users elastic,kibana,logstash_system.
The passwords will be randomly generated and printed to the console.
Please confirm that you would like to continue [y/N]y
Changed password for user kibana
PASSWORD kibana = PFUPZoLlTdIqECtoS0Qs
Changed password for user logstash_system
PASSWORD logstash_system = sw9XNc2fDskZstgiHdaJ
Changed password for user elastic
PASSWORD elastic = 8fAYzzJ2ZlGxdynE2zqf
Reference
X-Pack Kibana
Installation
$ bin/kibana-plugin install [x-pack|file:///path/to/file/x-pack-6.2.4.zip]
$ bin/kibana-plugin install file:///home/magnus/bin/x-pack-6.2.4.zip
Attempting to transfer from file:///home/magnus/bin/x-pack-6.2.4.zip
Transferring 309419696 bytes....................
Transfer complete
Retrieving metadata from plugin archive
Extracting plugin archive
Extraction complete
Optimizing and caching browser bundles...
Plugin installation complete
Configuration
$ vi config/kibana.yml
...
elasticsearch.username: "kibana"
elasticsearch.password: "PFUPZoLlTdIqECtoS0Qs"
...
Restart Kibana and open browser http://localhost:5601
Username: elastic
Password: 8fAYzzJ2ZlGxdynE2zqf
Using Kibana
Open http://localhost:5601 in your web browser and select DevTools. Now we will use examples from
https://www.elastic.co/pdf/getting-started-webinar-dev-console-commands.pdf
SSL/TLS Elasticsearch
Encrypting HTTP Client Communications: https://www.elastic.co/guide/en/elasticsearch/reference/6.2/configuring-tls.html#tls-http
Create a root, intermediate and localhost certificate according to http://magnus-k-karlsson.blogspot.se/2016/09/openssl-certificate-authority-ca.html
Server Certificate:
- /home/magnus/bin/ca/intermediate/certs/localhost.cert.pem
- /home/magnus/bin/ca/intermediate/private/localhost.key.pem-NOPWD
Server Certificate Chain:
- /home/magnus/bin/ca/intermediate/certs/intermediate.cert.pem
- /home/magnus/bin/ca/certs/ca.cert.pem
Elasticsearch do not have a seperate settings for server certificate chain, so you need to add the server certificate chain to the server certificate pem file.
$ cp localhost.cert.pem localhost.cert.pem+ca-chain.cert.pem
$ cat intermediate.cert.pem >> localhost.cert.pem+ca-chain.cert.pem
$ cat ca.cert.pem >> localhost.cert.pem+ca-chain.cert.pem
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: certs/localhost.key.pem-NOPWD
xpack.security.http.ssl.certificate: certs/localhost.cert.pem+ca-chain.cert.pem
A note about settings 'xpack.security.http.ssl.certificate_authorities', which one might think is server certificate chain, but is NOT. It is for client certificate authentication and is the trusted client certificate CA that are allowed to login. To enable two way SSL you also need to set 'xpack.ssl.client_authentication: required'.
Elasticsearch also supports PKCS#12 format, to use it.
$ cp intermediate.cert.pem ca-chain.cert.pem
$ cat ca.cert.pem >> ca-chain.cert.pem
openssl pkcs12 -export -out localhost.p12 -inkey localhost.key.pem-NOPWD -in localhost.cert.pem -certfile ca-chain.cert.pem
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/localhost.p12
Created elasticsearch keystore in /home/magnus/bin/elasticsearch-6.2.4/config
$ bin/elasticsearch-keystore create
Enter value for xpack.security.http.ssl.keystore.secure_password:
$ bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password
This setups transport crypto (HTTPS) and if you want client certificate authentication also, you need to set the truststore property. But using PKCS#12 format for client certificate CA is not a smooth way, specially when client certificate have a different CA than the server certificate. So I would recommend using the PEM settings.
To setup transport crypto for the internal traffic inside elasticsearch nodes, see Encrypting Communications Between Nodes in a Cluster.
To verify your settings you can either open 'localhost:9200' in a web browser and or use 'openssl s_client', to fully verify your SSL/TLS settings.
$ openssl s_client -connect localhost:9200 -showcerts
NOTE When enable SSL/TLS is HTTP disabled.
SSL/TLS Hardening Elasticsearch
The only secure SSL protocol version is TLSv1.2
xpack.ssl.supported_protocols: TLSv1.2
To hardening you cipher suite you need to install Java Cryptography Extension (JCE), if you are running OpenJDK it is already installed and if you are running Oracle JDK, you need to install it.
Here we will use OWASP TLS/SSL Cheat Sheet
$ openssl ciphers -v "EDH+aRSA+AESGCM:EDH+aRSA+AES:EECDH+aRSA+AESGCM:EECDH+aRSA+AES:-SHA:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:RSA+AESGCM:RSA+AES+SHA256:RSA+AES+SHA:DES-CBC3-SHA:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA"
DHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=DH Au=RSA Enc=AESGCM(256) Mac=AEAD
DHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=DH Au=RSA Enc=AESGCM(128) Mac=AEAD
DHE-RSA-AES256-SHA256 TLSv1.2 Kx=DH Au=RSA Enc=AES(256) Mac=SHA256
DHE-RSA-AES128-SHA256 TLSv1.2 Kx=DH Au=RSA Enc=AES(128) Mac=SHA256
ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(256) Mac=AEAD
ECDHE-RSA-AES256-SHA384 TLSv1.2 Kx=ECDH Au=RSA Enc=AES(256) Mac=SHA384
ECDHE-RSA-AES128-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AES(128) Mac=SHA256
ECDHE-RSA-AES256-SHA SSLv3 Kx=ECDH Au=RSA Enc=AES(256) Mac=SHA1
ECDHE-RSA-AES128-SHA SSLv3 Kx=ECDH Au=RSA Enc=AES(128) Mac=SHA1
AES256-GCM-SHA384 TLSv1.2 Kx=RSA Au=RSA Enc=AESGCM(256) Mac=AEAD
AES128-GCM-SHA256 TLSv1.2 Kx=RSA Au=RSA Enc=AESGCM(128) Mac=AEAD
AES256-SHA256 TLSv1.2 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA256
AES128-SHA256 TLSv1.2 Kx=RSA Au=RSA Enc=AES(128) Mac=SHA256
AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1
AES128-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(128) Mac=SHA1
DES-CBC3-SHA SSLv3 Kx=RSA Au=RSA Enc=3DES(168) Mac=SHA1
DHE-RSA-AES256-SHA SSLv3 Kx=DH Au=RSA Enc=AES(256) Mac=SHA1
DHE-RSA-AES128-SHA SSLv3 Kx=DH Au=RSA Enc=AES(128) Mac=SHA1
Then we filter out:
- only TLSv1.2 ciphers
- only encryption AESGCM
- only key exchange DH or ECDH
Which leaves us with only these four ciphers
DHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=DH Au=RSA Enc=AESGCM(256) Mac=AEAD
DHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=DH Au=RSA Enc=AESGCM(128) Mac=AEAD
ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(256) Mac=AEAD
ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
Now we need to map these ciphers to Java: https://docs.oracle.com/javase/8/docs/technotes/guides/security/SunProviders.html#SunJSSEProvider
TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
xpack.ssl.cipher_suites: TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
Then restart and verify
$ openssl s_client -connect localhost:9200 -showcerts
...
SSL-Session:
Protocol : TLSv1.2
Cipher : DHE-RSA-AES256-GCM-SHA384
...
Reference
SSL/TLS Kibana
When setting up SSL/TLS for Kibana you first need to set https in the URL to Elasticsearch.
elasticsearch.url: "https://localhost:9200"
Then we need to add a truststore in Kibanan, so it trust the Elasticsearch SSL/TLS. A few notes about the certificate management in Kibana:
- All files needs to be inside kibana directory, due to X-Pack Java Security Manager.
- You need to use absulte path, compared with elasticsearch.
elasticsearch.ssl.certificateAuthorities: [ "/home/magnus/bin/kibana-6.2.4-linux-x86_64/config/certs/intermediate.cert.pem", "/home/magnus/bin/kibana-6.2.4-linux-x86_64/config/certs/ca.cert.pem" ]
Then restart Kibana and test that you can access http://localhost:5601.
Username: elastic
Password: 8fAYzzJ2ZlGxdynE2zqf
When OK, then lets continue with setting up SSL/TLS for Kibana.
server.ssl.enabled: true
server.ssl.certificate: /home/magnus/bin/kibana-6.2.4-linux-x86_64/config/certs/localhost+ca-chain.cert.pem
server.ssl.key: /home/magnus/bin/kibana-6.2.4-linux-x86_64/config/certs/localhost.key.pem-NOPWD
And finally test it: https://localhost:5601/
Reference
No comments:
Post a Comment