Role of ElasticSearch in webMethods API Portal stack

webMethods API Portal guide

Elasticsearch is first and foremost a scalable full-text search engine. It can store arbitrary documents in JSON format, which it can access in near real-time, making it effectively also a NoSQL backend. Elasticsearch serves as the backend for several applications.

with respect to webMethods API Portal context, we heavily rely on API Portal for below scenarios.

• ElasticSearch is used as search index for collaboration and document content.
• ElasticSearch is the backend used to store all User Management Console(UMC) data (users, user groups, licenses, license assignments, ...) and also serves as backend for all currently valid user sessions.
• ElasticSearch is used as primary storage for storing transactional events captured from the mediation engines(API Gateway/Mediator) and hence this is also a backbone for the visualization of dashboards with API Portal.

In particular the essential role it has for UMC makes Elasticsearch a service whose availability is pivotal for the availability of API Portal as a whole. So if one wants to set up a distributed, highly available installation, one has to set up Elasticsearch in a highly available way.

ElasticSearch version

Below table summarizes the Portal version and its corresponding elastic version & port

ElasticSearch Troubleshooting

ElasticSearch related configuration/data/log files are located in work directory( $$SoftwareAG\API_Portal\server\bin\work\ work_elastic_s ).  

1. Configuration file(elasticsearch.yml): The key configuration file (elasticsearch.yml) located in $$SoftwareAG\API_Portal\server\bin\work\ work_elastic_s\config.
2. Log files will be located in ($$SoftwareAG\API_Portal\server\bin\work\work_elastic_s\logs).
   • If you face any issues in starting up elastic instance, the first place to look out is the elasticsearch.log in this logs directory.
3. Data folder contains the data pushed into elastic store ($$SoftwareAG\API_Portal\server\bin\work\work_elastic_s\data)

ElasticSearch Configuration

The key configuration file (elasticsearch.yml) will be generated dynamically during startup of elasticsearch from the runnable configuration.  The runnable specific configurations can be viewed using the acc command (show runnable <instance id>).

In the configuration parameters listed, the parameters prefixed with ELASTICSEARCH are translated to elastic search configuration parameters in the elasticsearch.yml. If incase you would like to add/update any specific configuration parameters to elasticsearch you can use reconfigure command to reconfigure elastic search runnable. For example if i want to reduce max_result_window from 500000 to 100000 you can do it as shown below

reconfigure elastic_s ELASTICSEARCH.index.max_result_window=100000

Known issues

• vm.max_map_count

One setting that is particularly troublesome on Linux systems is the vm.max_map_count setting, which in almost all environments will be set too low by default. Elasticsearch 5.x requires this value to be set to at least 262144. The corresponding error message in elasticsearch.log will be
"max virtual memory areas vm.max_map_count [xyz] is too low, increase to at least [262144]" 

where xyz corresponds to the current value set on your node.You can also check the current value by running the Linux shell command (e.g., via an SSH shell on the node)

sysctl -n vm.max_map_count

You can change it transiently (i.e., only until next reboot) by running the Linux shell command
sudo sysctl -w vm.max_map_count=262144
to change it persistently, as a root user, edit the file /etc/sysctl.conf on the node(s) on which the Elasticsearch runnable(s) is/are located and add the line
vm.max_map_count=262144