External application server requirements

An external application server's primary task should be to ensure uninterrupted operation. This means above all that it is important to plan ahead when configuring the Java VM heap size and garbage collection settings.

To configure the Java VM of an external application server, the same basic principles apply as for the FirstSpirit Server (see Configuring the Java VM):

Tuning memory management of the Java VM

Heap

The area of memory available for FirstSpirit to use is called the “heap” of the Java VM.

If the size of the heap is set too low, the Java VM cannot retrieve enough main memory from the operating system; this means the FirstSpirit Server can no longer process requests and instead responds with error messages due to “insufficient main memory”.

The heap should be set to as high a value as possible, but no larger than the main memory available in the operating system.

If the heap is significantly larger than 10 GB, it will be difficult to manage with the parameters given here. If this order of magnitude is necessary, garbage collection behavior via JMX must be analyzed with jconsole or VisualVM and the Java VM parameters tuned for the application in question.

If a heap larger than 10 GB is to be set up, the manufacturer (Crownpeak Technology GmbH) must be contacted first, since most cases like this require other, specific parameters to configure the garbage collector.

A large Java heap (over 1 GB) means parameters need to be optimized to tune the garbage collection, as described in the section below.

Configuring the initial and maximum heap size

The parameter –Xms defines the initial heap size (corresponds to parameter wrapper.java.initmemory or wrapper.java.initmemory.percent in the configuration file fs-wrapper.conf) assigned to the Java VM when it starts (see Java wrapper parameters).

The parameter –Xmx defines the maximum heap size (corresponds to parameter wrapper.java.maxmemory or wrapper.java.maxmemory.percent in the configuration file fs-wrapper.conf) made available to the Java VM (see Java wrapper parameters).

When the VM allocates or releases memory from the operating system memory, this can lead to delays. Avoid this by setting the initial and maximum heap size to the same value, e.g.:

wrapper.java.initmemory=6000 
wrapper.java.maxmemory=6000

The heap size selected should not be too large, since FirstSpirit attempts to fill the entire free space for cache.

The parameter -Xmn should no longer be used from Java 9 onward, as it has negative effects on the G1 garbage collector (Garbage-First).

Monitoring heap capacity

The parameter com.sun.management.jmxremote is used to monitor the heap capacity, including in production mode, so that trends related to the allocation of storage can be detected and the heap size can be increased, if necessary.

Current monitoring systems offer a JMX interface that allows monitoring of web application servers.

The "PSI Probe" web application is recommended for querying the current values (https://github.com/psi-probe/psi-probe).

The jconsole program can also be used for interactive queries. It is included in every JDK (not JRE) from Oracle.

Metaspace

Tuning the Java VM garbage collection

Large amounts of data in FirstSpirit projects or a high number of FirstSpirit users being active at the same time can overload the Java VM garbage collector. This overload becomes apparent when FirstSpirit Clients take longer than 10 s to respond. In the event of longer wait times than this, connections between a FirstSpirit Client and FirstSpirit Server can even be lost. The garbage collector, which brings the FirstSpirit Server to a temporary yet complete standstill for certain operations, is responsible for this wait time. This state is known as "“stop-the-world”" (STW).  

The garbage collector's log file (log/fs-wrapper.log or log/fs-gc.log) contains the message “Full GC” or “time exceeded” at this point in time.

In its default configuration, FirstSpirit uses the G1 garbage collector (Garbage-First). Amongst other things, the G1 collector splits the cleanup of aged objects (“Old Generation”) into several subareas, not all of which have to be “stop-the-world”. This means that even high loads (large amounts of data, lots of users) should cause hardly any loss of connection between the FirstSpirit Server and FirstSpirit Clients.

However, if problems do occur (loss of connection, long response times, not enough memory), the corresponding GC parameters can be tuned to optimize behavior (see, e.g., https://docs.oracle.com/en/java/javase/11/gctuning/garbage-first-garbage-collector-tuning.html#GUID-90E30ACA-8040-432E-B3A0-1E0440AB556A).

In certain cases, it may help to switch to a different GC collector (see https://docs.oracle.com/en/java/javase/11/gctuning/available-collectors.html#GUID-F215A508-9E58-40B4-90A5-74E29BF3BD3C).

Generally speaking: Monitoring of the heap capacity and the garbage collection is indispensable to determine exactly the right settings for the heap size and the GC parameters.