Documentation for Administrators

Introduction / FirstSpirit ServerManager / Project properties / Repository

Repository

Oracle Berkeley DB is used in FirstSpirit as a repository for saving content data, currently in version 7. This repository is provided in form of a system module (fs-berkeleydb7.fsm).

Backend: This section shows which Berkeley DB version is used for the specific project. Currently, this is version 7.x. If further versions should become available in the future, the version can be selected via this drop-down box.

Note: As a security measure for protection against the loss of data, Berkeley DB checks the remaining hard disk space. By default, no more storage operations are possible in the Berkeley database if the remaining disk reaches a value below 512 mebibytes (approx. 536 MB) (error message: Disk usage is not within je.maxDisk or je.freeDisk limits and write operations are prohibited.).

The value for “Exit server at” in the Global server properties should therefore be no less than 512 mebibytes.

Note about older FirstSpirit servers: As of FirstSpirit 2018-11, only Berkeley DB version 7 will be supported. FirstSpirit servers which were initially installed with FirstSpirit 2018-06 or lower and use Berkeley DB in a version < 7 must be converted because Berkeley DB 7 uses a different data format than versions 3 and 5 which were previously used by FirstSpirit. If the existing data was not converted to the new Berkeley DB 7 format, the server will no longer start with FirstSpirit 2018-11. For a comfortable conversion, the command-line utility “BerkeleyUtil.jar” is provided; for more information about this utility, see the section Conversion to Berkeley DB 7.

Compressor: A different compression can be set as necessary via this drop-down list. Compression affects both the amount of disk space required and the access speed.

[none]: No compression is used (recommended default setting).
Deflate: Algorithm with high compression ratio
LZ4: Algorithm designed to provide high speeds

Encryption

This area can be used to configure encryption of the repository contents (content, structures, media) for a project. For background information on repository encryption, see Repository encryption.

Requirements: To configure these project settings, a global server key file is required (see Creating the key file). The repository.encryption.keyFilePath parameter must be used to store the path to the key file in the fs-server.conf configuration file (for information on configuration, see Area: Storage Engine Properties).

Note concerning default values: In the case of new and imported projects, default values can be assigned to the form fields using global values from the fs-server.conf configuration file. This means that encryption can be enabled for a new project right away because the repository.encryption parameter is set to a value of 1 in the fs-server.conf configuration file (see Area: Storage Engine Properties). The project settings can be configured independently of the global values. As a result, encryption can be enabled or disabled for specific projects. In addition, specific projects can be encrypted with stronger or weaker algorithms as required.

Encryption enabled: This checkbox can be used to enable/disable repository encryption for each project on an individual basis. If this checkbox is selected, the encryption mechanism is applied to the current project (see Performing encryption/decryption), but if it is unchecked, the repository contents are not encrypted. In the case of existing projects, the checkbox must be selected initially in order for pre-existing repository content to be encrypted. The project cannot be used during the initial encryption process (temporarily disabled). If encryption is disabled, the repository is decrypted using a similar process. A default value can be assigned globally by using the repository.encryption parameter in the fs-server.conf configuration file (for new or imported projects) (see Area: Storage Engine Properties).

Encryption algorithm: The encryption algorithm to be used can be selected from the drop-down menu. The name, mode, and padding are specified in each case. The algorithms shown here are merely examples and are not to be construed as recommendations:

AES/CBC/PKCS5Padding
AES/CTR/NoPadding (default)
ARCFOUR
Blowfish/CBC/PKCS5Padding
Blowfish/CTR/NoPadding

As an alternative to the suggested values, you can enter a valid encryption algorithm in the field manually. The actual encryption process is handled by the Java Cryptography Extension. As a result, it is the Java platform used that determines which symmetric encryptions and modes can be configured here. For details of which algorithms, modes and key sizes are possible, please see the relevant JCE documentation:

as well as section Notes on specific Java versions.

A default value can be assigned globally by using the encryption.algorithm parameter in the fs-server.conf configuration file (for new or imported projects) (see Area: Storage Engine Properties).

Encryption key size: The length of the encryption key can be configured via this drop-down menu. As an alternative to the suggested values (64, 128, 192, and 256 bits), other values can be entered manually. The key size must be compatible with the configured algorithm (encryption, modes).

For more information on permitted key sizes, see:

as well as section Notes on specific Java versions.

A default value can be assigned globally by using the encryption.keySize parameter in the fs-server.conf configuration file (see Area: Storage Engine Properties).

Test encryption parameters: Click this button to test the encryption process with the currently configured parameters. The test checks the encryption process in accordance with the specifications of the Java Cryptography Extension, e.g. to see whether the specified algorithm and key size can be combined with one another.

Use fast-path repository conversion if possible: This checkbox currently has no effect. If newer versions of Berkeley DB should become available in the future, activating this checkbox enables faster data conversion.

When the user confirms the selection with OK, the system starts to convert the data using the desired settings. The relevant project is deactivated during the process.

To prevent any data loss, anyone using the project should log off first. Changes to the repository settings can take some time and should only be performed during a maintenance interval. We recommend you create a data backup before starting a repository conversion.

Conversion to Berkeley DB 7 (as of FirstSpirit 5.2R19)

(only for FirstSpirit servers using Berkeley DB with version <7!)

The conversion tool “BerkeleyUtil” can be used for the simple conversion of all data stored in Berkeley DBs on existing FirstSpirit servers to Berkeley DB 7 format by entering just a few commands in the command line (“project repositories” and “internal repositories”). The exact procedure is described in the section Recommended procedure for a conversion to Berkeley DB 7. As server-level data also has to be converted, a conversion of this nature can only be performed when the FirstSpirit Server is offline in order to prevent any loss of data. A conversion will generally be completed in a short time. Only in exceptional cases may it take up to several hours; for example, in the event of very large and/or numerous projects.

Usage

The “BerkeleyUtil” tool is contained in the fs-isolated-server.jar file and is rolled out to the “bin” directory when the server is started.

Prerequisites:

Oracle Java: min. version 8
The tool should only be used once the corresponding FirstSpirit server has been shut down.
Sufficient memory space: Three times the hard disk space of the biggest database that is being converted (without “blob” directory, this can be omitted from the calculation) will potentially be required temporarily.

Important: Conversion via BerkeleyUtil.jar must be executed by the user who will then be using the Berkeley databases. In practice, this means the user who is the administrator for the FirstSpirit Server or the Application Server.

Call and options:

java -jar BerkeleyUtil.jar COMMAND [OPTION]... PATH

First -jar must be used to specify the path under which the BerkeleyUtil.jar file is located on the FirstSpirit server, by default in the “bin” directory, e.g.

java -jar firstspirit/bin/BerkeleyUtil.jar

The following commands can be used for the COMMAND placeholder:

-c, --convert: converts all uses below the specified directory (see PATH below) to Berkeley DB version 7 (complete FirstSpirit server)
- --convert-acl: converts all ACL databases to Berkeley DB 7
- --convert-project: converts all project repositories to Berkeley DB 7
- --convert-server: converts all internal repositories to Berkeley DB 7
-d, --dump: repairs the desired repository as a copy (standard mode I)
--exclude: used to set directories which should be excluded from consideration and conversion. The directories must be provided as a suitable RegEx. All subdirectories will be excluded as well.
Example:
java -jar bin\BerkeleyUtil.jar --exclude webapps -l
=> excludes all directories whose names contain the character sequence “webapps”.
By default, all directories will be excluded whose names begin with . (period).
-h, --help: displays help text
-l, --list: lists the names, version, path and size of all repositories on the server and provides a summary of the amount of data to be converted
-r, --recover: attempts to restore the desired repository (standard mode II)
-R, --RECOVER: attempts to restore the desired repository (with advanced error corrections)
-t, --verify: checks the desired repository

The commands cannot be combined, i.e. each command requires a separate call.

The commands -d, -r or -R should only be used following a prior analysis. Please contact Technical Support for support issues.

The following values can be used for the OPTION placeholder:

--dump-dir <directory>: For the conversion or recovery of a repository, a copy is first created that is then imported. This call can be used to specify a directory in which the temporary copy is to be created.
Note: The amount of free storage space of the volume in which this directory is located must be at least as large as the largest database that is to be converted (without the “blob” directory, this may be excluded from the calculation).
-v, --verbose: outputs additional logs, such as the stacktrace for error messages
-f, --fast: allows faster in-place conversion of project repositories. However, this should only be done if there is an up-to-date backup, since in rare cases existing problems in a repository can lead to it being irreparably damaged during a conversion.
By default, a copy of the repository to be converted is created first, but when the option -f is used this copy is not created. Although it is more secure, the process of copying requires a little more time and temporary hard disk space. It also optimizes the repository to be converted so that it no longer contains unnecessary data
--no-log-file: The conversion actions are logged in a log file as standard. The creation of this log file can be deactivated with this option.

PATH must be used to specify the path to the directory to be taken into account by the tool, e.g.

to the root directory of the FirstSpirit server: /firstspirit
to a directory of a Berkeley DB on the FirstSpirit server, e.g. /firstspirit/data/projects/project_123/repository (repository of a project), /tomcat/webapps/ROOT/WEB-INF/acl (ACL database of a live project)

Example call:

java -jar firstspirit/bin/BerkeleyUtil.jar -l firstspirit

This call lists all Berkeley databases of the FirstSpirit server installed in the “firstspirit” directory and displays its versions.
By default, all directories whose names begin with . will be excluded. If other directories should be excluded, these must be provided as a suitable RegEx via the parameter --exclude.
Example:

java -jar bin\BerkeleyUtil.jar --exclude webapps -l

=> excludes all directories whose names contain the character sequence “webapps”.

Log file
The conversion actions are logged in a log file as standard. The log file name is berkeley_util_yyyyMMdd_HHmmss.log; for example:

berkeley_util_20180502_113208.log

If you convert an entire FirstSpirit Server, the file is stored in the log directory of the server. If you convert individual databases, the file is stored in the directory which was specified as the start directory (PATH parameter).

Calculating the amount of hard disk space that is likely to be required
While converting the Berkeley databases of a FirstSpirit Server using the tool, you may temporarily require three times the memory space of the largest database that is being converted.The latest FirstSpirit version now checks to see if there is sufficient free hard disk space before converting each database. If this is not the case, the conversion will not be executed. The conversion for this database then stops with an Insufficient free space error message. If this happens, create more space or remove files from the directory before starting a new conversion.
Note: In very rare cases, the conversion tool is unable to calculate the required storage space reliably. If the available disk space is too low, the conversion terminates with the following error message: Error converting BerkeleyDB. If this happens, please contact Technical Support.

Recommended procedure for a conversion to Berkeley DB 7

1) The FirstSpirit server must be powered down for a conversion via the conversion tool. For this reason, a conversion should be carried out during a maintenance interval.

2) Run the following example call (modified to the path of the root directory of the FirstSpirit server):

java -jar -Xmx#m firstspirit/bin/BerkeleyUtil.jar -c /firstspirit

The conversion process should be started with sufficient memory. As a rule of thumb, -Xmx#m should be used with the same value as defined for the wrapper.java.maxmemory for the FirstSpirit server.

The conversion process can take some time depending on the size and number of the repositories present on the FirstSpirit server (a maximum of a few hours). The conversion should not be canceled during this time as this could cause data loss and inconsistencies! In the event of canceling the process there is no rollback, and manual intervention will be required. Please contact Technical Support.

3) If the conversion was completed successfully, a corresponding message is output in the log, e.g

<timestamp> [INFO Bdb7Convert] BerkeleyDB version 7 conversion successful, marker file 
   written: firstspirit/data/server/berkeleydb.7

4) The following parameter must be entered in the fs-wrapper.conf file:

-DBerkeleyDB7=1

5) The FirstSpirit server can then be started and used as normal.

Troubleshooting

If a conversion is not completed successfully, a corresponding message is output in the log, e.g.

<timestamp> [WARN Bdb7Convert] BerkeleyDB version 7 marker file not written, 1 errors 
   during the conversion process.

In this case, running the conversion again can resolve the issue.
If this is not successful, please contact Technical Support. The same applies if a conversion process had to be canceled. Message in the log on server start:

FATAL <timestamp> (de.espirit.firstspirit.server.ServerManagerImpl): Incomplete 
   BerkeleyDB version 7 conversion detected

If there is insufficient free hard disk space for the conversion of a database, the conversion for this database then stops with an Insufficient free space error message. See paragraph above: “Calculating the amount of hard disk space that is likely to be required”.

For assistance from Technical Support, please have the log file from the last execution of the conversion tool to hand.

Saving data in the event of an error (backup)

Should problems occur during a conversion, the original data from the database is saved in a backup directory on the FirstSpirit Server. This enables the data to be analyzed at a later point in time if necessary. The directory is created in the same location as the original directory. Its name consists of the name of the original directory with the suffix backup and the time stamp of the conversion.
This directory contains the original data prior to the conversion. It can be deleted at a later point in time.

Note: When the databases are listed or verified ( -l / --list or -t / --verify), any backup directories that exist are logged as a warning:

<timestamp> [WARN BdbScanner] Incomplete conversion/restore detected: <path>...backup...

Print version | Page start