1. Introduction

In the progress of further development of SmartSearch, it is necessary to make adjustments to the configuration, schema, or components used that require manual changes.

This document describes such migration scenarios.

With the update to version 2.4.0 haupia has been renamed to SmartSearch. Therefore, it may happen that code examples or APIs still contain the term haupia.

Please note that "haupia" has been renamed to "SmartSearch" with version 2.4.0. In all instances where there may be discrepancies or confusion, "SmartSearch" should be considered the correct and updated term. This might not be reflected in old scripts and examples, so we apologize for any inconvenience and hope that the instances requiring correction will be minimal.

2. SmartSearch-update - General

The use of the functions, possibilities as well as performance adjustments of new SmartSearch versions requires regular version updates. With these, changes are potentially necessary both within the software of the SmartSearch itself and in the connected ZooKeeper as well as Solr instances. They are primarily minimal adjustments to the Solr schema or changes to the ZooKeeper structure in which the SmartSearch persists data. The update startup parameter allows these changes to be automatically applied during a version update by triggering the execution of the required modifications within ZooKeeper and Solr. Its use is necessary only if an exit code 1 occurs during the update process.

A SmartSearch update will never perform an automatic version update for ZooKeeper or Solr.

Furthermore, SmartSearch updates are generally backwards compatible and do not affect the functionality of the SmartSearch APIs as far as possible.

If in exceptional cases there are deviations from these two points, the release notes explicitly point this out and are to be considered in each case before the execution of an update. This ensures that connected systems are prepared in advance for possible adaptations.

SmartSearch update packages can be requested at Crownpeak | e-Spirit Support.

Performing a version update is always done using the steps below:

  1. The first step is to download the new SmartSearch archive and unpack it in any (empty) folder of the target system. It then contains various directories and files, of which the following are potentially relevant for the update:

    • application.yml

    • server.conf

    • server.jar

  2. Depending on the deployment scenario, either single operation requires this one node to be shut down, or cluster operation requires a single SmartSearch node to be shut down.

  3. Before performing any further steps, first check the release notes for any update notes.

  4. If the release notes contain any update notes, make the changes to application.yml or server.conf as described in the release notes.

  5. In the execution directory of the stopped instance, both the server.jar file and the shared_resources directory must then be replaced with the file or directory of the unpacked archive.

    Directories like taglib or documentation do not have to be replaced to ensure functionality, but they may be replaced as well without risk.

  6. In the last step, the previously stopped SmartSearch node must be restarted. The software automatically checks at each start whether all necessary configurations exist. If this is not the case or if ZooKeeper respectively Solr need additional changes, it issues a corresponding message and exits with an exit code 1.

    In the case of an exit code 1, the information about the missing adjustments is contained in the log of the SmartSearch process and these are to be carried out manually accordingly.

    Then, to restart the SmartSearch node, the update parameter must be used (for example ./server.jar update)

    The software performs a recheck and makes necessary adjustments to the ZooKeepers or Solrs data structure automatically. These adjustments are also recorded in the SmartSearch log.

After successfully making all necessary changes, the software will start as usual.

If the SmartSearch does not start successfully after being called with the update parameter, check the SmartSearch log for appropriate error messages.

In cluster mode, steps 3-6 must then be performed for each SmartSearch node. The use of the update parameter is only necessary in these cases if the release notes explicitly refer to it.

2.1. Troubleshooting

Within the course of a project, different situations can arise that require specific action steps. For this reason, the following sections describe some approaches to solving known challenges.

Problems with loading the keystore

The keystore password defined in the application.yml file must match the keystore used when starting the server. This also applies if the supplied keystore is used for development. In this case, the password from the application.yml template, which is also supplied, must be used.

When using an incorrect password, the following error message appears in the log:

Error message with wrong keystore password 
org.springframework.context.ApplicationContextException: Unable to start web server; nested exception is java.lang.IllegalStateException: org.springframework.boot.web.server.WebServerException: Could not load key store './../shared_resources/keystore/keystore.p12'
...
Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
        ... 20 common frames omitted

3. Migration to version 4.2.0

3.1. Solr Upgrade to Version 8.11.2

With the SmartSearch version 4.2.0, the Solr version used has been updated from 8.6.3 to 8.11.2. The following subchapter describes the procedure for the required Solr upgrade.

  1. Stop both the SmartSearch and Solr server:

    sudo systemctl stop smart-search
sudo systemctl stop solr

  2. Backup the Current Solr Folder:

    Before making any changes, ensure you have a backup of your current Solr folder.

    cp -r /opt/solr-8.6.3 /opt/solr-8.6.3_backup

  3. Download and Extract the New Solr Version:

    Download the latest Solr version from the official Solr website and extract it.

    wget https://archive.apache.org/dist/lucene/solr/8.11.2/solr-8.11.2.tgz
tar -xzf solr-8.11.2.tgz -C /opt/solr-8.11.2

  4. Copy server/solr Path to the New Solr Folder:

    Copy only the server/solr directory from the old Solr folder to the new Solr folder. This step ensures that your core configurations and data are transferred.

    cp -r /opt/solr-8.6.3/server/solr /opt/solr-8.11.3/server/solr

  5. Change the owner and group of the new folders:

    sudo chown -R solr:solr /opt/solr-8.11.2

  6. Switch Over the Symlink:

    Update the symlink to point to the new Solr folder.

    # Remove the existing symlink
rm /opt/solr
    # Create a new symlink to the updated Solr folder
ln -s /opt/solr-8.11.2 /opt/solr

  7. If changes to the service configuration were made, it might be necessary to restart the service demon:

    Restart the service demon to reload potential changes.

    systemctl daemon-reload

  8. Restart Solr:

    Restart the Solr service to apply the changes.

    sudo systemctl start solr

  9. Verify the Update:

    Check the Solr admin interface to ensure that the update was successful and that Solr is running correctly.

    http://localhost:8983/solr

  • Permissions: Ensure that the permissions and ownership of the new Solr folder match those of the original folder as described.

  • Configurations: Verify that any custom configurations or plugins from the old Solr installation are properly integrated into the new version.

  • Testing: Thoroughly test the new Solr instance in a staging environment before applying it to production.

4. Migration to version 4.0.0

With SmartSearch version 4.0.0 a mandatory new entry in the application.yml must be added. Without it, the application will fail to start.

The error message should be along the lines of: 
***************************
APPLICATION FAILED TO START
***************************

Description:

The dependencies of some of the beans in the application context form a cycle (...)
Please make sure that this entry is present in your application.yml on startup 
spring:
  main:
    allow-circular-references: true

5. Migration to version 3.10.0

With version 3.10.0 a new dynamic field *_date_range is introduced. This requires a change to the schema. This happens automatically, provided that the SmartSearch server is started with the update parameter. Details about this can be found in chapter SmartSearch-update - General.

6. Migration to version 3

With SmartSearch version 3, the Solr version used was upgraded from 8.1.1 to 8.6.3. The following subchapter describes the procedure for the Solr upgrade required by this.

This procedure follows the official Solr documentation. After the update the collections are reindexed to convert them to the new version. The SmartSearch release therefore contains the so-called reindex tool, the functionality of which is described in the following chapter.

A change to the application.yml of the SmartSearch instance is not necessary.

6.1. Reindex tool

After a Solr update the file format of the index may change. To adopt the new file format, a reindexing of the data and signals collection is necessary. The reindex tool helps to do this. It creates a new collection and copies the data from the old collection to the new one. Finally, the tool checks the copied data and, if successful, creates an alias that the SmartSearch uses.

During reindexing, the SmartSearch can be used searchingly, but data generatoren must not be run. The final check of the data ensures the completeness of the data collection. For the signals collection, on the other hand, it merely checks that all signals in the new collection match those in the old collection. As a result, it is possible that signals stored during the runtime of the tool are lost.

The reindex tool must be able to access the Solr server.

The use of the reindex tool requires the extraction of the corresponding zip file from the delivery into a directory. The zip file is located in the tools directory of the delivery and the name starts with "reindex-tool". The adjustment of the configuration happens in the file reindex.properties. At least the following parameters have to be adjusted:

  • solr.url: The URL to a Solr node. For example: http://solr1:8983/solr.

  • mandator: The company name from the license. In addition, a collection or alias corresponding to the name must exist on the Solr server.

If required, the configuration of an authentication is possible. An example is included in the properties file.

After the successful configuration the following command starts the reindex tool:

Start commans 
java -jar reindex-tool.jar

Since Solr versions 7.4.0 to 7.7.3 and 8.0.0 to 8.11.0 are affected by the Log4J CVE-2021-44228 issue, the recommendation is to adjust the following configuration: if such a version is being used on older installations.

(Linux/MacOS) Edit your solr.in.sh file to include: SOLR_OPTS="$SOLR_OPTS -Dlog4j2.formatMsgNoLookups=true"
(Windows) Edit your solr.in.cmd file to include: set SOLR_OPTS=%SOLR_OPTS% -Dlog4j2.formatMsgNoLookups=true

More information on the topic can be found here.

6.2. Solr-upgrade to version 8.6.3

The following description assumes that the Solr installation has been done into the /opt directory and that the data is located in the /opt/solr-data directory. It is also based on the assumption that the control of all services is done with systemd and that they are named smartsearch, solr and zookeeper.

The Solr update requires an exchange of the schema and the solrconfig.xml file. In addition, the MANDATOR variable must be replaced with the customer name from the license.

The following description assumes a running SmartSearch server and one or more Solr and ZooKeeper servers and is organized in four steps:

  • Solr-upgrade

  • Recording of the configurations of the data collection

  • Import of the configurations of the Signals Collection.

  • Completion of the migration.

Solr-upgrade

  1. Download the Solr archive version 8.6.3 from official website.

  2. Unpack the installation/upgrade script:

    tar xzf solr-8.6.3.tgz solr-8.6.3/bin/install_solr_service.sh --strip-components=2

  3. Stop both the SmartSearch and Solr server, and then run the install/upgrade script:

    sudo systemctl stop smart-search
sudo systemctl stop solr
sudo ./install_solr_service.sh solr-8.6.3.tgz -d /opt/solr-data -n -f

  4. Change the owner and group of the created symbolic link and folder:

    sudo chown -R solr:solr /opt/solr
sudo chown -R solr:solr /opt/solr-8.6.3

  5. Add the execute right to the zkCli.sh script to download the collection configuration:

    sudo chmod +x /opt/solr/server/scripts/cloud-scripts/zkcli.sh

Importing the configuration of the data collection

  1. Download the configuration to the /tmp/backup/SmartSearch directory:

    /opt/solr/server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181/solr -cmd downconfig -confdir /tmp/backup/SmartSearch -confname MANDATOR

  2. Create a backup of the managed-schema and solrconfig.xml files in the /tmp/backup/SmartSearch directory.

  3. Replace the managed-schema and solrconfig.xml files in the /tmp/backup/SmartSearch directory with the files of the same name in the main directory from the reindex tool zip file.

  4. Upload the new configuration:

    /opt/solr/server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181/solr -cmd upconfig -confdir /tmp/backup/SmartSearch -confname MANDATOR

  5. Delete the previously created folder /tmp/backup/SmartSearch. The data contained in it is no longer needed.

    rm -rf /tmp/backup/SmartSearch

Importing the configuration of the Signals Collection

  1. Download the signals configuration to the /tmp/backup/SmartSearch directory:

    /opt/solr/server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181/solr -cmd downconfig -confdir /tmp/backup/SmartSearch -confname MANDATOR_signals

  2. Create a backup of the managed-schema and solrconfig.xml files in the /tmp/backup/SmartSearch directory.

  3. Replace the managed-schema and solrconfig.xml files in the /tmp/backup/SmartSearch directory with the files of the same name from the signals directory from the reindex tool zip file.

  4. Upload the new configuration:

    /opt/solr/server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181/solr -cmd upconfig -confdir /tmp/backup/SmartSearch -confname MANDATOR_signals

  5. Delete the previously created folder /tmp/backup/SmartSearch. The data contained in it is no longer needed.

    rm -rf /tmp/backup/SmartSearch

Completing the migration

  1. Start both the Solr and SmartSearch servers and the reindex tool. From now on, search queries are executable again. The command requires the previously executed configuration of the tool.

    sudo systemctl start solr
sudo systemctl start smart-search
java -jar reindex-tool.jar

  2. In the Solr backend, determine the names of the new collections for data and signals.

  3. Verify that the status of both collections is healthy. In the following call, the -c parameter sets the name of the collection.

    /opt/solr/bin/solr healthcheck -c MANDATOR_20210902141120 -z localhost:2181/solr

7. Migration to version 2.3.0

The SmartSearch version 2.3.0 contains an internal update from Spring Boot to 2.3.8. As part of this upgrade, some configuration adjustments are required in the application.yml file. The adjustments relate to renaming the keys listed below. For more details, see the Spring Boot 2.3 Release Notes.

The following list shows the keys that need to be renamed and their new names.

Table 1. table Old and new configuration keys

Old key name

New key name

spring.http.encoding.charset

server.servlet.encoding.charset

spring.http.encoding.enabled

server.servlet.encoding.enabled

spring.http.encoding.force

server.servlet.encoding.force

8. Migration to version 2.2.0

To use the new features of SmartSearch version 2.2.0, adjustments to the Solr configuration are required. Therefore, the new configuration must be manually uploaded to the Solr server after the update. To do this, run the following command in the SmartSearch installation directory. Replace the placeholder MANDATOR with the client name - in the notation as it is entered in the license file.

The command is based on the assumption that the Solr installation was done into the /opt directory. If this is not the case, the paths must be adjusted accordingly.
Update command 
/opt/solr/server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181/solr -cmd upconfig -confdir ./shared_resources/configsets/SmartSearch -confname MANDATOR

To verify that the transfer was successful, you can connect to the Solr web interface. This can be accessed by calling the URL localhost:8983 in your browser. The web interface contains two dropdown boxes on the left side. In the upper box, select the collection with your client name and click on 'Files' in the following folder view. Then select managed-schema at the very bottom of the list to see the contents of the selected file in XML format on the right. If in the second line the attribute version shows the value 2.1, the transfer was successful.

If grouping is used with a Prepared Search, note that fields with the _keyword suffix are no longer supported by the new scheme. Fields with the _token suffix should be used instead.

9. Migration to version 2.1.0

With the SmartSearch version 2.1.0 the used Solr version was updated to 8.1.1. The following subchapters describe the procedure for the Solr upgrade required by this.

This procedure follows the official Solr documentation. After the update the Collections are reindexed, in order to convert them to the new version. The SmartSearch_ delivery therefore contains the so-called reindex tool, the functionality of which is described in the following chapter.

In addition, the SmartSearch requires Java 11 as of this version. Older Java versions are no longer supported. As part of the migration, the Java version upgrade must be performed after upgrading the Solr version 6 to 7.

A change to the application.yml of the SmartSearch instance is not necessary.

9.1. Reindex tool

After a Solr update the file format of the index may change. To adopt the new file format, a reindexing of the data and signals collection is necessary. The reindex tool helps to do this. It creates a new collection and copies the data from the old collection to the new one. Finally, the tool checks the copied data and, if successful, creates an alias that the SmartSearch uses.

During reindexing, the SmartSearch can be used searchingly, but data generatoren must not be run. The final check of the data ensures the completeness of the data collection. For the signals collection, on the other hand, it merely checks that all signals in the new collection match those in the old collection. As a result, it is possible that signals stored during the runtime of the tool are lost.

The reindex tool must be able to access the Solr server.

The use of the reindex tool requires the extraction of the corresponding zip file from the delivery into a directory. The zip file is located in the tools directory of the delivery and the name starts with "reindex-tool". The adjustment of the configuration happens in the file reindex.properties. At least the following parameters have to be adjusted:

  • solr.url: The URL to a Solr node. For example: http://solr1:8983/solr.

  • mandator: The company name from the license. In addition, a collection or alias corresponding to the name must exist on the Solr server.

If required, the configuration of an authentication is possible. An example is included in the properties file.

After the successful configuration the following command starts the reindex tool:

Start commans 
java -jar reindex-tool.jar

9.2. Solr upgrade to version 7.7.3

The following description assumes that the Solr installation has been done into the /opt directory and that the data is located in the /opt/solr-data directory. It is also based on the assumption that the control of all services is done with systemd and that they are named smartsearch, solr and zookeeper.

In Solr version 7, the ukrainian.dict, which enables the analysis of the Ukrainian language, has been removed. The following steps describe the necessary changes to adapt the schema accordingly. In the shell statements, the placeholder MANDATOR denotes the client name in the notation as it is entered in the license file.

Starting from a running SmartSearch server and one or more Solr and ZooKeeper servers, perform the following steps:

  1. Download the Solr archive version 7.7.3 from official website.

  2. Unpack the installation/upgrade script:

    tar xzf solr-7.7.3.tgz solr-7.7.3/bin/install_solr_service.sh --strip-components=2

  3. Stop both the SmartSearch and Solr servers, and then run the install/upgrade script:

    sudo systemctl stop smart-search
sudo systemctl stop solr
sudo ./install_solr_service.sh solr-7.7.3.tgz -d /opt/solr-data -n -f

  4. Change the owner and group of the created symbolic link and folder:

    sudo chown -R solr:solr /opt/solr
sudo chown -R solr:solr /opt/solr-7.7.3

  5. Add the execute right to the zkCli.sh script to download the collection configuration:

    sudo chmod +x /opt/solr/server/scripts/cloud-scripts/zkcli.sh

  6. Download the configuration to the /tmp/backup/SmartSearch directory:

    /opt/solr/server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181/solr -cmd downconfig -confdir /tmp/backup/SmartSearch -confname MANDATOR

  7. Create a backup of the managed-schema file in the /tmp/backup/SmartSearch directory.

  8. Modify the schema as follows to comment out the filter for the Ukrainian language:

    sed -i 's/<filter class="solr\.MorfologikFilterFactory" dictionary="org\/apache\/lucene\/analysis\/uk\/ukrainian.dict" \/>/<!-- filter class="solr.MorfologikFilterFactory" dictionary="org\/apache\/lucene\/analysis\/uk\/ukrainian.dict" \/ -->/' /tmp/backup/SmartSearch/managed-schema

  9. Upload the new configuration:

    /opt/solr/server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181/solr -cmd upconfig -confdir /tmp/backup/SmartSearch -confname MANDATOR

  10. Delete the previously created folder /tmp/backup/SmartSearch:

    rm -rf /tmp/backup/SmartSearch

  11. Start both the Solr and SmartSearch server as well as the reindex tool. From now on search queries are executable again.

    sudo systemctl start solr
sudo systemctl start smart-search
java -jar reindex-tool.jar

  12. In the Solr backend, determine the names of the new collections for data and signals.

  13. Verify that the status of both collections is healthy. In the following call the parameter -c sets the name of the collection.

    /opt/solr/bin/solr healthcheck -c MANDATOR_20200612141120 -z localhost:2181/solr

9.3. Solr upgrade to version 8.1.1

From version 2.1.0 the SmartSearch requires Java 11. For this reason, Java version 11 must be guaranteed from this point on. Older Java versions are no longer supported.

The following description assumes that the Solr installation has been done into the /opt directory and that the data is located in the /opt/solr-data directory. It is also based on the assumption that the control of all services is done with systemd and that they are named smartsearch, solr and zookeeper.

With the change to Solr version 8, a configuration adjustment was made to the collections whose names are required for the migration: For the data collection, this corresponds to the company name from the license, which is subsequently referred to as the client or mandator. The name for the signals collection is composed of the client and the suffix _signals.

The following description assumes a running SmartSearch server and one or more Solr and ZooKeeper servers and is divided into four steps:

  • Solr upgrade

  • Importing the configuration of the data collection

  • Import of the configuration of the Signals Collection.

  • Completion of the migration.

Solr upgrade

  1. Download the Solr archive version 8.1.1 from official website.

  2. Unpack the installation/upgrade script:

    tar xzf solr-8.1.1.tgz solr-8.1.1/bin/install_solr_service.sh --strip-components=2

  3. Stop both the SmartSearch and Solr servers, and then run the install/upgrade script:

    sudo systemctl stop smart-search
sudo systemctl stop solr
sudo ./install_solr_service.sh solr-8.1.1.tgz -d /opt/solr-data -n -f

  4. Change the owner and group of the created symbolic link and folder:

    sudo chown -R solr:solr /opt/solr
sudo chown -R solr:solr /opt/solr-8.1.1

  5. Add the execute right to the zkCli.sh script to download the collection configuration:

    sudo chmod +x /opt/solr/server/scripts/cloud-scripts/zkcli.sh

Importing the configuration of the data collection

  1. Download the configuration to the /tmp/backup/SmartSearch directory:

    /opt/solr/server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181/solr -cmd downconfig -confdir /tmp/backup/SmartSearch -confname MANDATOR

  2. Create a backup of the managed-schema and solrconfig.xml files in the /tmp/backup/SmartSearch directory.

  3. Replace the managed-schema and solrconfig.xml files in the /tmp/backup/SmartSearch directory with the files of the same name in the main directory from the reindex tool zip file.

  4. Upload the new configuration:

    /opt/solr/server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181/solr -cmd upconfig -confdir /tmp/backup/SmartSearch -confname MANDATOR

  5. Delete the previously created folder /tmp/backup/SmartSearch. The data contained in it is no longer needed.

    rm -rf /tmp/backup/SmartSearch

Recording the configuration of the signals collection

  1. Download the signals configuration to the /tmp/backup/SmartSearch directory:

    /opt/solr/server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181/solr -cmd downconfig -confdir /tmp/backup/SmartSearch -confname MANDATOR_signals

  2. Create a backup of the managed-schema and solrconfig.xml files in the /tmp/backup/SmartSearch directory.

  3. Replace the managed-schema and solrconfig.xml files in the /tmp/backup/SmartSearch directory with the files of the same name from the signals directory from the Reindex tool zip file.

  4. Upload the new configuration:

    /opt/solr/server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181/solr -cmd upconfig -confdir /tmp/backup/SmartSearch -confname MANDATOR_signals

  5. Delete the previously created folder /tmp/backup/SmartSearch. The data contained in it is no longer needed.

    rm -rf /tmp/backup/SmartSearch

Completion of the migration

  1. Start both the Solr and SmartSearch servers and the reindex tool. From now on, search queries are executable again. The command requires the previously executed configuration of the tool.

    sudo systemctl start solr
sudo systemctl start smart-search
java -jar reindex-tool.jar

  2. In the Solr backend, determine the names of the new collections for data and signals.

  3. Verify that the status of both collections is healthy. In the following call, the -c parameter sets the name of the collection.

    /opt/solr/bin/solr healthcheck -c MANDATOR_20200612141120 -z localhost:2181/solr

10. Migration to version 2.0.62

The SmartSearch version 2.0.62 contains an internal update from Spring Boot 1 to 2. This update facilitates SmartSearch operation in the future. One example is the ability to monitor the SmartSearch server through a Prometheus endpoint.

This major version upgrade requires some configuration adjustments in the application.yml file. Further details are listed in the Spring-Boot 2-Migrationguide. Additionally, an improvement of the structure of the application.yml file was made.

The following list shows the changes made in the application.yml:

Removal of the key cache-period

The key spring.resources.cache-period in application.yml defines the cache control max age value for static files (for example JS or CSS). This key is no longer needed and is not included in the configuration anymore.

Exclusion of autoconfigurations

In the Spring context, both the MongoDB module and the MongoDB driver are disabled for the current time. As a result, the MongoDB-specific autoconfigurations are no longer needed and their activation must be excluded. For this purpose, the application.yml has to be extended by the following keys:

Keys for MongoDB autoconfiguration exclusion 
spring:
   autoconfigure:
      exclude:
      - org.springframework.boot.autoconfigure.mongo.embedded.EmbeddedMongoAutoConfiguration
      - org.springframework.boot.autoconfigure.mongo.MongoAutoConfiguration
      - org.springframework.boot.autoconfigure.data.mongo.MongoDataAutoConfiguration
      - org.springframework.boot.autoconfigure.data.mongo.MongoRepositoriesAutoConfiguration
UTF-8 file encoding

Trouble-free server operation requires UTF-8 file encoding. The configuration causes a check performed at startup whether the Java parameter file.encoding is set to UTF-8. If this is not the case, the startup is aborted. For the definition of the parameter the application.yml has to be extended by the following key:

Key for UTF-8 file encoding 
spring:
   mandatory-file-encoding: UTF-8
UTF-8 for communication with the server

To prevent problems with server operation, HTTP communication with UTF-8 must be ensured. For this purpose the application.yml has to be extended by the following keys:

Keys for the communication with UTF-8 
spring:
   http:
      encoding:
         charset: UTF-8
         enabled: true
         force: true
Disabling Spring Session

Due to a current lack of usage, Spring Session can be disabled. Disabling it requires adjusting the spring.session.store-type key, whose value should be changed from hash-map to none.

Configuration of the actuators

The configurations of the actuators have changed substantially. The Spring Boot 2 changelog lists the details.

The old key in application.yml is no longer taken into account and is therefore no longer present:

Obsolete key for the actuators 
management:
   contextPath: /api/boot

It is recommended to reset the {ref_monitoring}[configuration of the actuators].

11. Migration fron Version 1.9 to 2.0

With SmartSearch version 2, basic technologies have changed compared to version 1, making some manual steps necessary. This enables some features that are necessary for modern enterprise software, for example hot standby.

This chapter describes the steps required to migrate the configurations from version 1 to version 2. If the current version corresponds to a version prior to 1.9.x, an update to this version must be performed first.

The migration assumes that an old and new SmartSearch instance exist and that access to the SmartSearch cockpit and the file system is possible on both instances. The new SmartSearch instance has already been started once and thus all necessary data in the ZooKeeper and Solr are initialized.

11.1. Important changes in version 2

With version 2, the previously embedded Solr server was switched to an external server. This must run in so-called cloud mode. This change makes it possible to operate all components redundantly and thus to set them up in a fail-safe manner.

The Solr schema used has been fundamentally revised. For example, it now has a separate field for each supported language in order to process the content language-specifically. This is transparent in the search query. For example, if the title field of a Prepared Search is defined for searching, the actual search query considers the fields of all existing language variants, but returns only the value as title in the end. This way, the language used does not need to be known when evaluating the response of the Prepared Search REST API, which makes integration much easier.

Further, it is no longer possible to use the REST services without user credentials. For this, it makes sense to create a technical user. Additionally, the REST API now uses HTTPS to further increase security.

11.2. Data generator and Prepared Search

With SmartSearch version 2 the location of the configuration has changed from a local database to ZooKeeper. For this reason, a transfer of the existing configuration to ZooKeeper is required. The takeover of the existing configurations for data generatoren and Prepared Searches requires the following manual steps.

In SmartSearch cockpit version 1.9.x the Exportiere Datengeneratoren-Konfiguration button on the data generatoren page enables the download of the current data generatoren configuration. Similarly, the Export Prepared Searches Config button on the Prepared Search list page enables the download of the current Prepared Search configuration. The expected names of the two files are datagenerators.xml and preparedsearches.xml.

The import expects both files in the SmartSearch 2 directory in the migration folder. The optional key haupia.migration.path in the application.yml defines an alternative path. If the files are present in the correct directory and the SmartSearch is started, the migration is logged.

The imported configuration requires a check of the data in the SmartSearch cockpit for completeness. After the migration is complete, the migration files are no longer needed and should be removed.

The imported configuration requires data verification Migration of Prepared Searches and data generatores checks if configurations with the names already exist and does not overwrite them.

Since Prepared Searches and data generators reference each other, parallel import of all Prepared Searches and data generators is advisable to avoid invalid intermediate states.

11.3. Stopwords

Version 1.9.x stored the stopword in the stopwords.txt file in the SmartSearch directory under solr/core0/conf. The new version moves from a global stopword file to one configuration per language, each stored in the ZooKeeper. The migration assumes that the formerly global stopword data is associated with a language. The following description migrates the stopword to the language German (de).

Storing the stopword in the ZooKeeper allows all Solr nodes to access the data. This procedure is referred to as Managed Resources. The naming of the Managed Resources is done by the SmartSearch based on the prefix stopwords_general_ and the language abbreviation. In the case of the German language, this results in the name stopword_general_de. The following URL provides the current stopword for this Managed Resource:

http://SOLR_URL:8983/solr/COMPANY/schema/analysis/stopwords/stopwords_general_de

The COMPANY in the URL is a placeholder for the company name from the license.

Saving as Managed Resource requires conversion of the global data to JSON.

The following example shows the contents of a global stopword file:

stopword file 
der
die
das

The first step of the conversion is to remove all lines that start with #. The result is a text list whose entries all become one entry in a JSON list.

From the example, the following JSON results:

JSON list 
["der",
"die",
"das"]

The storage of this JSON list is done with a PUT call to the URL of the Managed Resource.

The import does not replace any existing data, but supplements it.

When calling PUT, the header Content-type:application/json must also be set.

11.4. Synonyms

Analogous to the stopword, a conversion to Managed Resources was also made for the synonyms. Unfortunately, a simple export and import process as for the stopword is not possible in this case, because the way of handling synonyms on Solr side is more complex. Therefore, the maintenance via the SmartSearch cockpit on the synonyms side has to be done manually for each language.

11.5. Taglib

The new version has adopted the taglib as much as possible. However, the conversion of REST services from HTTP to HTTPS and authentication make changes to the context tag necessary. The tag now has attributes to pass the username and password. The property file referenced in the propertyContext tag also contains additional fields to define the credentials.

The context tag no longer manages the definition of the SmartSearch-REST endpoint via the server, path, and port parameters, but combines them into the haupiaServer parameter. There is now also the optional ignoreSslErrors parameter, which ignores validations of the SSL certificate and whose use makes sense only in the development context.

The following code block shows the context tag in its previous usage as well as the adaptation to be made:

context-Tag 
// previous variant
<haupia:context server="localhost" path="haupia" port="8080">
</haupia:context>

// adaptation to be made
<haupia:context user="admin@host.de" password="s3cret" haupiaServer="https://foo.de:8181">
</haupia:context>

The tag propertyContext respectively the required data in the referenced property file change analogously.

11.6. JAVA-API

The Java API has changed fundamentally. To reflect the increased complexities of queries, the API now consists of fluid builders instead of static method calls. In addition, new external dependencies are added.

The following example contains the simplest variant for querying a Prepared Search and shows a comparison between the old and new approach.

JAVA-API 
// previous variant
final String haupiaRestBaseUrl = "http://localhost:8080/haupia-backend-3_0/service";
final String preparedSearchName = "preparedSearchName";
final String query = "*:*";
final HaupiaResult result = PreparedSearch.execute(haupiaRestBaseUrl, preparedSearchName, query);

// new fluid variant
final String preparedSearchName = "preparedSearchName";
final String query = "*:*";
final PreparedSearch preparedSearch = new PreparedSearchBuilder(preparedSearchName).query(query).build();
final PreparedSearchResponse response = PreparedSearchExecutor.execute(this.serverUri, preparedSearch, "user", "password");

11.7. External data generator REST-API

To make the API of external data generators independent of the programming language used, Java support for the external data generator has been removed. At the same time, as of version 2, the REST API for external data generators has properties to allow it to be used with any programming language. The documentation for the new API is in a separate document.

11.8. JSON response of Prepared Search

Unlike version 1.9.x, a Prepared Search in version 2 no longer returns the SLR server’s JSON as a response. Instead, version 2 introduces a new JSON format that requires an adjustment to how the JSON is used. The new format is both easier to read and required field revisions with the introduction of i18n support. The main differences are listed below.

The documents found are in another place.

found documents 
// old version: documents in response/docs
{
   "response": {
      "docs": [
         {
            "[elevated]": false,
            "uid": "WEBCRAWLER.haupia:1758918fda5f8e63b7ef26f7c88e4ffd",
            "_storage": "WEBCRAWLER.haupia",
            "link": "...",
            "title": "..."
         },...
      ]
   }
   ...
}

// new version: documents in results
{
   "results": [
      {
         "[elevated]": false,
         "uid": "WEBCRAWLER.haupia:1758918fda5f8e63b7ef26f7c88e4ffd",
         "_storage": "WEBCRAWLER.haupia",
         "link": "...",
         "title": "..."
      },...
   ]
   ...
}

The total number of results is in a different place.

totla number of results 
// old verison: total number of results is in response/numFound
{
   "response": {
      "numFound": 92
   }
   ...
}

// new version: total number of results is in numRows
{
   "numRows": 92
   ...
}

Since the configuration of the fields has changed fundamentally, different return values may occur: Fields, such as title, can be returned as a simple value or array. This depends on how many values the field has.

The new format in version 2 simplifies the return list of facet entries. The additional return of the key filterQuery also allows to apply the corresponding filter to the entry.

Facet 
// old version: example for facet
"facet_counts": {
   "facet_fields": {
      "keywords_facet_string": [
         "Content Management",
         32,
         "Digital Marketing",
         32
      ]
   }
}

// new version: simpler representation of a facet
{
   "facets": [
      {
         "name": "keywords_facet_string",
         "counts": [
            {
               "value": "Content Management",
               "count": 32,
               "filterQuery": "keywords_facet_string:Content\\ Management"
            },
            {
               "value": "Digital Marketing",
               "count": 32,
               "filterQuery": "keywords_facet_string:Digital\\ Marketing"
            }
         ]
      }
   ]
}

12. Legal hints

The SmartSearch is a product of Crownpeak Technology GmbH, Dortmund, Germany. Only a license agreed upon with Crownpeak Technology GmbH is valid for using the SmartSearch.

13. Help

The Technical Support of the Crownpeak Technology GmbH provides expert technical support covering any topic related to the FirstSpirit™ product. You can get and find more help concerning relevant topics in our community.

14. Disclaimer

This document is provided for information purposes only. Crownpeak Technology GmbH may change the contents hereof without notice. This document is not warranted to be error-free, nor subject to any other warranties or conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability or fitness for a particular purpose. Crownpeak Technology GmbH specifically disclaims any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document. The technologies, functionality, services, and processes described herein are subject to change without notice.