SmartSearch

In the SmartSearch cockpit, you can create data generators to capture searchable data.

There are three types of data generators:

Using a Prepared Search, you can configure the search result, for example by bundling content from multiple data generators into one Prepared Search.

In an Adaptable Result, you can actively modify the search results, for example to display certain search results higher in the list, or to exclude search results from the list.

You can specify a synonym to associate a term with the search that does not appear in the website text, but is meaningfully associated with another search term. synonyms can be derived from common search queries, for example.

You can use stopwords to ignore certain terms in the search query. A list of typical stopwords for each language is provided and can be extended.

The customer-side gathering of the required data is done by the so-called data generators, which are a part of the Data area. For their management, the SmartSearch provides the Prepared Searches. These allow optimizing the search results by prioritizing individual data.

The creation and administration of the Prepared Searches is done in the interface of the same name, which can be called via the menu entry Configuration  Prepared Search.

The area shows a list of all already existing Prepared Searches and is initially empty.

New Prepared Search

For the creation of a new Prepared Search there is a separate view, which can be called by clicking on the button New Prepared Search and is divided into the three tabs General, Facets and Preview.

List of existing Prepared Searches

After the Prepared Search is saved and successfully created, it is added to the list of existing Prepared Searches in the tab of the same name. The list is divided into two columns with the following information:

Editing a Prepared Search

The button with the symbol of a pen enables editing a Prepared Search. Clicking it opens the detailed view with all the configurations made. These can be changed according to the previously described possibilities.

Deleting a Prepared Search

The button with the symbol of a trash can enables the deletion a Prepared Search. Clicking it opens a dialog with a confirmation request to confirm the deletion.

Alternatively, it is possible to select several Prepared Searches for deletion via the checkboxes that are visible in the overview list in front of each Prepared Search. The button Delete selected removes these selected Prepared Searches after a confirmation request has been confirmed.

Every written text contains articles, filler words and other terms which are irrelevant for a search query and therefore should be ignored. For this reason, the SmartSearch has a list of so-called stopword for each of the 26 supported languages, which it provides in the Configuration  Stopwords menu area in the form of a drop-down menu. The terms contained in these lists are ignored by the data generators when the data is collected and are not included in the search index.

To edit a stopword list, first select it from the drop-down menu. The interface will then display all stopword contained in it.

The field Add new stopword allows the comma-separated specification of further stopword, which are transferred to the list by clicking on the button with the plus symbol. The buttons with the symbol of a pen or a trash can, which are visible per stop word, allow editing or deleting a single entry of the list.

Each change to the selected list finally requires being saved via the button with the symbol of a checkmark, which is displayed next to the drop-down menu. Only with this click the adaptations will be applied, even if they are already visible in the list.

Similar to stopwordn, which can be used to filter out irrelevant terms from the captured dataset, synonyms and substitutions allow for the addition of specific terms. In the menu area Configuration  synonyme there is the option to select a language in a drop-down menu. Below this there are two tabs for entering substitutions and synonyms. After the language has been selected, a list of substitutions or synonyms for the selected language appears in the selected tab. This list is initially empty and can be expanded using the corresponding input field.

Substitutions

This is a list of substitutions. Each entry maps a search term to a series of words. The 'Add new search term' field allows you to enter a search term. After confirming with 'Enter' or clicking on the button with the plus symbol, the word will appear together with an empty row of substitutions in the list below. Via the field 'Add new substitution' the corresponding substitutions can be assigned to the search term.

After saving, a search for the entered term will be replaced by the terms in the corresponding row.

Furthermore, the buttons with the symbol of a pen or a trash can, which are visible per search word, allow editing or deleting a single entry of the list.

Synonyms

This is a list of synonymseries. Entries of a synonymseries are treated as synonymous by the search. This means that a search for one term from a series will return results for all terms from the series. The field 'Add new synonyms' allows for the creation of a new series. Here it is also possible to enter several, comma-separated terms. In this case, they will appear as individual terms in a new synonymseries in the list below.

Any change made to the selected list finally requires saving it using the button labeled "Save" displayed next to the drop-down menu. Only with this click will the adjustments be applied, even if they are already visible in the list beforehand.

The Statistics area allows the analysis of the search queries of a specific Prepared Search. The field of the same name lists all existing Prepared Searches for this purpose, from which the Prepared Search to be analyzed is to be selected by a checkbox.

The following fields are used to specify the data to be analyzed:

Examples of the usage of tags can be found in the tags section.

The Adopt button performs the determination of the most frequent search words that result from the configuration made for the selected Prepared Search. They are then displayed as a list in the interface.

The Download CSV button provides both the defined configuration data and the search words determined on their basis in the form of a CSV file. In addition to this information, it contains a deep link and the absolute sum of all search queries - with and without filtering. The deep link enables the file to be downloaded again.

The CSV file which was generated from the statistics above contains the deep link (csv deep link) and the following statistic parameters:

The deep link contained in the CSV is also provided by the Copy to clipboard > Deep Link CSV button. The Copy to clipboard > Deep Link Cockpit button provides a link that can be used to call up the Statistics area again, including the configuration made. This way, both the settings and the result based on them can be made available to third parties.

The storage of the contents determined by the SmartSearch happens in a multiplicity of various fields. These can be assigned a weight during the configuration of a Prepared Search to prioritize them differently. This gives them a so-called score, which is calculated automatically and determines the order of a search result list. The manual control of these search result lists, the so-called Adaptable Results, is done in the area of the same name.

The area displays a list of all existing Adaptable Results, each of which matches the search query for a keyword of a Prepared Search.

Create an Adaptable Result

For the creation of a new Adaptable Result there is a separate view, which can be called by clicking on the button New Adaptable Result.

Within the view, a Prepared Search is to be selected and any search term is to be entered for which the order of the search results should be changed. A click on the button Adopt deactivates the two fields and, on the basis of the entries, determines the corresponding result list, which is displayed in the form of a table. The Reset button hides the table and reactivates the two fields.

In addition to the title, the link and the evaluated tokens of the displayed document, each entry in the table shows the score assigned to it, from which its position in the list results. This can be influenced by the buttons Elevate and Exclude which are also displayed for each entry:

The Add page manually field allows manual addition of further documents by specifying their URL. These must already be included in the index of the data generators assigned to the selected Prepared Search. The corresponding URL must be entered in the form http[s]://www.newdocument.com and confirmed with the plus button. If the document is included in the index, it will be added to the results list. Otherwise, a corresponding note will appear.

Each change to the search results list must be persisted using the Save button. Only with this click the adjustments are applied, even if they are already visible in the list before.

List of existing Adaptable Results

After the Adaptable Result is saved and successfully created, it is added to the list of existing Adaptable Results in the area of the same name. The list is divided into three columns with the following information:

Edit an Adaptable Result

The button with the symbol of a pen allows editing an Adaptable Result. Clicking on it opens the detailed view with the entries made. These can be changed according to the previously described possibilities.

Delete an Adaptable Result

The button with the symbol of a trash can allows to delete an Adaptable Result. A click on this button opens a dialog with a confirmation request to confirm the deletion.

Alternatively, it is possible to select multiple Adaptable Results for deletion using the checkboxes visible in front of each Adaptable Result in the overview list. The Delete selected button removes these selected Adaptable Results after a confirmation request has also been confirmed beforehand.

The SmartSearch provides so-called data generators for the customer-side collection of the required data. Their creation and administration takes place in the interface of the same name, which can be called up via the menu entry Data  Data Generators.

The area shows the list of all already existing data generators and is therefore initially empty. When creating further data generators, three types have to be distinguished:

Create a data generator

For the creation of a new data generator there is a separate view, which can be called by clicking on the button New Data Generator and is divided into three tabs. Besides the two tabs General and Enhancer, which are identical for all data generators, each of them has an additional specific tab.

The following sections first describe the two identical tabs before an explanation of the specific tabs follows:

The remaining settings allow you to specify the data generator’s search queries:

List of existing data generators

After the data generator is saved and successfully created, it is added to the list of existing data generators in the tab of the same name. The list is divided into a four-column table with the following information:

Edit a data generator

The button with the symbol of a pen enables editing the data generator. Clicking it opens the detailed view with all the configurations made. These can be changed according to the previously described possibilities.

Execute a data generator

The button with the play icon manually starts the execution of the data generator. It then collects the data from the configured source and transfers it to the Solr server.

Deleting a data generator

The button with the symbol of a trash can enables the deletion a data generator. Clicking it opens a dialog with a confirmation request to confirm the deletion.

Alternatively, it is possible to select several data generators for deletion via the checkboxes that are visible in the overview list in front of each data generator. The button Delete selected removes these selected data generators after a confirmation request has been confirmed.

Erasing the indexed documents of a data generator

The button with the symbol of an eraser can enables the emptying of a data generators data, deleting all documents from the index. Clicking the button Empty selected opens a dialog with a confirmation request to confirm the erasure of data.

For the display of the data collected by the data generator and persisted by the Solr server the SmartSearch provides a Browser. This can be called via the menu item Data  Browser.

For the display, a data generator must first be selected in the drop-down menu. The Browser then lists all documents that have been collected by this data generator. Clicking on such a document will also display all fields and the meta information stored in them. The Previous document and Next document buttons allow browsing through the captured documents, whose total number can be read between them.

In the course of data generation, it is often necessary to adjust the documents prepared for the index. This is required to increase the discoverability of documents, to remove invalid documents or to increase data quality in general.

The SmartSearch uses so-called Enhancer for this purpose. These are both supplied and project-dependently configured components that are part of a data generator configuration. During each data generation, the Enhancers are executed for each captured document and their effects are applied to the document to be indexed.

Generally, there are three types of Enhancers, which are executed in the following order:

The Enhancer tab in the configuration view of a data generator enables the creation of configured Enhancers. These are used to modify the data collected by the data generator before it is indexed on the Solr server. Per data generator configuration multiple Groovy script Enhancers can be defined. These are processed in the order in which they are stored in the configuration. The creation of an Enhancer is done by clicking on the New Enhancer button.

Groovy script Enhancer

This Enhancer allows the inclusion of a Groovy script to be implemented by the user. The editing mask required for this opens after creating the Enhancer via the menu item Groovy Script Enhancer. Such a script allows additional modifications of the dataset, and access to the data on the document generated during the execution of the general Enhancers. For example, additional documents can be added to the dataset or existing documents can be edited or deleted. An input made can be checked for possible syntax errors using the Validate button.

The following code example shows the structure of a Groovy script Enhancer:

The Groovy script is passed an object document, a string html and an object jsoupDocument:

If a document is not to be included in the index during generation, this can be achieved by defining the boost value 0:

Further examples of Groovy scripts and their usage are included in the Examples of Groovy Scripts Enhancers chapter.

Extracting names and nouns

According to its name, this Enhancer extracts names and nouns and stores them in the fields names_multi_keyword and nouns_multi_keyword.

With the SmartSearch a data structure - the so-called Solr-schema - is delivered, which defines the fields existing for captured documents. After a data gathering, these fields at the document contain structured contents, which are stored in the index in relation to the document. Since the contents of various fields potentially affect the results of a search query, some relevant fields are explained below. Some fields are automatically created and populated with values during the run of data generators. In addition, the Groovy Script Enhancer provides the possibility to add more fields and values.

Basically, a field consists of an identifier and one or more values. There are three types of fields, which can differ in their use and function:

A field can have one or more values depending on the field type. For more information, see the section "Fields with multiple values per document".

To use the SmartSearch it is important to understand the use cases and content of the available fields. The following tables give an overview of the three field types and the relevant part of the available fields and outline their content:

Mandatory fields

The identifiers as well as values of mandatory fields are obligatory on every document, the former are also unchangeable.

The following table shows an overview of fields of this type:

Static fields

The identifiers of static fields are unchangeable, just like those of mandatory fields. Unlike the latter, however, static fields do not necessarily have a value.

The following table lists some typical static fields:

Dynamic fields

Dynamic field identifiers follow a naming convention and may or may not be present depending on the usage and document. Dynamic fields are intended to make the schema flexible for future necessary data and thus the application more robust.

Dynamic fields behave like static fields except that their identifiers contain a placeholder. If documents are indexed by starting a data generation, data that cannot be assigned to a static field can be caught by a dynamic field.

For example, a document may contain a custom_date field that does not exist in the Solr schema. The data in this field would still be included in the index via the *_date dynamic field.

For more information on dynamic fields, see the Solr documentation.

An overview of the most important dynamic fields available can be found in the following table:

Static/dynamic fields for language abbreviations

In addition to the types of fields described above, further fields exist which can be present per language on documents and whose naming schemes are illustrated in the following table:

Fields with multiple values per document

When using SmartSearch, multiple values for a field name can be indexed for fields in certain cases. This creation of multiple values is not possible for every static and dynamic field.

Which static and dynamic fields support multiple values can be found in the table in section "Available fields and field types". User-defined fields, which are not mapped to any of the table values, generally allow multiple values.

The SmartSearch persists the content collected on the customer side in a variety of different fields, for the definition of which it provides the Prepared Searches. During execution, a language can be passed to these in order to perform the data collection in a language-dependent manner. The table below shows an overview of all languages currently configured for the SmartSearch, including their abbreviations (according to ISO 639-1) and the features available for them. The features are applied automatically when processing the captured information. More information about the individual languages is available in the Solr documentation.

Currently, the SmartSearch assigns the following features to the languages:

Access to the SmartSearch cockpit requires an active user. For this reason, a master admin exists for the initial login, with which further users can be created. The permissions of a user result from its group membership, which is to be defined during its creation. The creation and administration of the users takes place in the tab of the same name of the User Management.

The tab shows a list of all existing users, which initially contains only the master admin.

Create users

For the creation of a new user there is a separate view, which can be called up by clicking the New User button.

Within the view, the e-mail address of the new user must first be entered and a password must be defined. Furthermore, the user can be added directly to one or more groups at this point. Therefore, the drop-down menu Groups shows the list of existing groups. The assignment of the user to a group selected from the list is done via the plus symbol next to the drop-down menu. The trash can icon, which is visible for each assigned group, can be used to undo an assignment.

List of existing users

After the new user is saved and successfully created, it is added to the list of existing users in the tab of the same name. The list is divided into a four-column table with the following information:

With the exception of the master admin, the table also displays two buttons per user that allow editing and deleting the user.

Edit users

The button with the symbol of a pen enables editing a user. Clicking it opens the detailed view with all the information. In the view, the user’s group assignment can be changed by adding him to additional or removing him from groups. The assignment is done by selecting a group from the Groups drop-down menu and then clicking the plus icon. Removing the user from a group is done by using the trash can icon visible for each assigned group.

Deleting users

The button with the symbol of a trash can enables the deletion of a user. Clicking it opens a dialog with a confirmation request to confirm the deletion. The dialog contains a checkbox that is activated by default. It causes the removal of all references of the user to be deleted to his user account. Instead, they are replaced by the technical user ANONYMUS and thus anonymized. Deactivating the checkbox preserves the user’s references. They can be anonymized afterwards using the Remove references of deleted users button.

Alternatively, it is possible to select several users for deletion via the checkboxes that are visible in the overview list in front of each user. The Delete selected button removes these selected users after a confirmation request has also been confirmed beforehand.

Even when deleting several users, their existing references can be replaced by a technical user and thus anonymized. If this option was deactivated, the references can be removed afterwards using the Remove references of deleted users button. It provides the same function and anonymizes the references of all users that no longer exist by replacing them with the technical user ANONYMUS.

The access to the areas and functions of the SmartSearch cockpit requires different permissions. They are defined via groups to which the individual users are assigned. Initially, there is only one master admin, for which the necessary permissions for creating new users and groups must first be activated. The creation and administration of the groups is done in the tab of the same name within the User Management.

The tab shows a list of all already existing groups, which initially contains only the admin and user group.

Create a group

For the creation of a new group there is a separate view, which can be called up by clicking the New Group button.

Within the view a group name is to be assigned, which may contain only letters, numbers and hyphens. Furthermore, one or more users can be added directly to the group at this point. Therefore, the drop-down menu Users shows the list of existing groups. The assignment of a user selected from the list is done via the plus symbol next to the drop-down menu. The trash can icon, which is visible for each assigned user, can be used to undo an assignment.

In addition, the permissions of the group can be determined in this view. For this purpose, all areas of the cockpit are listed, for which the permissions Reading, Edit and Delete can be individually enabled or disabled via checkbox. For the areas data generator and Search the permission Execute is also available. The Activate all button enables the simultaneous activation of all displayed permissions.

List of existing groups

After the new group is saved and successfully created, it is added to the list of existing groups in the tab of the same name. The list is divided into a two-column table with the following information:

With the exception of the groups ADMIN and USER, the table also displays two buttons per group that allow editing and deleting the group.

Edit a group

The button with the symbol of a pen enables editing a group. Clicking it opens the group`s detailed view with all the information. In the view, the users assigned to the group as well as the permissions can be changed. The assignment of a user to the group is performed by selecting it from the User drop-down menu and then clicking the plus icon. Removing a user from the group is done by using the trash can icon visible for each assigned user. Disabling/enabling of permissions is possible using the checkbox displayed per permission.

Deleting a group

The button with the symbol of a trash can enables the deletion of a group. Clicking it opens a dialog with a confirmation request to confirm the deletion. The deletion removes the group and all permissions defined by it.

Alternatively, it is possible to select several groups for deletion via the checkboxes that are visible in the overview list in front of each goup. The Delete selected button removes these selected groups after a confirmation request has also been confirmed beforehand.

In principle, there are two different modes of operation and usage for the SmartSearch: The use of SmartSearch as a cloud service and the use of self-operated SmartSearch instances.

The usage scenarios of the APIs available in the SmartSearch as well as the parameters and invocation methods required for these scenarios are identical in both modes. However, the invocation target differs situationally:

Accesses to SmartSearch server APIs are subject to the user permissions described above. Which permissions must be configured for the accessing user for which type of access depends on the type and purpose of the access. In the following use cases, the specific permissions required are explicitly stated.

If authentication is required for an access, a corresponding HTTP header Authorization must be set and sent to the server. The content of this header must be "Basic" followed by the value of the expression username:password encoded as a Base64 representation. In the following example, the value YWRtaW5AbG9jYWxob3N0LmRlOmFkbWlu corresponds to the Base64 representation of the expression admin@localhost.de:admin.

The simple search is the basic functionality of the SmartSearch. For a newly created Prepared Search, it returns up to ten search hits by default. These are sorted by relevance. Based on the results, the search can be refined step by step.

The simplest form of the query for a Prepared Search mithras with the search string solar looks as follows. The string here corresponds to the value of the query parameter.

A possible response to this query might be as follows:

If a search returns a lot of search results, limiting them may help searchers to get the desired result. To achieve this, the SmartSearch allows restrict the results. To restrict on specific areas facets are used. facets provide the ability to narrow hit lists by fields that are present in a document. The configuration of facets is done in the Prepared Search.

The following example shows a query for the term solar for the Prepared Search mithras, whose result contains only German hits. The restricting is achieved by the facet language which is passed as search parameter.

Among other things, the configuration dialog of a Prepared Search has the Hits per page field. The value defined for this field determines the maximum number of hits that a simple search query will return. They correspond to the first page of the pagination. To get the search results of additional pages, the haupia_pageNumber parameter must be added to the query. Counting starts from 0, so the second page, for example, requires passing the value 1.

The haupia_pageSize parameter allows to override the number of search results per page configured in the Prepared Search. The following query returns n results starting from the result ((m-1)*n)+1.

In addition to classic pagination, the SmartSearch offers the option of scroll pagination, which focuses on infinite scrolling. In this case, additional search results are loaded as soon as the end of the currently displayed results is reached.

A newly created Prepared Search returns a maximum of ten results by default. To get the next ten hits, it is necessary to define a starting point of the results. This can be set using the haupia_start parameter.

The haupia_rows parameter allows to override the number of search results per page configured in the Prepared Search. The following query returns n results starting from the m result.

The simplest form of an autocomplete query for a Prepared Search mithras for the string "sun" looks like the following. The string corresponds to the value of the prefix parameter.

If such a query has been sent by users, the response (depending on the database) may look like the following:

The previous example considers the case where an autocomplete query is language-independent across all documents. However, this may result in terms from multiple languages present in the index being suggested:

By adding the language parameter to the query, it is possible to restrict to a specific language:

Thus, the preceding exemplary query for the language German with the abbreviation de leads to a corresponding result:

In some use cases, it may be desired to expand the suggested terms across spaces rather than words separated by spaces. A typical example are search terms like "model 4A version 2", "latest news" or "cancel contract".

If the autocomplete function should suggest such multi-word terms, there are basically two ways to implement this:

Multi-word autocomplete function for undefined content

For a two-word autocomplete across a space, users have the option of using dynamic fields of type _expandend_autocomplete. These are automatically populated during a data generation for the content field, and can be retrieved by appending the _expanded suffix via the field parameter:

If the contents of other fields are to be used for such a multi-word autocomplete, then the contents must be transferred to a corresponding field of type *_expanded_autocomplete_[language abbreviation] during data generation in a Groovy Script Enhancer. In this example, a free text is copied to the text_expanded_autocomplete_en field:

Users can now send a query against this field (without the _autocomplete_[language abbreviation] suffix):

The response to this query includes a space-crossing autocomplete if corresponding terms are present in the indexed data:

Multi-word autocomplete function for defined terms

As a search provider, in some cases it might be helpful to be able to control the word suggestions very precisely. For example, if the indexed contents are products, completing the input "mod" should possibly offer the suggestions "model 4a version 1" as well as "model 4a version A".

To achieve this behavior, the language-independent dynamic field _keyword_lc is available. In order to apply such a field in a Prepared Search a data generator need to populate the field. This is done by applying a Groovy Script Enhancer. The following example Groovy Script code sets the value of the dynamic field model_keyword_lc to model 4a version 1:

Now, if the index contains documents with the values model 4a version 1 or model 4a version 2, a user of the autocomplete function can specifically use the field model_keyword_lc to complete the input "mod" by using the parameter field:

Unlike the previous examples, where only matching excerpts from field contents were returned, the response to such a query includes all complete values of the passed dynamic field. In the current example, the response could look like the following:

The autocomplete function does not suggest words that are contained in stopword lists. If changes are made to a stopword list, re-indexing by a data generator run is necessary for the change to be reflected in the list of delivered autocomplete suggestions.

It is advisable not to send autocomplete queries to the SmartSearch for prefixes with less than three letters. Although possible in principle, queries with such short prefixes may cause an increased load. Also, the suggestions in the responses of such short queries are often not useful, since the number of possible suggestions is still very long and thus not very narrowing.

In principle, the suggestion lists are extendable by so-called filter queries. Such filter queries further restrict the list of completion suggestions by reducing the set of underlying documents by a query clause. This can be achieved by extending the fq parameter with such a clause. Accordingly, the following example will only display suggestions from documents in whose indexed content the mime_type field contains the value 'html'.

Adding a single document is done by calling the http method PUT towards a URL which contains the name of the data generator as well as the unique document ID. The explicit specification of the language is optional:

https://localhost:8181/rest/api/v1/smartsearchconnect/myDatagenerator/00001

https://localhost:8181/rest/api/v1/smartsearchconnect/myDatagenerator/de/00001

The structure of the JSON document to be sent is a flat structure of fieldname-value pairs, whereas values can also be arrays of values, as can be seen in the following exemplary document:

For example, the following invocation can submit this document towards the API:

If multiple documents are to be indexed, multiple JSON documents can be passed to a http-method POST. In this case, besides bracketing the documents into an array, the JSON structure differs by the necessary addition of a respective document ID uid, since this is not part of the POST URL:

For example, the corresponding cURL call would look like the following:

To remove a document, a call must be made with the http method DELETE to the same URL that could be used to index the document:

data generators of type API are created automatically when a first document is sent and no datagenorator with that name exists. Operations with the generic API require a data generatorname under which they are subsequently configurable in the cockpit, for example to add enhancers. Such a data generator is of type API, and does not need to be started or stopped in the cockpit. Also, when the data generator is automatically created as mentioned, an identically named Prepared Search is created, and linked to the data generator.

Operations against the generic API do not take effect immediately in the index, but after a very short time in the range of minutes. A more precise time specification is not possible due to technical aspects of the indexing process.

The generic API is partly also used the integration of FirstSpirit and the SmartSearch Connect module. Thus, occasional references to this usage can be found in the context, such as references to corresponding mandatory fields, for example fsTitle. However, these annotations can be ignored for FirstSpirit-independent usage of the API.

The hot injection is, in the context of the SmartSearch, the fastest way to add data to the index of a data generator. Data added in this way is available in the index after fifteen minutes at the latest.

Basically, the HTTPS call to use the hot injection takes place at the endpoint defined as follows:

The HTTP status code of the response to such a request normally corresponds to one of the following values:

The format of the data passed in a hot injection request depends on the type of the data generator to which the data is added.

Submit to a Web data generator

The Web data generator accepts a list of URLs to be processed. Their processing is done in the same way as if they were captured by the Crawler as part of a data generation. The list of URLs can be passed either as line separated text or as a JSON array:

Depending on the format chosen for the transfer, the HTTP header Content-Type must be set at the request. This must have either the text/plain or application/json value. By default, the SmartSearch takes the value text/plain, which assumes the passing of a line separated URL list.

Furthermore, the following points must be noted when transferring URL lists:

Submit to an XML data generator

For an XML-data generator, the hot injection accepts a list of URLs in the same way as for a Web data generator. These can consist of both data XML files and seed XML files. Any mixture of data XML files and seed XML files is also possible, as shown in the following example from the XML data generator chapter:

With the hot injection of an XML data generator it is also possible to pass data directly. To do this, the content type text/xml must be set and the body of the query must conform to the XML format of the XML data generator:

The corresponding request to this document structure might look like this:

The hot deletion allows quick and immediate removal of documents from the index. Its call takes place at the following endpoint:

Passing one of the following information defines the document to be deleted. The information must be passed as the JSON body of the call.

The HTTP status code of the response to such a request normally corresponds to one of the following values:

The creation of a facet is done at a data generator by generating a field with the suffix _keyword (for example color_keyword). This can be done using a Groovy script Enhancer, for example. The fields created this way appear in the edit view of a Prepared Search once the corresponding data generator is selected. They are marked with the term facet.

The following code example shows a Groovy script that creates a facet named content_category.

The SmartSearch responds to each search query with JSON data that already contains all the necessary information for faceting. The following snippet shows such a server response to a search query in abbreviated form.

The facet color can take three values in this example: red, green and blue. For each value, the key urlFilterQuery specifies the exact string with which to expand the query URL to limit the results to that facet value. For example, restricting the results to red shoes requires the following query URL:

In addition, the value of the multiSelect key in facetConfigs indicates whether the corresponding facet allows multiple values.

The facet language is a special case because extending a query with the ?language URL parameter (instead of ?facet.filter.language) also allows to filter for a specific language.

To set a filter query in the Groovy script, it must be added to the SolrQuery. This is done using the addFilterQuery method. For example, to have only news in the search hits that are not older than one week, the call in the Groovy script must look like this:

Multiple filter queries can be added.

The use of filter queries in the Groovy script is useful if the range of values to be searched should not be influenceable by the user. If, on the other hand, the value of the filter query is to be flexible, it is a good idea to use a parameter that is appended to the request

As an alternative to the usage in a groovy script, filter queries can be added to the request as parameters. The parameter is called fq. For example, to only include products in the search hits that are not older than one week, the request must look like this:

The parameter fq can be passed multiple times.

When using SmartSearch.js, the filter queries can be passed using parameters as follows: .Filter query usage at SmartSearch.js

To better understand filters applied to fields of type _date, here are some examples:

If the field is a field of type _date_range, then a filtering for certain years can be implemented e.g. like this:

More detailed information for the use of filters in relation to date fields can be found in the SOLR documentation.

To demonstrate a spatial search using SmartSearch, a collection of documents containing cities’ names and their respective geo-coordinates will be taken. The goal is to perform a spatial search and find all cities within a certain distance from a given point or to sort by the distance.

For this example, we will add the geo-coordinates of three random cities to our documents: Munich, Dortmund, and Paris. We will use the Groovy Script Enhancer to add a field named "city_pnt" containing the geo-coordinates of the respective city to each document.

Then we perform a spatial search to find all the cities located within 100km of Augsburg (about 60km west of Munich) and another search to sort by the distance to this city.

To perform a spatial search, the documents must contain a field ending with "_pnt" in the field name and the latitude and longitude values in a comma-separated format.

In the example below, a Groovy Script Enhancer adds such a field during indexing:

In the example below, * a new field named "city_pnt" containing the respective city’s latitude and longitude values is created. * a new field named "city_keyword" is created to check the search results for correctness.

Add a Groovy Script to a Prepared Search to perform a spatial search. The two examples below use the data specified above:

In both examples, the "pt" parameter specifies the point around which the search is performed, and the "d" parameter specifies the distance in kilometers.

In the first example, the "fq" parameter specifies a filter query, which uses the geofilt function to perform a spatial search. The "sfield" parameter specifies the field that contains the latitude and longitude values.

In the second example, the "fq" parameter specifies a filter query, which uses the bbox function to perform a spatial search. The "sfield" parameter works the same way as in the geofilt example.

The main difference between bbox and geofilt is the shape of the search area. Geofilt performs a circular search around the specified coordinates, while bbox performs a rectangular search.

The choice between bbox and geofilt depends on the specific needs of your system. The following statements are generalizations, and the actual performance and accuracy of each approach may depend on the specific implementation and used data:

You can also sort the search results by distance from a specified point. There Groovy Script example below features a spatial search and sorts the results by distance:

In the example above, the "fq" parameter specifies the filter query, which uses the geofilt function to perform a spatial search. The "sfield" parameter specifies the field that contains the latitude and longitude values. The "pt" parameter specifies the coordinates around which the search is performed, and the "d" parameter specifies the distance in kilometers. The "sort" parameter specifies the sorting order. It uses the geodist function to sort the results by distance in ascending order.

The examples above only show some possibilities of a spatial search with SmartSearch. Many other approaches and techniques can be used depending on the specific needs of your system. For more information about spatial search with Solr, refer to the Solr documentation at https://solr.apache.org/guide/8_6/spatial-search.html.

The SmartSearch works with a consistent user and rights system. New users are created and managed via the user management. After creation the user is known on the server and can log in (with a valid login) via the SmartSearch cockpit. Access to the configuration elements is granted via group rights/roles.

The SmartSearch uses log files to track actions and events on the SmartSearch server. Log files are collected to maintain secure operations. They can be used to analyze and troubleshoot error states.

Each time an editor makes a change to configuration items (for example, an Prepared Search), it is noted on it who made the last change and when. This overwrites any existing last change.

+ When a new configuration item is created, it notes who created the item and when.

Cookies are information that can be stored in the browser of the viewer about a visited website.