Automatic image cropping

Often, image material must first be cropped so that it can be optimally used for a website: For example, if the proportions of the original image do not fit the proportions and sizes needed for the website. Or if, in the case of images with several motifs or objects, a particular section is to be placed in the center.

In addition, FirstSpirit works with so-called “resolutions” (see Resolutions (→Documentation for Administrators)): A suitable section must be selected by the editor for each resolution of an image.

The “FirstSpirit Image Recognition” module offers the editor support in image processing to varying degrees: fully automatic cropping or motif recognition. Alternatively, the module can also be used to prevent compressed images if, for example, resolutions with a fixed aspect ratio are present in the project.

Live and in color: See "Smart Cropping" as video

The module "FirstSpirit Image Recognition"

This module integrates AI-assisted image recognition (“Artificial Intelligence”) into the editorial processes so that faces, objects and text on images are automatically recognized. The information is stored as “tags” for the images (along with generic terms and synonyms), in form of FirstSpirit Metadata, e.g.

Person, Laptop, Door, Chair, Pants, Clothing etc.

They are available in the FirstSpirit search and can be displayed in the Client with the image for better identification. This facilitates quick retrieval and supports efficient organization of the images.

The module not only knows what is on an image, but also where: The tags are linked to coordinates on the image.

The images are analyzed using the image recognition service “Amazon Rekognition”. This returns tags and coordinates for the recognized motifs of an image. These are then stored in FirstSpirit if the metadata configuration for the analyzed images is correct.

Fully automatic cropping ("AutoCropping")

Based on this information, images (including all resolutions) can be cropped fully automatically during upload. Artificial intelligence recognizes the subjects in each picture and determines the main subject using a strategy that can be adapted to specific requirements. With this information, FirstSpirit automatically creates suitable crops that look both harmonious and professional. This is how FirstSpirit can generate the right format for all your online channels using any type of source image material – without manual intervention being required.

For automatic cropping, subjects are recognized in the following order by default:

• Face
  If a face is recognized, it is used as the focus area.
• Several faces
  If several faces are recognized, the focus area frames all the faces.
  (“Around 4 faces”)
• Object
  If an object is recognized, it is used as the focus area.
• Several objects
  If several objects are recognized, the focus area frames all the objects.

This order or the analysis focus can be influenced by creating media folders with appropriate metadata: If an image is uploaded to the folder in question, priority is given to recognizing the subjects defined in the metadata.
See also Influencing recognition and automatic setting of the focus area.

Note: In some cases, analysis performed by the Amazon Rekognition API returns tags, but these cannot be located. This is not an error on the part of FirstSpirit. You are only provided with focus areas for subjects that can be located by the Amazon Rekognition API.

Detection of motifs ("Focus areas")

Alternatively, instead of automatic cropping, you can also use the analysis function separately to automatically recognize subjects / motifs on images (also called “Focus areas”). These are identified by frames, and you can conveniently select the desired section with just one click. Crops for all required formats are generated automatically. For maximum control and accuracy, you can fine-tune the crops manually.

Restrictions and recommendations

• Legal: When using “FirstSpirit Image Recognition” and a corresponding project configuration, images are analyzed by Amazon Rekognition during upload to FirstSpirit or by manually activating the function. The use of the service is GDPR compliant.
• Financial: In general, there is a charge for using Amazon Rekognition.
  However, you already have a monthly quota of analyzed images included in your cloud offering, which is sufficient in most customer scenarios.
  Any additional costs will be billed separately.
• Image formats: Currently the JPEG, PNG and WebP image formats are supported.
  No metadata is stored for images in other formats. This means that automatic cropping is not possible, and no automatically determined focus areas are available
• Image size and resolution: For optimal analysis results, Amazon Rekognition makes recommendations on the resolution and minimum size of images and the faces and objects to be recognized. Blurry images can affect the quality of the analysis result. See also Documentation about "Amazon Rekognition".
• Amazon-supplied data: The logic used to analyze images is based on Amazon Rekognition's APIs. Crownpeak has no control over if or what is recognized on images.
  Crownpeak can attempt to correct any programming issues that arise during image analysis using the Amazon API – there is no entitlement to this.
• Language: The data returned by Amazon Rekognition is in English.

Module: Installation and configuration

This step is only necessary for non-cloud customers.

Non-cloud customers must first install the module:

imagerecognition-module-[version].fsm

(See also Modules (→Documentation for Administrators).)

If the “FS ImageRecognition - AWS Connector” service is started (default setting), images are created with the appropriate aspect ratio during upload: There is no more potential compression of the images.

• If you want to display focus areas in analyzed images, corresponding metadata fields must be defined in the metadata template of the respective project (see Template development).
• If you want images to be cropped automatically, you must also activate the “Automatic cropping enabled by default” option: either globally for all projects on the server (see the following step) or for individual projects (see Project component).

Configuration of the “FS Image Recognition - AWS Connector”

Automatic cropping enabled by default: To enable automatic cropping of images server-wide, this checkbox must be selected. By default, images are not automatically cropped.
However, this global setting can be overridden by a corresponding setting in a project (“FS ImageRecognition - Configuration” project component, “Automatic Cropping enabled” checkbox).
See also Deactivating /activating the functionality.

Access key: Access key of the respective IAM user (IAM = AWS Identity and Access Management, see AWS Documentation (IAM)).

Secret key: Secret key of the respective IAM user.

S3 Bucket name: (optional) For analyzing images larger than 5 MB, a so-called “bucket” is required. The images are only temporarily stored there.
For further information please refer to AWS Documentation (Buckets).
If this field remains empty, S3 is not used and only images with a size < 5 MB can be analyzed.

Preferred folder name: (optional) An AWS folder can be specified where images uploaded to FirstSpirit will be saved.
Permitted characters: Numbers, letters, . (dot), - (hyphen) and _ (underscore)
Folder names must start and end with letters or digits and be at least two characters long.
By default (if nothing is specified in this field), images are stored at the top level of the configured S3 bucket.

AWS Region: The default uses the automatically detected region if it is an AWS host, otherwise Ireland.

Open IAM console: Opens the IAM console.

Test configuration: Tests the configuration.

Adding the web component
Then the “FS ImageRecognition - ContentCreator Resources” web component must be added to the global ContentCreator WebApp.

The WebApp must then be updated / deployed (Active web server: button “Update”).

See also Web applications (→Documentation for Administrators).

Activating or deactivating the functionality

The “FirstSpirit Image Recognition” module encompasses several sub-functions. By default, there is no automatic cropping of images, nor are there any suggestions for focus areas. Images can be cropped manually, depending on requirements with system support or without (see Documentation for the ContentCreator).

To be able to take advantage of the functionalities of the module, configuration settings are required, which are described below (via project component and/or metadata).
If required, you can activate or deactivate all subfunctions of the “FirstSpirit Image Recognition” module on a project-by-project basis.

In general, the prerequisite is that the “SmartCropping” option is activated for the corresponding resolution in the Project Properties (see Resolutions (→Documentation for Administrators)).

Legend
Metadata: The data returned by Amazon Rekognition is stored in image metadata fields. For this purpose, these metadata fields must be defined with specific identifiers in the metadata template.
See Template development.
Images are only analyzed according to content types that have defined metadata fields.

• “Metadata active”: At least one of the metadata fields is defined.
• “Metadata inactive”: No metadata fields are defined, and no analysis is performed.

Project component It is possible to control whether automatic cropping is applied by using a setting in the FirstSpirit ServerManager on a project-by-project basis in the “Project components” area.
By activating/deactivating the “Automatic Cropping enabled” checkbox in the project component “FS ImageRecognition - Configuration”, the function can be activated or deactivated again for the particular project.
(See also Project components (→Documentation for Administrators).)

Local project ContentCreator instances

In general, one ContentCreator instance is used per FirstSpirit server. In this case, the ContentCreator is also referred to as a “global” web application.

In some cases it may be preferable to create and operate separate ContentCreator instances for individual projects on a FirstSpirit server. Specific functions can then be provided on these instances, for example via a module. In this case, the ContentCreator is also referred to as a “local project” instance (see also Individualizing web applications (→Documentation for Administrators)).

We recommend using the ContentCreator as a global web application due to resource as well as maintenance reasons.

If, nonetheless, the image recognition function is to be configured (for example for test purposes) just for a local project ContentCreator, the web component “FS Image Recognition - ContentCreator Resources” must be added to the required instance. This is done in the FirstSpirit ServerManager, Project properties / Web components / area “ContentCreator”). See also Web components (→Documentation for Administrators).

After adding the web component, this change must be deployed (“Update” or “Install” button). During deployment, the corresponding ContentCreator is briefly not available to editors.

Template Development

Metadata template

The data that Amazon Rekognition returns is stored as metadata for each image.
To do this, the project needs to have a metadata template (see also Options (→Documentation for Administrators)).

For images to be analyzed for the following information, components of type CMS_INPUT_TEXTAREA must be created with the following identifiers:

• fsai_facesPayload: This field stores information returned by Amazon regarding recognized faces for each image.
• fsai_objectsPayload: This field stores information returned by Amazon regarding recognized objects for each image.
• fsai_textPayload: This field stores information returned by Amazon regarding recognized text for each image.
• fsai_unsafeContentPayload: This field stores information returned by Amazon regarding potential inappropriate content for each image.
• fsai_celebritiesPayload: This field stores information returned by Amazon regarding recognized celebrities for each image.
• fsai_tags: This field stores the “tags” (including generic terms and synonyms) returned by Amazon for each image. This data can be used for tagging.

Recommendation: Each analysis of an image against one of the categories listed above represents a chargeable request to the corresponding Amazon API. Therefore, you should only collect the metadata that is relevant for your images and your specific use. The fields fsai_textPayload, fsai_unsafeContentPayload and fsai_celebritiesPayload are usually only required for specialized application cases.

Influencing recognition and automatic setting of the focus area

A face in an image is not always the main subject. To influence which image components should primarily be used as the focus area, it is possible to assign Media folders with suitable metadata. Depending on which folder an image is uploaded to, the object specified by the metadata is then primarily used as the focus area.

For this purpose, an input component of type CMS_INPUT_TEXTAREA with the identifier

fsai_focusPreference

must be used in the metadata.

The necessary terms must be entered in the metadata form of the relevant media folder. You can check which term it is in each case using the metadata of images that have already been analyzed. These are stored in the fsai_tags metadata component.

Example

Car

For better results, you can also use multiple terms, so as not to limit the image recognition too much.
If multiple terms are used, they must be separated by commas, e. g.

Tablet Computer, Mobile Phone

Only objects for which a focus area is recognized will be considered. The terms are analyzed in the order given (this means that the first term is prioritized the most).

By default, metadata defined on a folder also apply to any subfolders. In this way, metadata for hierarchically dependent subtrees can be conveniently reused. However, inheritance can also be disabled. See Documentation for the ContentCreator or Metadata (→Documentation FirstSpirit SiteArchitect).

If in fsai_focusPreference no values ​​are entered, the default automatic focus area selection applies.

Example form:

<CMS_INPUT_TEXTAREA name="fsai_focusPreference">
    <LANGINFOS>
      <LANGINFO lang="*" label="FS AI - Focus Preferences"/>
    </LANGINFOS>
  </CMS_INPUT_TEXTAREA>

<CMS_INPUT_TEXTAREA name="fsai_facesPayload">
 <LANGINFOS>
    <LANGINFO lang="*" label="FS AI - Faces Payload"/>
 </LANGINFOS>
</CMS_INPUT_TEXTAREA>

<CMS_INPUT_TEXTAREA name="fsai_objectsPayload">
 <LANGINFOS>
    <LANGINFO lang="*" label="FS AI - Objects Payload"/>
 </LANGINFOS>
</CMS_INPUT_TEXTAREA>

<CMS_INPUT_TEXTAREA name="fsai_textPayload">
 <LANGINFOS>
    <LANGINFO lang="*" label="FS AI - Text Payload"/>
 </LANGINFOS>
</CMS_INPUT_TEXTAREA>

<CMS_INPUT_TEXTAREA name="fsai_unsafeContentPayload">
 <LANGINFOS>
    <LANGINFO lang="*" label="FS AI - Unsafe Content Payload"/>
 </LANGINFOS>
</CMS_INPUT_TEXTAREA>

<CMS_INPUT_TEXTAREA name="fsai_celebritiesPayload">
 <LANGINFOS>
    <LANGINFO lang="*" label="FS AI - Celebrities Payload"/>
 </LANGINFOS>
</CMS_INPUT_TEXTAREA>

<CMS_INPUT_TEXTAREA name="fsai_tags">
 <LANGINFOS>
    <LANGINFO lang="*" label="FS AI - Tags"/>
 </LANGINFOS>
</CMS_INPUT_TEXTAREA>

The component identifiers (attribute name) must be used accurately to ensure functionality.

Metadata in the ContentCreator

To view the information that Amazon returns for an image, you can generally use the FirstSpirit SiteArchitect (see also Metadata (→Documentation FirstSpirit SiteArchitect)).

To be able to view the metadata form in the FirstSpirit ContentCreator, this must be properly configured in the FirstSpirit ServerManager, Project properties / ContentCreator settings, “Metadata” area (see also ContentCreator settings (→Documentation for Administrators)).

Excluding resolutions

By default, the Image Recognition function is applied to all resolutions in a project.

If no automatic focus areas should be determined for one or more resolutions and therefore no automatic cropping performed, this can be configured for each resolution individually in the project settings.

This is done in the FirstSpirit ServerManager, Project properties / Resolutions / Option “SmartCropping” (see also Resolutions (→Documentation for Administrators)).

By default the “SmartCropping” option is activated for all resolutions in a project. This means that when using the “Image Recognition” module, a focus area is automatically defined and / or cropping is performed for each resolution.

If the option is deactivated, no focus area is defined and no cropping is performed for the resolution in question.

These resolutions can be found in the cropping dialog of each image in the “Manual croppings” group; editors can select the desired cropping manually for these resolutions (see also Documentation for the ContentCreator).

Data returned by Amazon

The data returned by Amazon is stored as metadata in the relevant metadata fields for that image. This is done in JSON format.

Detailed documentation of the data returned by Amazon Rekognition can be found in the corresponding Amazon Rekognition API:

• DetectFaces
  (recognition of faces)
• DetectLabels
  (recognition of objects)

Further information

• Resolutions: Resolutions (→Documentation for Administrators)
• Metadata template: Options (→Documentation for Administrators)
• Operation by the editor
• Amazon Rekognition Image