A/B-Testing

An experiment must be created before it can be carried out. Experiments can be created in various ways in the SiteArchitect and the ContentCreator.

Figure 2. Create experiment button in the SiteArchitect

In both clients, the A/B-Testing bar appears when the experiment is created. It contains two tabs: one for the original page and another for the variant created automatically. The variant is selected for immediate editing (see figure A/B-Testing bar in ContentCreator and in the SiteArchitect preview).

Figure 3. A/B-Testing bar in ContentCreator and in the SiteArchitect preview

Any number of new variants can be added to or existing variants can be deleted from a current experiment. Moreover, a distribution rate can be defined for each variant, along with a participation rate for the entire experiment.

Figure 4. Edit settings

Once an experiment has been created and edited, it can be started with the Start button. If the experiment starts successfully, the button turns green and its label changes to Running.

A current experiment can be stopped with the Finish experiment button (displayed as a flag). The selected variant is set as the new original page and the A/B-Testing bar disappears from view.

Use the abtesting-<version number>.fsm file supplied to add the module on the FirstSpirit Server. To install the module, open the ServerManager and select Server properties → Modules.

Figure 5. Module management in the server properties

The main panel contains a list of modules installed on the FirstSpirit Server. After clicking Install, select the abtesting-<version number>.fsm file supplied with the module and click Open to confirm your selection. After successful installation, an A/B-Testing folder is added to the list and must be given All permissions (see figure Module management in the server properties).

A number of templates and project-specific settings are necessary in order to use the A/B-Testing module. Templates can be imported and the remaining configuration settings can be made via the project component, which must be added to the project you are using. To add the project component, open the ServerManager and select Project properties → Project components.

Figure 6. Project components in the project properties

A list of all existing project components is displayed in the main panel. After clicking Add, select the A/B-Testing ProjectApp and click OK to confirm your selection. The project component is then added to the list in the main panel and will need to be configured (see figure Project components in the project properties). To configure the project component, select the entry in the list and click Configure to open the associated dialog (see figure Configuration dialog for the project component).

Figure 7. Configuration dialog for the project component

Workflow to start the experiment

First, specify a workflow to start an experiment in a combo box in the configuration dialog. This must be a release workflow that is to be created on a project-specific basis and releases all pages and page references involved in an experiment.

Name of the prioritized transition

A workflow often contains several outgoing transitions from a single status. In such cases, the user must decide which of these transitions is to be selected. The workflow pauses here until it is continued in the current status. Accordingly, the reference name of a transition to be prioritized can be specified for the workflow selected in the project component. This transition is executed by the workflow and the other transitions are ignored.

Selecting permitted page templates

The configuration dialog also displays a list of all page templates that are available to editors. This means that technical or similar templates that are hidden in the selection lists of the two clients are not displayed here. Select the permissible templates for carrying out an experiment from the list in the configuration dialog. The button or menu item for creating an experiment will be hidden in both clients for page references which have an underlying page based on a template that has not been selected. Use the key combination CTRL + A to select all of the templates in the list. In this case there is no restriction at all and an experiment can be created on every page reference.

Deleting pages

Next, the user must decide what happens to the pages of the individual variants when variants are deleted from a current experiment or when an experiment is finished. The corresponding checkbox Delete pages is deactivated by default. In this case, the pages of all variants are retained following the removal of a variant from a current experiment or after an experiment finishes. If the checkbox is activated, the pages of the variants that are not being used are deleted.

Importing templates

Finally, the various templates can be transferred to the project with the Import templates button. They provide the functions of the A/B-Testing module to the editor.

A web component must be added to the project. To add a web component, open the ServerManager and select Project properties → Web components.

Figure 8. Web components in the project properties

Inside the main panel, various tabs are visible, which contain a list of the existing web components. Select all the tabs in succession and click Add. Next, select the A/B-Testing WebApp and click OK to add it. The web component is added to the list in the main panel (see figure Web components in the project properties).

The web component must be installed on an active web server and then activated. The server can be selected using the selection box.

More detailed information on how to add web components is available in the FirstSpirit Documentation for Administrators.

When different variants are created for an experiment, the associated dispatcher page reference is saved with them. This assignment is made using the metadata.

The template must be specified in the project properties and has to exist in the project for this purpose. If the template does not exist, start by creating it. If the template already exists, open the ServerManager and select Project properties → Options. Then, click the corresponding button to select the metadata template (see figure Options in the project properties). Click OK to save the changes you have made.

Figure 9. Options in the project properties

Tracking during the runtime of an experiment is essential in order to analyze the success of each individual variant. Tracking is not part of the A/B-Testing module and must be carried out on a project-by-project basis. An analytics tool is required for this purpose (users are essentially free to choose which one). In this regard, the A/B-Testing module has a completely open architecture and is not limited to specific services.

The Google Analytics plug-in was added to the project with the import completed during the configuration of the project component. As its name suggests, this tracking plug-in is designed for Google Analytics and, therefore, requires a corresponding account.

Please refer to the supplier's documentation when configuring the analytics tool you have chosen to use.

An experiment consists of a dispatcher (created automatically) and any number of variants. Each variant must be assigned to the dispatcher page of the corresponding experiment. This assignment is made with the following input component:

Metadata input component. 

<CMS_INPUT_TEXT name="md_experiment_uid" hFill="yes" singleLine="no" useLanguages="yes" hidden="yes">
	<LANGINFOS>
		<LANGINFO lang="*" label="Dispatcher" description="Dispatcher"/>
	</LANGINFOS>
</CMS_INPUT_TEXT>

The input component is hidden. It must be added to the metadata template of the project you are using. The UID of the dispatcher page reference of the experiment is written to it automatically when a new variant is created.

If a metadata template does not already exist in the project, one must be created.

Select the metadata template in the project properties.

More information about metadata is available in the FirstSpirit Documentation for Administrators.

The functions of the A/B-Testing module are provided via four format templates. These templates must be referenced in the HTML code of all permissible page templates. The A/B-Testing Head, A/B-Testing Body, and Traffic Allocation Plug-in templates were added to the FirstSpirit project during the import completed when the project component was configured. The fourth template corresponds to the tracking plug-in to be integrated.

All four format templates are essential to the use of the A/B-Testing functions. They must be referenced in the page template you are using with CMS_RENDER calls:

Referencing format templates in the page template. 

<head>
   [...]
   $CMS_RENDER(script:"has_experiment", pageref: #global.node)$
   $CMS_IF(hasExperiment)$
      $CMS_RENDER(template:"abtesting_head")$
      $CMS_RENDER(template:"traffic_allocation_plugin")$
      $CMS_RENDER(template:"TRACKING PLUGIN REFERENCENAME")$ ❶
   $CMS_END_IF$
</head>
<body>
   $CMS_RENDER(template:"abtesting_body")$
   [...]
</body>

The JSTL tag library below and the JSP page directive must be integrated in addition to the format templates. However, unlike the format templates, both calls should be placed at the start of the HTML code.

If this is a JSP project, the lines may already be there.

JSTL tag library and JSP page directive. 

<%@ page language="java" contentType="text/html; charset=$CMS_VALUE(#global.encoding)$" %>
<%@ taglib uri="http://java.sun.com/jsp/jstl/core" prefix="c" %>

It must also be ensured that all variants involved in an experiment are JSP pages. To make sure of this, the file extension jsp must be defined in the Properties of the corresponding page template (see figure Defining a file extension).

Figure 10. Defining a file extension

It is essential for editorial permissions to be in place in order to execute the various functions supported by the A/B-Testing module. The table below explains which permissions are needed during the runtime of the experiment and which function belongs to each permission.

Select the root node of the PageStore or the SiteStore to distribute the editorial permissions. Then, right-click to open the context menu, select Extras → Change permissions, then open Permission assignment. Assign the relevant permissions to the editors of your project in the dialog that appears.

The workflow selected in the project component is executed on the dispatcher when carrying out an experiment via the A/B-Testing bar. However, it can also be carried out manually on the individual variants using the general FirstSpirit functions. In both cases, it is essential to release all pages and page references involved in the experiment in order to avoid errors during generation and when live.

The module is supplied with an example workflow which meets this requirement. However, it is also possible to adapt an existing project-specific workflow. Both options are full-fledged alternatives. They are described in the following sub-chapters.

5.4.2. Workflow supplied

The A/B-Testing module is supplied with a workflow which meets the requirements described. To use it, select it in the configuration dialog for the project component. Since this workflow is based on the BasicWorkflows, these are a prerequisite of the project.

Installing the BasicWorkflows module

If the BasicWorkflows are not already part of the project, before using the workflow supplied, you must install the BasicWorkflows module on the FirstSpirit Server and activate the web component. Proceed as described for installing the A/B-Testing module and activating the associated web component. However, the web component for the BasicWorkflows is only needed on the ContentCreator tab.

In addition, the workflow must be activated for ContentCreator by selecting the Element Status Provider supplied. To do this, open the ServerManager and select Project properties → ContentCreator. Then change the Element Status Providers entry to the BasicWorkflows Status Provider entry and confirm the change by clicking OK.

Figure 11. Selecting the Element Status Provider

Importing scripts and the workflow

Next, the BasicWorkflows scripts used for the release workflow must be added to the project. To do this, go to the Template Store in SiteArchitect and click to select Import from the context menu. This opens an import dialog box where you should select the export_basic_release import file from the workstation directories. Click Open to confirm; a second dialog is displayed listing all of the elements contained in the file. Since only the scripts are needed, the import of the release workflow can be deactivated (see figure Importing scripts). Finally, repeat the same process to import the workflow supplied with the A/B-Testing module.

Figure 12. Importing scripts

The imported workflow must now be authorized in the individual stores so that it can be executed on FirstSpirit elements. To do this, select Change permissions from the context menu of the stores' root nodes to call up the permission assignment. Next, on the Workflow permissions tab, activate the Authorized and Use release permission checkboxes for the imported workflow. To finish, click the OK button to close the dialog.

Functionality

Once installation and import processes are complete, the workflow can be used in the project. The workflow can essentially be executed on various elements:

Dispatcher

When an experiment is started, updated, and continued via the A/B-Testing bar, the workflow is executed on the associated dispatcher page. In this case, both the dispatcher and all of the variants associated with the element are released automatically.

Variant

The general FirstSpirit function can also be used to execute the workflow on a variant. In this case, the editor must confirm a security prompt in order to release the associated experiment. If the editor confirms this prompt, all variants and the dispatcher are released. If the editor does not confirm this prompt, the workflow is canceled and the variant remains in its previous status.

Only one workflow can ever be executed per experiment. This means: The workflow selected in the project component cannot be executed via the A/B-Testing bar if one of the variants is locked or already involved in a workflow. Manual execution of a workflow on a variant is not possible if the page or page reference of the associated dispatcher is already involved in a workflow. In both cases, the editor is notified accordingly and the corresponding workflow cannot be started.

If the workflow is executed manually using the general FirstSpirit function, it is necessary to also manually reload the preview in order to update the status of the A/B-Testing bar.

Element without reference to an experiment

If the workflow is executed on a FirstSpirit element that is not involved in an experiment, it will behave as per its standard functionality.

You can find more detailed information on the BasicWorkflows in the associated documentation.

The Google Analytics plug-in is supplied with the module. Designed for Google Analytics, as its name suggests, the plug-in is added to the project with the import completed during the configuration of the project component.

In addition to having a Google Analytics account, you simply need to add some information to the project settings template and select it in the project properties. It needs to be referenced in the page template too.

The plug-in also requires a Google Analytics Id and a Dimension Id:

All of the necessary steps are described below.

6.1.1. Registration and configuration of a Google Analytics account

A Google Analytics account is required in order to use the Google Analytics plug-in that is supplied with the module. If you do not have an account, you can register here: http://www.google.com/analytics/

A wizard guides you through the steps of the registration process that must be completed in order to create an account.

Figure 13. Registration of a Google Analytics account

Confirm the Google Analytics terms of use to complete the registration process.

More information about configuration is available in the Google Analytics help.

<head>
   [...]
   $CMS_RENDER(script:"has_experiment", pageref: #global.node)$
   $CMS_IF(hasExperiment)$
      $CMS_RENDER(template:"abtesting_head")$
      $CMS_RENDER(template:"traffic_allocation_plugin")$
      $CMS_RENDER(template:"google_analytics_plugin")$
   $CMS_END_IF$
</head>
<body>
   $CMS_RENDER(template:"abtesting_body")$
   [...]
</body>

6.1.3. Expansion of the project settings template

The Google Analytics plug-in supplied with the module uses the following input component, which must be added to the project settings template. If a project settings template does not already exist in the project, one must be created. The template must also be selected in the project properties.

Project settings input component. 

<CMS_GROUP>
   <LANGINFOS>
      <LANGINFO lang="*" label="A/B-Testing"/>
   </LANGINFOS>

   <CMS_INPUT_TEXT name="ab_googleid" hFill="yes" singleLine="no" useLanguages="no">
      <LANGINFOS>
         <LANGINFO lang="*" label="Google Analytics Id"/>
      </LANGINFOS>
   </CMS_INPUT_TEXT>
</CMS_GROUP>

Within the project, the Google Analytics Id must be added to the project settings by entering it in this field. The Id is used by the plug-in. It enables the variants involved in an experiment to be tracked by Google Analytics.

You will find the Google Analytics Id in your Google Analytics account on the following path: Admin → Account → Property → Tracking Info → Tracking Code. Figure 14. Path to find the Google Analytics Id

6.1.4. Specification of the project settings template

If you are using the Google Analytics plug-in, in addition to the metadata template, the project settings template also has to be added to the project properties. The template is used for the provision of the Google Analytics Id; it must exist in the project and be expanded as described in the previous chapter. If the template does not exist, complete these steps first.

If the template already exists, open the ServerManager and select Project properties → Options. Then click the corresponding button to select the project settings (see figure Options in the project properties). Click OK to save the changes you have made.

Figure 15. Options in the project properties

6.1.5. Specification of the Dimension Id

The Google Analytics plug-in adds a text box to the settings dialog of an experiment. Enter the Id of the Custom Dimension created in Google Analytics in this box.

To do this, click the Edit settings button to open the corresponding dialog of an existing experiment (see figure Edit settings). Then enter the Dimension Id in the designated text box and click OK to close the dialog and save your entry.

More detailed information about the Custom Dimension is available in the Google Analytics help under Dimensions and measured values.

Figure 16. Edit settings

If you are using a different analytics tool for tracking, this can be added as a tracking plug-in at any time on a project-by-project basis. In this regard, the A/B-Testing module has a completely open architecture and is not limited to specific services.

The following code corresponds to the basic implementation of this type of tracking plug-in. The general structure of the code is shown:

Base code. 

<script>
   var myPlugin=( function(){
      $-- init variables --$
      var myVar; ❶
      return {
         $-- add gui --$
         addGui: function (container) { ❷
            myVar=('myPlugin' in config && 'myVar' in config['myPlugin']) ? config['myPlugin'].myVar : "";
         },

         $-- store additional params --$
         storeParams: function () { ❸
            var addParams={
               $-- new parameter: store --$
               "myVar":myVar
            }
            return addParams;
         },

         $-- add analytics code --$
         addTrackingCode: function (variant) { ❹
            var trackingId='$CMS_RENDER(script:"getconfiguration", param:"myPlugin:myVar",
            srcUid:#global.node.uid)$';
         }
      };
   })();
   pluginRegistry.register('myPlugin', myPlugin); ❺
</script>

variable = ('pluginname' in config) ? config['pluginname'].variable : "";

var pluginname=( function(){
   $-- init variables --$
   var variable;

   return {
      $-- add gui --$
      addGui: function (container) {
         [...]
      },
      [...]
   };
})();

"<NAME>":<VALUE>

$CMS_RENDER(script:"getconfiguration", param:"pluginname:variable", srcUid:#global.node.uid)$';

pluginRegistry.register('pluginname', pluginname);

To check this initial assumption, a variant of the original page is created with the Actions → Create experiment menu item. This variant and the original page are displayed as tabs on the A/B-Testing bar which then appears, with the variant selected for immediate editing (see figure A/B-Testing bar in ContentCreator).

Figure 19. A/B-Testing bar in ContentCreator

At the same time, alongside the variant, a corresponding dispatcher page is created automatically in each of the PageStore and the SiteStore in SiteArchitect. The dispatcher page contains a list of all variants involved in the experiment; a reference is also created between it and the metadata of the variants. This generates a two-way assignment.

An experiment can only be created if experiments are allowed to be carried out for the selected page and it is not already involved in an experiment. If experiments are not allowed or if the page is already involved in an experiment, the menu item will not be visible. An error-free configuration and an adequate permission definition are also prerequisites. In its absence, the corresponding menu item will be visible in the ContentCreator but not activated. These rules also apply for the SiteArchitect, where the Create experiment button is used to create an experiment. Figure 20. Creating an experiment in the SiteArchitect It can only be used on page references in the SiteStore. Otherwise, it too is hidden from view. The A/B-Testing bar is then displayed in the preview. Figure 21. A/B-Testing bar in SiteArchitect

Any number of variants can be added by selecting the Add variant button (displayed as a plus sign) or using the duplicate option listed in the drop-down menu of each tab. A tab is displayed for each variant added. Only one variant is required for the story. This variant was created automatically as part of the process to create the experiment.

The changes required according to the assumption formulated above are made to the variant: a banner is added containing an image and a header. The teaser We visit you! is repositioned so that it is in the direct line of sight, and a different image is used (see figure Variant).

Figure 22. Variant

Variants can be removed from an experiment by selecting Delete variant from the drop-down menu of each tab on the A/B-Testing bar. The corresponding tab then disappears from view on the bar and the page reference is deleted in the SiteArchitect.

The original page is a special case in this regard. It is the initial reference on which the experiment is based. Therefore, as a general rule, it should not be removed from the experiment. However, if you do wish to delete the original page, you must first confirm a prompt (see figure Deleting the original page in the ContentCreator).

Figure 23. Deleting the original page in the ContentCreator

Once all variants have been added with Add variant, click the Edit settings button (displayed as three intermeshing cogwheels) and use the slider in the next dialog that opens to set a percentage value defining how many visitors to the website are to take part in the experiment (see figure Edit settings). If a percentage value is not set, all visitors to the website will always take part in an experiment.

Figure 24. Edit settings

The distribution rate for each variant can also be defined at this point. By default, all variants are displayed with the same distribution rate (100/n %). However, this can be changed freely by the user. The dialog can thus contain both data that has been calculated and data that has been entered manually. To differentiate between these two types of data, calculated values are displayed against a gray background.

If a manual definition only exists for some variants, the difference is divided equally between the remaining variants. If distribution rates have been set for all variants, they are used. The total of all values entered must add up to 100%.

This is illustrated in the examples below.

Example 1. No user input

In the case shown in the figure below, all variants have a distribution rate of 25% each. As this distribution rate has been calculated for all variant, the form fields are displayed against a gray background.

Figure 25. Default uniform distribution

Example 2. User input for just one variant

In the case shown in the figure below, Variant_1 has a user-defined distribution rate of 40% and the other variants have a calculated distribution rate of 20% each.

Figure 26. Distribution with user input

Example 3. User input for multiple variants

In the case shown in the figure below, the original page and Variant_1 have user-defined distribution rates of 20% and 30% respectively and variants 2 and 3 have a calculated distribution rate of 25% each.

Figure 27. Distribution with multiple user inputs

In order to query further information, additional elements can be incorporated into the dialog with the addGui tracking plug-in method. The number of additional elements is determined by the tracking plug-in implemented. In figure Edit settings, for example, the Google Analytics plug-in, which queries a dimension, has been used.

Once all necessary variants have been created and the participation rates and all other settings for the experiment have been configured, click Start to start the experiment. The button triggers the workflow selected during the configuration of the project component. The workflow must release all pages and page references involved in the experiment so that they will go live with the next deployment.

Once the workflow has been completed successfully, the name of the button changes to Running and the release is visualized by the status indicators in the ContentCreator and on the A/B-Testing bar changing color in the familiar way (see figure Running experiment).

Figure 28. Running experiment

In many cases, workflows contain manual steps too. An example of this is the principle of double-checking used in releases. With this approach, the release is initially just requested; it is not processed immediately. During such manual steps, the experiment is at the workflow stage until it advances. The button on the A/B-Testing bar is labeled Continue (see figure Experiment in the workflow). Since the experiment counts as edited and not released in this status, the status is also visualized accordingly.

Figure 29. Experiment in the workflow

It is possible to modify an experiment during its runtime. This can be done by editing or deleting an existing variant, for example, or by adding a variant. In this case, the name of the button on the A/B-Testing bar changes from Running to Update and the status is displayed in the familiar colors (see figure Experiment to be updated).

Figure 30. Experiment to be updated

Tracking during the runtime of an experiment is essential in order to analyze the success of each individual variant. In this story, Google Analytics is used for tracking purposes, as it is required in order to use the tracking plug-in supplied with the module. However, in principle, any analytics tool can be used.

In Google Analytics, tracking takes the form of a custom report which is restricted to a segment created in advance. Restriction to the segment means that only visitors who are relevant to an experiment are captured. The report contains an overview of the individual variants restricted to a defined period of time in the form of a diagram and a table (see also figure Google Analytics - Custom report).

While the diagram shows visitor numbers during the defined period of time, the table contains information about the individual variants. It shows how often the form to request a consultation was completed and submitted. The values are indicated as both absolute and percentage values as well as being set against the total number.

For more information about custom reports and segments, refer to Reporting tools in the Google Analytics help.

Figure 31. Google Analytics - Custom report

Once the variant which comes closest to achieving the predefined aim has been identified, the experiment can be stopped. For the story, figure Google Analytics - Custom report shows that the modified design encouraged more visitors to the website to request a consultation. The variant created has proved to be better than the original page. The analysis confirms the initial assumption made and Variant_1 should be used as the original page. The experiment must be opened again in the ContentCreator and the corresponding tab selected.

To stop the experiment, click the Finish experiment button (displayed as a flag). A confirmation prompt is displayed (see figure Confirmation prompt). Confirm the prompt to apply the selected variant as the new original page.

Figure 32. Confirmation prompt

When the experiment finishes, the A/B-Testing bar disappears from view and the page references of the remaining variants, along with those of the dispatcher, are deleted. The URL of the dispatcher is also transferred technically to the new original page. This ensures that all existing references to the page will continue to function and the URL always looks the same to the outside world. This applies regardless of which variant was displayed to the visitor during the experiment. In addition, for the transfer of the URL, the prefix added to the pages and page references involved when creating the experiment is removed again. As a result, the new original page contains the original display names. These changes are persisted in the PageStore and the SiteStore by releasing the page and the page reference of the new original page, as well as the respective parent folder.