Step 4) Exporting project content (via External Synchronization)

From the FirstSpirit project into the file system

In this step, External Synchronization is used to

• export the development state out of the local FirstSpirit project instance into the target folder in the file system.

This step is needed to transfer any modified content into the central development state.

The command line tool “FSDevTools” (fs-cli) is used for the export process.

See figure Step 4) Export.

All developers involved in a project should always use the same export call. The call depends on the project and the elements referenced within the project templates.

This helps to prevent problems resulting from incomplete project contents, incomplete references (in local development environments), and, as a result, conflicts.

Initial situation

What can be exported?

The following can be exported:

• Project content (objects): Content that is created and managed using SiteArchitect or ContentCreator. This includes templates, project structures, subtrees, and complete stores.
  When exporting project content via External Synchronization, it is filed according to a defined structure in the file system (see Sync directory structure). The system attempts to duplicate the structures from the project using a file hierarchy in the file system. Further internal information is also exported to the FirstSpirit objects, e.g., via the storeelement.xml. This file contains information such as the FirstSpirit object’s ID, reference name, etc.
  Using this information as a basis (folder hierarchy and internal meta-information), objects can be imported from the file system back into a FirstSpirit project.
• Project properties: Settings that can be defined for a project via the ServerManager. These properties can be defined for specific projects (“Project properties”) or on a global level (“Server properties”) (e.g., languages).
  For some projects (especially in the context of FirstSpirit CaaS), user-defined project properties (custom properties) can be defined. These properties can also be exported.

Preparation

The developer’s local repository has already been updated using the version control system (see 1) Update). During this process, the central development state was transferred out of the remote repository. The project content and, if applicable, project properties are in the Sync directory (file system) and ready for import.

Before further development, all content is imported into the local FirstSpirit project instance (see 2) Import).

The project content is then edited (see 3) Modification). Once a stable state has been achieved after editing is complete, the modified development state from the local FirstSpirit project instance is ready for export.

Starting the command line tool (fs-cli)

The command line is opened in the “bin” directory (in the FSDevTools installation directory). All instructions issued via the command line start with the call fs-cli and a corresponding instruction.

D:\fs-cli\bin>fs-cli [command]

Establishing a connection to FirstSpirit (fs-cli)

To establish a connection to the (local) FirstSpirit server and the (local) project, the corresponding parameters (for host -h, Port-port, project name -p, user name -u, and password -pwd) are transferred, e.g.:

-h example.com.de -port 4242 -c HTTP -p "DevProject" -u userA -pwd mypwd

Optionally the parameter -sz can be transferred to specify a servlet zone.

To establish a connection to the file system, the path to the target folder -sd in the (local) file system also has to be specified, e.g.: -sd D:/Git/DevProject

You also have the option of testing the connection using the call test, whereby the placeholder [connection_parameters] has to be replaced with the project-specific connection settings:

D:\fs-cli\bin>fs-cli [connection_parameters] test

Log output if the test is successful:

#########
Test was successful
#########
User: userA
Host: example.com
Port: 4242
Servlet zone: null
Connection Mode: HTTP
Project: DevProject
Connection to FirstSpirit closed!
Execution time: 10.591s

Exporting

Exporting project content (fs-cli)

In the next step, the current state is exported out of the local FirstSpirit project instance into the target folder in the local file system (see figure “Exporting”).

The export process is launched with the command export, whereby the placeholders [connection_parameters] and [export_objects] have to be replaced with the project-specific values:

D:\fs-cli\bin>fs-cli [connection_parameters] export [export_objects] 

When performing an export, the following processes take place within the file system

• new folders and files are created
• existing folders and files are modified
• existing folders and files are moved
• Existing folders and files are removed

Help function (fs-cli)

Further project-specific settings for the export process can be viewed using the FSDevTools’ help function:

D:\fs-cli\bin>fs-cli help export

Which state is taken into account?

In the case of templates, there is no distinction drawn between the approved and current state: the template’s current editing state is always exported.
In the other stores, the corresponding object’s current state is also always taken into account for exports. When exporting content, it therefore makes no difference if the objects to be exported have been approved or not.

Recommendations for exports

Distributed development is based on a foundation of one central FirstSpirit instance (or a central repository),which collects the modified artifacts of all developers (see figure below “Dev_Project”, “Remote-Repository”).

Each developer also works in his own local development environment with his own FirstSpirit Server, his own local repository, and his own local FirstSpirit project instances (see figure below “Local Dev_Project”, “ Local Repository”).

During distributed development, FirstSpirit project content is transported between the central development state and the individual developers’ local development states, i.e.:

• At the start of his work, the developer begins by moving the content from the central development state into his local development environment (see Updating the repository – Import).
• At the end of the editing process, the content from the local development states are then transferred back into the central development state. During this process, the projects’ local development states are exported into the local file system’s target folder using External Synchronization (see figure “Exporting”). From there, the content is transferred into the central state via the VCS.

To make sure the synchronization of content between different projects and different environments runs as smoothly as possible, requirements are set out for the projects and the transported content.

Recommendations for projects

Project settings:
All FirstSpirit project instances involved (local target projects, central source project) should use identical settings (language sets, etc.) (see Recommendations for imports (settings)).
Project properties can be added to the export using the parameter projectproperty.
Note: Some project properties are dependent on server properties. These dependencies are automatically taken into account.

All project properties of a project (incl. custom properties) can be exported via
export projectproperty:ALL

Specific project properties can be exported using defined key terms, e. g. languages and resolutions via:
export projectproperty:LANGUAGES projectproperty:RESOLUTIONS

Further key terms for exporting project properties:

• COMMON (general project properties (via ServerManager's “Project Properties” / “Options” menu item))
• FONTS (fonts to be available for the current project; dependency on server properties)
• LANGUAGES (languages to be available for the current project; dependency on server properties)
• RESOLUTIONS (resolutions defined for the current project)
• USERS (users who have access to the project; dependency on server properties)
• GROUPS (groups that have access to the project)
• SCHEDULE_ENTRIES (schedule entries and action templates of the project; dependency on server properties)
• TEMPLATE_SETS (template sets of the project; dependency on server properties)
• MODULE_CONFIGURATIONS (Configurations of project and web components of the project; dependency on server properties)
  Note: The project or web components themselves are not imported or exported via External Synchronization!
  The relevant component must be added to the target project either manually or via the API.
  (Interface ModuleAdminAgent, Package: de.espirit.firstspirit.agency, FirstSpirit Access API).
• CUSTOM_PROPERTIES (user-defined project properties)
  Note: These properties can only be defined via the API (setCustomProperties method) (not via the FirstSpirit ServerManager).
  (Interface Project, Package: de.espirit.firstspirit.access.project, FirstSpirit Access-API)

Project structure:
All FirstSpirit project instances involved should have a standardized structure.
To make the synchronization of project content as simple and clear as possible, all project content relevant for template development should also be managed in separate folders (see Getting started - Setting up and configuring a project).

Recommendations for project content

Templates in particular tend to have various dependencies (references) to other FirstSpirit objects. These may be other templates or objects from other stores, e.g., CSS files from the Media Store.

Normally, these referenced objects also have to be taken into account in an export. An export should therefore be as complete as possible and take into account all dependencies between the individual exported objects.

All developers involved in a project should always use the same export call. The call depends on the project and the elements referenced within the project templates.

This helps to prevent problems resulting from incomplete project contents, incomplete references (in local development environments), and, as a result, conflicts.

Templates:
All templates within a project should always be exported, e.g., the entire template set using the parameter templatestore.

Structures (page references):
Furthermore, all page references referenced within the templates should be exported, e.g., the template preview pages (“properties” tab). If these page references are in their own separate folder “preview” in the project, the entire folder can be added to the export, e.g., using the parameter pagereffolder:preview.

Content (pages):
Furthermore, all pages referenced within the exported objects should be exported, e.g., pages connected to the page references (see above). If these page references are in their own separate folder “previewp” in the project, the entire folder can be added to the export, e.g., using the parameter pagefolder:preview.

Media:
All media referenced in the exported objects should also be exported, e.g.:

• Image files for icons
• Image files for buttons
• Image files for navigation elements
• CSS files
• JavaScript files

A reference to these media is normally established under the presentation channel tabs (e.g., using $CMS_REF(media:"...")$). If the media are in a separate folder “layout” in the project, the entire folder can be added to the export, e.g., using the parameter mediafolder:layout.

Data sources:
Furthermore, all database contents referenced in the exported objects should be exported. Both the datasets and the associated data sources have to be exported.
If the datasets are in their own separate folder “glossary” in the project, the entire folder can be added to the export , e.g., using the parameter path:/ContentStore/glossary and the corresponding data sources can also be added, e.g., using the parameter entities:glossar.

In rare cases (see below: Support for shared access to a database) the use of an additional mapping file may be useful.

Exporting and importing (extensive) database contents can require high loads.

Global content:
Furthermore, any global content referenced in the exported objects should be exported, e.g., global content referenced on the templates’ presentation channel tabs (e.g., using $CMS_VALUE(#global.gca("..."))$). If this content is in a separate folder “global_export” in the project, the entire folder can be added to the export, e.g., using the parameter path:/GlobalStore/GlobalContentArea/global_export.

Optional: Support for shared access to a database (Mapping.xml)

Using the “FirstSpirit External Synchronization” mechanism, database content (datasets) and the associated database structures (schemas, tables, columns) of a FirstSpirit project (source project) can be exported and imported into other FirstSpirit projects (target projects). The dbnames of column names and tables are recalculated during synchronization. This behavior can, under the following conditions, lead to the problem that columns and tables in the target projects cannot be found in the database:

• the FirstSpirit schemas of the participating projects reference the same database tables (and do not point to their own database tablespaces) and
• the dbnames do not match the automatically calculated dbnames

For this special use case an additional mapping file (Mapping.xml) can be used when exporting and importing database schemas. In this file the database names of the source project are stored together with FirstSpirit's own unique identifiers (UUIDs), e. g.:

<xs:gid dbName="j_categories_spalte_x" uuid="66843bbe-da0d-46e3-a136-76bd93d782bb"/>
<xs:gid dbName="j_categories_spalte_y" uuid="4ba29641-26af-4edd-bac0-5a352913974d"/>
...

Using this unique assignment, it is possible to synchronize the database schemas of read-only-connected databases (“No Schema Sync”) via “External Synchronization” as follows:

• If a schema is changed in the source project and synchronized with other FirstSpirit projects using “External Synchronization” mechanisms, the names of database tables and database columns are transferred from the source project to the target projects.
• If a new database table or database column is created in the target project, the corresponding dbname of the source schema is preferred.

When using “FirstSpirit External Synchronization” the mapping file (Mapping.xml) can be added to the export file via an option in the interface ExportOperation.SchemaOptions (setExportGidMapping=true) as follows:

ExportOperation exportOperation = ... // get the export operation via agent; 
ExportOperation.SchemaOptions schemaOptions = exportOperation.addSchema(schema); 
schemaOptions.setExportGidMapping(true); 
ExportOperation.Result result = exportOperation.perform(...);

The command line tool FSDevTools supports the new option from version 2.6 on. By default, no mapping file is created when using FSDevTools (setExportGidMapping=false).

Without a mapping file (default behavior when using FirstSpirit External Synchronization) the dbnames of the column names and tables are recalculated during synchronization.

The mapping file should only be used to resolve existing conflicts in the target projects that may occur during shared access to a database (see “conditions” above). It is not recommended to use the mapping file for all other regular use cases.

After the export

Reproducing content in the file system

Following a successful export via External Synchronization, the FirstSpirit content is filed in the target directory as files, where they can also be edited (see Editing in the file system). During the export process, the system attempts to duplicate the structures from the project in the file system’s folder hierarchy. A distinction is drawn between files with content-related (specialist) information about the FirstSpirit objects and files with meta-information (checksums, reference tables, etc.).

The following files are saved for each FirstSpirit element:

• one folder
  A folder for each FirstSpirit element is created in the local file system. The path from the object selected in FirstSpirit up to the corresponding root node is duplicated in the form of folders. The file name is derived from the reference name of the FirstSpirit element in question.
• StoreElement.xml
  An XML file entitled StoreElement.xml is filed for each FirstSpirit element. This file contains information such as the ID and reference name for the FirstSpirit object and can be edited externally.
• further XML files
  Depending on the FirstSpirit object type, further XML files may also be created, e.g., regarding the content of sections (data.EN.xml), datasets in a data source (Entities.xml), information about the individual tabs in templates (e.g., form area, rule definitions), etc.
• one content file
  For some FirstSpirit elements, specialist content that cannot be displayed in XML format is stored separately. The file name is derived from the reference name of the FirstSpirit element in question. The format varies depending on the type of the FirstSpirit element:
  • for media: the binary files for an image or file, for example
  • for the HTML presentation channel of a template: an HTML file.
• specialist TXT files
  TXT files containing specialist information are created for certain objects, e.g., snippet definitions for templates (SnippetHeader.txt).
• meta TXT files (FS_....txt, “meta files”)
  TXT files with the prefix FS_ (FS_Files.txt, FS_Info.txt, FS_References.txt, etc.) contain meta-information. These files should not be edited externally.

In principle, files are only created if the project contains corresponding information, for example, if a particular field has been filled out. Not all of the files listed have to be included in an export and no empty files are created – apart from a few exceptions.

For further information, see Configuration – Notes on versioning.

Sample overview

Next steps

Transferring changes to the central state

Once work on a work package is complete, the modified artifacts are transferred to the central development state (remote repository). They are then available to all developers.

Continue with Step 5) Commit/Push.