.. _project-export-import: Project export and import ========================= Various tools are used to export and import a project structure. Those tools can be use to export and import inside the same Tuleap instance or between two different Tuleap instances. .. attention:: Known issues / limitation Before going into details, here is the list of the known issues and limitation of the export and import scripts: 1. The export and import scripts have to be run as ``codendiadm``. If they are run by ``root``, you can experience some permissions denied while copying an imported artifact with attachments. 2. Copied artifacts have their first changeset not well exported in XML. 3. Permissions on artifact can have a different value during the export in a specific case. If the field is checked to restrict access to ``all_users``, we assume that the field has no value set. The access to the artifact is not changed, only the field value. 4. The cross-references in followup comments are modified by adding a space between # and the number in order to not leak data in the import in another Tuleap instance. Project Export '''''''''''''' The export script is located in ``/usr/share/tuleap/src/utils/export_project_xml.php`` and must be use like: .. code-block:: bash $> su - codendiadm $> cd /usr/share/tuleap/ $> src/utils/php-launcher.sh src/utils/export_project_xml.php \ -p PROJECT_ID_EXPORT \ -u SITE_ADMIN_USERNAME \ -t TRACKER_V5_ID \ -o PATH_TO_ARCHIVE This will generate a zip archive with: * a ``project.xml`` file that contains project data (usergroups with members + tracker structure + artifacts historized) * a ``users.xml`` file that contains users involved in project data * a folder ``data`` that contains artifact attachments, svn repository and so on. This archive is ready to be imported in a Tuleap instance. Project Import '''''''''''''' The imported archive contains a list of users. Those users are not necessarily present on the target Tuleap instance. Therefore we need to know what action must be taken for unknown users; in some cases we will need to map to another existing users (*jdoe* on source instance may be *john.doe* on target instance), or we may have to create a new user. This is done thanks to a mapping file in csv format. Hopefully a script is here to generate this mapping file based on the archive and current users on target instance: .. code-block:: bash $> su - codendiadm $> cd /usr/share/tuleap/ $> src/utils/php-launcher.sh src/utils/generate_user_mapping_for_project_import.php \ -u SITE_ADMIN_USERNAME \ -i PATH_TO_ARCHIVE \ -o PATH_TO_MAPPING_FILE The generated file will look like: .. code-block:: text name,action,comments jdoe,noop,"Status of existing user jdoe is [S]" nterray,create:S,"Nicolas Terray (nterray) must be created" yrossetto,map:,"There is an existing user yrossetto but its email does not match . Use action "map:yrossetto" to confirm the mapping" ... In this example: 1. ``jdoe`` is a suspended user on the target instance. You may decide to do nothing (``noop``), or you can decide to map ``jdoe`` to another user (see below). 2. ``nterray`` does not exist, it should be created. It will be created with suspended status ``create:S``, with active status ``create:A`` or with restricted status ``create:R``. Default status is suspended. 3. ``yrossetto`` seems to already exist on the target platform but the email is not the same than the source one. In order to not impersonate someone, you must confirm that both accounts relate to the same person. Maybe it is someone else, therefore you can ``map:`` to another account. Comments column should give you all needed information about current status. .. note:: Please note that this column is only informative and will not be used during the import. .. note:: The following diagram explains how the décision is made while parsing the users.xml: .. figure:: ../images/diagrams/mapping-users-during-project-import.png :align: center :alt: Agile Dashboard Configuration :name: Agile Dashboard Configuration After having reviewed/edited the mapping file, you should pass it through a script in order to know if your choices are valid in regards to current status of the target instance: .. code-block:: bash $> src/utils/php-launcher.sh src/utils/check_user_mapping_for_project_import.php \ -u SITE_ADMIN_USERNAME \ -i PATH_TO_ARCHIVE \ -m PATH_TO_MAPPING_FILE This will generate some feedback about the wellness of the mapping file. You must fix any remaining errors before doing the real import: .. code-block:: bash $> su - codendiadm $> cd /usr/share/tuleap/ $> src/utils/php-launcher.sh src/utils/import_project_xml.php \ -p PROJECT_ID_IMPORT \ -u PROJECT_ADMIN_USERNAME \ -i PATH_TO_ARCHIVE \ -m PATH_TO_MAPPING_FILE Import format ''''''''''''' Tuleap is able to import a project with it's content from an XML file. This section describes what is the content of this file and how to proceed to generate an XML compatible with the import tool. .. note:: As of today, the following things are covered by the import tool: - user groups definition with members - trackers with contents and history - subversion - Git - Agile dashboard - FRS - Mediawiki General informations ******************** The import should be a zip archive with - description of project content in ``project.xml`` file - list of all users that will appear in project.xml in ``users.xml`` file - data blob (should be referenced in project.xml) in ``data`` directory Users management **************** In ``project.xml``, when we have to refer to a user the pattern is the following ``identifier`` where ``format`` can either be: - ``id`` a user unique id (integer) - ``username`` a user unique login name (string) - ``ldap`` user reference in LDAP, useful for in house data migration (string) You can pick whatever format you want. .. important:: Anonymous users Another format can be used: ``email``. This is only to reference anonymous users on the source instance. At import time, if the email match an existing user, it will be used (instead of treating the user as anonymous). As soon as you reference a user in ``project.xml``, this user must be defined in ``users.xml`` like: .. sourcecode:: xml 101 john_doe The 5 nodes should be present but the most important are ``username``, ``realname`` and ``email`` those elements will be used to decide whether the user already exists on the platform or not (and create the corresponding account if needed). ``id`` should be unique but will only be used if you are using ``123`` references in project.xml. We recommend to have a numeric sequence (1, 2, 3...) here. ``ldapid`` can be empty but if you move data from one service to another within the same organization, this might be useful to share the user base. This should be the unique identifier of the user in LDAP. Possible values from LDAP fields ``uid``, ``uuid``, etc. 99% of the time it's the unique part of LDAP ``dn``. Example of users.xml: .. sourcecode:: xml 101 john_doe 102 alice .. danger:: if a user in ``project.xml`` is not referenced in ``users.xml`` then the script will stop during the import, leading to half imported data. Core **** All projects related information (core & services) are stored in ``project.xml``. Core information imported as of today: - Project metadata: unix name, full name, description, long description, and access (for more information see "Create a new project") - User groups and membership (user are referenced by username or ldapId, and group name can be dynamic group names where applicable) - Services to be enabled. If a service is not specified, the enabled status is taken from the template project. .. sourcecode:: xml This is the long description of project 42 joey_star alice bob alice bob joey_star .. note:: Users that are suspended won't be part of the imported project. All services can then be configured using its own tag. Trackers ******** Basics: - ```` node contains a list of ````, then ```` and finally ```` - Within a ```` there is first the structure of the tracker and then the data themselves within ```` node. - The tracker structure is made of metadata (like ````), fields (````), semantics (````), Workflow & field dependencies (````, ````), reports (````) and permissions (````). - An ```` is made of ````, each ```` corresponds to a modification of the artifact. Order matters! the first is the artifact creation. - A ```` is composed of a ```` (can be in ``text`` or ``html`` format) and a set of ````. Each ```` refers to a field referenced in the ```` section of ````. Example of a simple tracker with a few fields --------------------------------------------- The example below is a simple tracker made of following fields - Id (type: artifact id) - Submitted by (type: submitted by) - Title (type: string, associated to "title" semantic) - Dependency (type: artifact link) - Description (type: text) - 2 structure fields columns (C1 and C2) - Product (type: multiselectbox) - Status (type: selectbox, associated to "status" semantic) Some insights to better understand how this works: - L73: definition of status semantic uses reference to field F6680, this will automatically refer to the field defined L51. And same applies for values considered as "Open" (, L78) that uses references V7678, V7679 and V7680 defined L56-59. - L146: the artifact creation set a value to this status field () and the value (7678) refers to V7678 L56. .. sourcecode:: xml :linenos: :emphasize-lines: 51,56,57,58,59,78,146 ... ... simple inca_silver id submitted_by title depends description c1 product c2 status title Define the title of an artifact status Define the status of an artifact Default The system default artifact report Results vaceletm 2015-11-10T09:05:19+01:00 7675 7678 vaceletm 2015-11-10T09:05:46+01:00 vaceletm 2015-11-10T09:05:46+01:00 7680 vaceletm 2015-11-10T09:05:19+01:00 7675 7678 vaceletm 2015-11-10T09:05:46+01:00 vaceletm 2015-11-10T09:05:46+01:00 1234 12 42 7680 Artifact link types ------------------- When importing a project that uses custom artifact link types (aka. natures), you should check that the target Tuleap platform has the same artifact link types by adding a ```` node. The XML importer will check that all artifact link types defined in this node exist on the target platform. If one or more types don't exist, it will stop the import safely before anything is imported. .. sourcecode:: xml ... my_custom_nature_shortname .. attention:: You must set the ``use-natures`` attribute to ``true`` on trackers using artifact link types (see the examples below) Folders ------- If you have existing folders you want to keep at import, you have to define the tracker which will be flagged as folder with the attribute "is_folder". To import your artifacts with link "In folder", the system nature _in_folder will be used. .. sourcecode:: xml ... ... simple inca_silver computed_field_name ... vaceletm 2015-11-10T09:05:19+01:00 12 ... Computed field import --------------------- The new computed field can have have manual or autocomputed values. The node enable to keep the manual value for the field, the node keeps the field in autocompute mode. .. sourcecode:: xml :linenos: :emphasize-lines: 15,16,17,18,19,20,29,32 ... ... simple inca_silver computed_field_name ... vaceletm 2015-11-10T09:05:19+01:00 1 ... Permissions ----------- In the previous example, you can see a block for permissions management at the end .. sourcecode:: xml There are 2 scopes for permissions: * `tracker` * `field` Fore each we have the same type of attributes: * `REF` reference a field defined in tracker structure * `ugroup` is one group that have the given `type` of permission * `type` is the access right granted (depend of the scope) Supported `ugroup` values: * `UGROUP_ANONYMOUS` * `UGROUP_REGISTERED` * `UGROUP_PROJECT_MEMBERS` * `UGROUP_PROJECT_ADMIN` * Any custom ugroup name If several ugroups should have the same permission, there should be several lines Supported `type` value for `tracker` scope: * `PLUGIN_TRACKER_ADMIN` grant tracker administration capabilities to group * `PLUGIN_TRACKER_ACCESS_FULL` grant access to tracker to group * `PLUGIN_TRACKER_ACCESS_ASSIGNEE` grant access to all artifacts assigned to people of the group (eg. Alice and Bob belong to "project_members", Astrid belong to "project admin". Alice will see all artifacts assigned to herself or assigned to bob but not those assigned to Astrid) * `PLUGIN_TRACKER_ACCESS_SUBMITTER` grant access to all artifacts submitted by people from the group (see previous example but with "submitted by" instead of "assigned to") * `PLUGIN_TRACKER_ACCESS_SUBMITTER_ONLY`, for this group people see only the artifacts they submitted (eg. HelpDesk). Supported `type` for `field` scope: * `PLUGIN_TRACKER_FIELD_READ` ugroup can see the field content * `PLUGIN_TRACKER_FIELD_SUBMIT` ugroup can set value for the field at artifact creation * `PLUGIN_TRACKER_FIELD_UPDATE` ugroup can upgrade the field value after creation Subversion repository ********************* A single subversion repository can be imported. The XML syntax is: .. sourcecode:: xml ... ... ... ... ... - A single ```` tag containing: - A ```` tag per repository containing: - The attribute ``name`` which contains the repository name. It is required. A good default choice is to use the project unix name. - The attribute ``dump-file`` which references a dump file generated by ``svnadmin dump`` - The tag ```` containing the ``.SVNAccessFile`` inline - ```` tags, one for each monitored path. E-mails are separated by commas. Git repositories **************** Multiple Git repositories can be imported. The XML syntax is: .. sourcecode:: xml ... ... Contributors ... project_members ... Contributors ... project_admins ... ... ... ... - A single ```` tag containing: - An optional ```` tag containing: - One or more ```` containing the ugroup name. Note that the project administrators ugroup will always be part of the git administrators. - One or more ```` tag containing: - The attribute ``bundle-path`` which references a bundle file generated by ``git bundle create --all`` - The attribute ``name`` to specify the name of the repository. Note that it accepts a "path" notation (see the repository list in the git plugin) - The attribute ``description`` containing the description of the repository - The tag ```` containing one or multiple ```` tags. It allows to give the different permissions of the repository to the given repository. The ```` tag must contain only one ugroup name. Another synthax exists for the git XML import: .. sourcecode:: xml ... ... Contributors ... project_members ... Contributors ... project_admins ... Contributors ... project_admins ... Contributors ... project_admins ... ... ... ... The big change is in the ```` tag: - The tag ```` containing ```` and an optional tag named ```` - The tag ```` containing one or multiple ```` tags. It allows to give the different permissions of the repository to the given repository. The ```` tag must contain only one ugroup name. - The tag ```` containing one or multiple ```` tag. - The attribute ``enabled`` which defines if this permission option is used for the repository or not - The attribute ``use_regexp`` which defines if patterns with regexp are authorized or not - The tag ```` containing ```` tags. - The attribute ``value`` which defines the gitolite path checked by the pattern - The attribute ``type`` which defines if this permission is a branch or a tag permission. The possible values are ``tag`` and ``branch`` Releases and Files (FRS) ************************ The released files (FRS) can be imported, the XML syntax is: .. sourcecode:: xml ... ... ... ... The XML must contain a single ```` element containing zero or more packages. The packages can contain zero or more releases, which can contain zero or more files. A package is constructed using a ```` XML element containing: - A ``name`` attribute - A ``rank`` optional integer attribute - A ``hidden`` optional boolean attribute - A single ```` element containing the access definitions. - A ```` element per releease. A release is constructed using a ```` XML element containing: - A ``name`` attribute - A ``time`` attribute containing the ISO-8601 representation of the release date. - A ``preformatted`` attribute containing a boolean that indicates if the release notes and changelog are preformatted. - An ``artifact_id`` attribute referencing an artifact that must be present in the XML content. The FRS plugin must be installed to take into account the value. - A single ```` element containing the access definitions. - A single ```` element containing the release notes - A single ```` element containing the changelog - A single ```` element describing the user who made the release - A ```` element per file contained in the release A file is constructed using a ```` XML element containing: - A ``src`` attribute pointing to the file to import (relative to the XML file) - A ``md5sum`` attribute containing the MD5 hash of the file (checked on import, optional) - A ``name`` attribute: the file name (optional) - A ``release-time`` attribute: the ISO-8601 representation of the time when the file was released (optional, current time used if not provided) - A ``post-date`` attribute encoded in ISO-8601 (optional, current time used if not provided) - An ``arch`` attribute: the name of the architecture of the file as appearing in the table ``frs_processor``. On a fresh install, the processor list is ``i386``, ``x86_64``, ``PPC``, ``MIPS``, ``Sparc``, ``UltraSparc``, ``IA64``, ``Alpha``, ``Any``, ``Other``. - A ``filetype`` attribute: the name of the file type of the file as appearing in the column ``frs_filetype``. On a fresh install, the file types available are ``Binary .rpm``, ``Binary .deb``, ``Binary .zip``, ``Binary .bz2``, ``Binary .gz``, ``Binary .tar.gz, .tgz``, ``Binary .jar``, ``Binary installer``, ``Other Binary File``, ``Source .rpm``, ``Source .zip``, ``Source .bz2``, ``Source .gz``, ``Source .tar.gz, .tgz``, ``Other Source File``, ``.Documentation (any format)``, ``text``, ``html``, ``pdf``, ``Other``. - An optional ```` element containing a file description - An optional ```` element describing the user who posted the file Access definitions consists of zero to many ```` tags, each containing the name of the user group that is allowed access. Mediawiki ********* The Mediawiki can also be imported. For the moment only pages import is supported. The XML syntax is: .. sourcecode:: xml project_members project_admins ``language`` is the language of the wiki. Currently only ``en_EN`` and ``fr_FR`` are supported. ``pages-backup`` is the path to the wiki pages backup done with the ``dumpBackup.php`` `maintenance script`_ provided by the Mediawiki project. ``files-folder-backup`` is the path to the directory where the files to upload are stored into. All the files must be stored at the root of the folder. ``read-access`` and ``write-access`` must contain ``ugroup`` tags with ugroup names. At the moment, you should use the script located at ``$TULEAP_ROOT/plugins/mediawiki/bin/mw-maintenance-wrapper.php`` to export the pages of a tuleap Mediawiki . For example, to export the mediawiki of the project named "toto" use ``$TULEAP_ROOT/plugins/mediawiki/bin/mw-maintenance-wrapper.php toto dumpBackup.php --full``. The schema is available here : `mediawiki schema`_. .. _mediawiki schema: https://www.mediawiki.org/xml/export-0.7.xsd .. _maintenance script: https://www.mediawiki.org/wiki/Manual:DumpBackup.php References ********** If you have existing references you want to keep at import, you can define them, services by services. Supported references: * artfXXX for artifacts * pkgXXX for packages * relXXX for releases * wikiXXX for wiki pages * cmmtXXX for git or svn commits .. note:: You need to install and activate the following packages to make it works: * referencealias_core * referencealias_git * referencealias_mediawiki * referencealias_svn * referencealias_tracker .. note:: When you import serveral times the same project (for test purpose for instance) it can be convenient to have the references that points to the last version of the import. To do so you can use the --force=references switch on the import command line. Example of references: .. sourcecode:: xml :linenos: ... ... ... ... ... ... ...