Administration HowTo

Importing an existing CVS repository in Tuleap

How to properly install a project team’s existing CVS repository into the Tuleap project specific repository

Preamble

The main thing to understand about Tuleap is *each* project has its own CVS repository. This root dir is created and initialized by Tuleap whenever a new project is registered. Its name is “/cvsroot/projectname”

I insist on this point because most of the time in many organizations there is one single CVS repository where all projects are stored. The problem with this typical organization is that the directory named “CVSROOT” which contains CVS config and admin files is unique and global to all projects managed by CVS. The directory named “CVSROOT” contains files such as “modules” to declare module name aliases, or commitinfo,rcsinfo and loginfo to trigger specific actions when commit happens (such as e-mail notification to certain people,etc.), or taginfo to put constraints on the TAG formats, or readers and writers which are 2 files to allow/deny access to the CVS repo.

If we had one single CVS repository on Tuleap then the unique and global “CVSROOT” directory would rapidly become a big mess and it would be extremely difficult to maintain all the project specific configurations in this single “CVSROOT” directory. Hence the decision to have one CVS repository per project on Tuleap.

The real work

  1. Register the project. Tuleap Admin must approve it and wait for the 30mn cron update to create a fresh /cvsroot/projectname CVS repository with a fresh CVSROOT directory in it. If you are in a hurry you can force the cron update by hand.

  2. Ideally the tar file provided by the team should provide both the CVS top directories *PLUS* the “CVSROOT” directory from the original CVS repository.

  3. Having the CVSROOT in the tarball is not mandatory but if the project had any specific settings in any of the CVS administration file (see above) then there is no chance that the Tuleap admin can rebuilt the same environment on Tuleap. In practice very few projects actually use these admin files but you can’t never know hence the request for the CVSROOT directory to be included in the tarball. One file that is always used though is the CVSROOT/history file where all the actions like commit, add, checkout, etc. are automatically logged by CVS. If the team wants this file to be preserved then again the CVSROOT has to be provided

  4. Create a temporary directory: cd /tmp mkdir projectnamecd projectname

  5. Unpack the cvs tarball provided by the project team in this temporary directory: `` tar xvfz projectname_cvstarball.tgz``

  6. Assuming that the structure is now as follows

    /tmp/projectname
   - CVSROOT
       - history  (as well as ,v files)
       - modules
       - commitinfo
       - .....
   - top_level_dir1
   - top_level_dir2
   - ....
   - top_level_dirN

  7. Move all top project dir to the project specific repo: mv top_level_dir*  /cvsroot/projectname/ Caution!! Leave the CVSROOT directory in place for the moment

  8. Go to the project CVS repo: cd /cvsroot/projectname

  9. Change the ownership of the entire CVS repo: chown -R projectadminname.projectname top_level_dir* where projectadminname is the Tuleap login of the project administrator

  10. Ensure that setgid bit is properly set on directory: find /cvsroot/projectname -type d -exec chmod g+s {} \;

  11. The CVS admin files in the CVSROOT directory must be examined one by one and the project specific instructions must be merged with the files by the same name in the CVSROOT directory created by Tuleap.

    • **Never touch** the CVSROOT/readers, writers, passwd and config files created by Tuleap
    • Be careful not to overwrite what is already in the CVSROOT/loginfo and val-tags file. Append the content of these given files at the end of the corresponding ones generated by Tuleap.
    • For the history file : extract the lines belonging to the hosted project from the history file provided by the project team (remember it is very likely that this file is global to many projects) and update the content (especially the name of the directories and path listed in the history file if they are not the same on Tuleap)
    • For all the other files, look at them one by one and see if there are project specific settings in it. If so append the settings at the end of the corresponding Tuleap files.

  12. Delete the temporary directory: rm -rf /tmp/projectname

  13. Ask the project team to perform a checkout as explained on their CVS project page to check that everything is working fine.

That’s all folks!

Regenerating a fresh CVS or SVN repository for a Tuleap project

When the first import goes wrong or the project team screwed up everything and want to start again from scratch

The fastest and safest way (where xxx is the project name):

  • Log as root
  • cd /cvsroot for a CVS repository, or cd /svnroot for a SVN repository
  • Make a backup copy of the current tree in /tmp just in case…: tar cvfz /tmp/xxx_repository.tgz ./xxx
  • Delete the CVS or SVN tree: rm -rf  xxx
  • Wait for the SYSTEM_CHECK system event to run.
  • Delete corresponding entries from cvs_checkins, cvs_commits and cvs_descs tables to prevent showing them while browsing commits with viewVC.

Now you have a fresh CVS or SVN repository and the team can start the import again.

How to delete a user

Project member

If the user was part of a project then do the following:

  • Set the Web status to ‘Deleted’
  • Set the Unix Status to ‘Suspended’

The Unix account is deleted for real, the user home directory is deleted as well.

Change privileged account passwords

Steps to change privileged account passwords

Some companies have specific policies regarding passwords.

root account:

# Change unix password from root account:
$ password
[enter new password]
[retype new password to confirm]

# Change *mysql* password:
$ mysqladmin --user=root -p password [new\_password]
enter password: [enter old password]

codendiadm account:

# Change *unix* password from **codendiadm** account:
$ password
[enter new password]
[retype new password to confirm]

# Change mysql password:
mysqladmin --user=codendiadm -p password [new\_password]
enter password: [enter old password]

$ vim /etc/tuleap/conf/database.inc # change **codendiadm** password

admin account:

  • The “admin” account is the first user that is created when a new Tuleap site is created.
  • The “admin” account is used as an “administrator” for orphaned projects.
  • The “admin” unix account is not a privileged account.
  • If you have defined other site administrators, you might as well suspend the “admin” account (thus, you won’t need to update its password).

Change unix and mysql password for admin from Tuleap website:

  • Go to the Tuleap web site
  • Log in as admin
  • Click “Account Maintenance” in left bar
  • Click “[Change Password]”
  • Follow directions for changing password

mailman account:

  • The “mailman” account has the universal password for mailing lists.
  • To test the mailman account:
  • Go to http://lists.example.com/mailman/admin
  • Click on any mailing list
  • Type new global admin password

If you see the admin page of the selected mailing list, then it is OK.

Enabling ‘Restricted Users’ on a Tuleap server

Overview

One of Tuleap main benefits is to enable code sharing and re-usability in a corporation. The goal is to provide visibility of your project and code in the whole community of users. Still, in some cases, this visibility becomes an issue:

  • When Tuleap is used for customer interaction, e.g. to collect customer support requests and bugs.
  • when some project members are from other companies (contractors), and should not have access to the whole site

In both cases, external people (customers and contractors) need an access to the server; yet, they should not be granted access to all the other projects hosted on Tuleap.

A solution to this problem has already been put in place: use two servers instead of one (one for internal use, one for ‘partners’). Yet, there is an issue with projects that are being developed internally and that have customer feedback. In this case, you need to duplicate the project, which is not convenient; customers cannot access commit pages referenced in commit emails (because they point to the internal server),etc.

Now, there is another solution by enabling restricted users on your server: these users (typically, external users), are granted access to their project pages only, and are denied access to any other page (other projects, software map, etc.). Normal users still have access to the whole server.

Configuration

local.inc

In order to setup restricted users, you need a coherent local.inc.

Edit /etc/tuleap/conf/local.inc and make sure that the following variables are properly set:

$sys_user_approval = 1;
$sys_is_project_public = 1;
$sys_allow_anon = 0;
$sys_allow_restricted_users = 1;

Some explanation:

  • $sys_user_approval = 1 is needed because it is during the user approval phase that the Tuleap administrator is able to set the user status to ‘Restricted’ (R) or ‘Active’ (A). If the variable is set to ‘0’, users will be created as Active by default.
  • $sys_is_project_public = 1 is not absolutely necessary. It just seems useless to have restricted users if all projects are private.
  • $sys_allow_anon = 0 is needed. Otherwise, anonymous users (i.e. users who have not logged in) would have more privilege than restricted users. So a restricted user would simply need to log out to be able to browse other projects, etc.
  • $sys_allow_restricted_users = 1 simply enables the ‘Restricted’ status for users.

Restricted Shell

By default, restricted users do not have a regular shell access: they are given a restricted shell access that only supports a few command (only ‘cvs’ today).

The default shell is ‘/usr/lib/tuleap/bin/cvssh-restricted’. It grants CVS access to projects the user is member of, and forbid access to all other projects repositories.

If you need to completely remove shell access (and forbid CVS), you need to modify the shell manually in the administration interface for each user: set it to /sbin/nologin.

Setup: in order to use CVS, the restricted user must do the following:

  • Set the ‘CVS_RSH’ environment variable to ‘ssh’
  • use the following command line: cvs -d:ext:username@example.com:/cvsroot/projectname co module

See also ‘Tuleap Installation Guide’.

Setting restricted users privileges

You may fine tune the privileges of restricted users on your system. Simply copy /usr/share/tuleap/site-content/en_US/include/restricted_user_permissions.txt in /etc/tuleap/site-content/en_US/include/ and edit it. This is a sample configuration file:

// comment/uncomment forbidden URLs
$forbidden_url = array(
          '/softwaremap/', // browsable software map
          '/new/',         // list of the newest releases made on the Tuleap site
          '/search/',      // search for people, projects, and artifacts in trackers!
          '/people/',      // people skills and profile
          '/stats/',       // Tuleap site statistics
          '/top/',         // projects rankings (active, downloads, etc)
          '/project/register.php',    // Register a new project
          '/export/',      // Tuleap XML feeds
          '/info.php'      // PHP info
          );

// Use true/false for those options
$allow_welcome_page=false;// Allow access to Tuleap welcome page (at e.g. http://example.com/)
$allow_news_browsing=false;     // Allow restricted users to read/comment news, including for their project
$allow_user_browsing=true;      // Allow restricted users to access other user's page (Developer Profile)
$allow_access_to_codendi_forums=true;   // Tuleap help forums are accessible through the 'Discussion Forums' link
$allow_access_to_codendi_trackers=false;// Tuleap trackers are used for support requests on Tuleap
$allow_access_to_codendi_docs=false; // Tuleap documents (Note that the User Guide is always accessible)
$allow_access_to_codendi_mail=false; // Tuleap mailing lists (Developers Channels)

Other considerations

  • Restricted users must be denied access to the pserver protocol to access CVS: only the SSH method is supported (through the restricted shell). If you want to disable the pserver access, make sure you edit/etc/xinetd.d/cvs, change the ‘disable’ parameter to ‘yes’ and restart xinetd (service xinetd restart). You may also fine-tune the configuration file to allow pserver for some IP addresses and deny it for others..
  • Access to projects web sites by Restricted Users is not controlled. If a project web site displays sensitive data, then it should put in place access restriction mechanisms (e.g. a ‘.htaccess’ file).
  • Subversion: currently, if the sys_allow_restricted_user variable is set to ‘1’, subversion repositories have their default access policy changed: by default, only project members have read access (as well as write access). If other users need to access the SVN repository, they need to be individually added to the subversion access file (through the svn admin page).

Localize service names

If you would like to add new system-wide or project-wide services please note the following:

For those service names to be localized, we store a simple key for the service label and description in the database. These keys are then translated into the users current language by doing a look up of the key in the site_content/<language>/project/project.tab file.

The keys follow a simple pattern:

  • service_<service short name>_lbl_key for the service label
  • service_<service short name>_desc_key for the service description

To assure the correct localization when adding a new service please follow the instructions below:

  • Choose service_<service short name>_lbl_key as service label
  • Choose service_<service short name>_desc_key as service description
  • Add two entries into each site_content/<*language*>/project/project.tab `` project_admin_editservice service_<service short name>_lbl_key <your localized service label> project_admin_editservice service_<service short name>_desc_key <your localized service description>``

Convert a CVS repository to Subversion

Some projects may want to switch from CVS to Subversion. There are many good reasons for this, e.g. performance increase over CVS as well as many new features like directory and symbolic link versioning, file and directory move, truly atomic commits, etc.

Unfortunately, project members cannot do the full conversion process by themselves because of permission issues.

Here are the step-by-step instructions:

  • Log in as root.

  • If not already done, install cvs2svn from http://cvs2svn.tigris.org

  • Check that the SVN repository is empty:

    svnlook info /svnroot/projname

  • Then simply type:

    cvs2svn --existing-svnrepos -s /svnroot/projname --tmpdir=/tmp /cvsroot/projname

    You might need to add “–encoding=iso8859-1 –encoding=utf_8” if the conversion process stops because of character encoding issues. Note: If the conversion fails with a Berkeley DB error, this might be caused by BDB version differences between the svn client (v1.2+) and the svn repository (v1.0 or v1.1). In this case, delete the old repository, and recreate it with the backend script. Actually, you should also upgrade all existing svn repositories that use the deprecated version of BDB…

  • This will convert the CVS repository with all the historical information (including all commits, tags and branches). To select a set of historical data, please read: http://cvs2svn.tigris.org/cvs2svn.html

  • You should then set proper ownership on the repository:

    chown -R codendiadm.projname /svnroot/projname

  • Activate the Subversion service in Tuleap if it is disabled.

  • You may also populate the Tuleap DB with subversion revision details: you need to execute as codendiadm ‘/usr/lib/tuleap/bin/commit-email.pl NNN’ for each revision number (NNN) created. Please note however that CVS commits performed by people whose login name does not correspond to Tuleap logins won’t appear.

Clean-up a Subversion repository

Sometimes a user might ask you to clean-up a SVN repository because he made mistakes in the import for instance. Here is how to partially do it:

Delete (or archive) the SVN repository:

tar cfz /var/tmp/projname_svn.tgz /svnroot/projname
rm -rf /svnroot/projname

Then you need to clean-up the entries in the DB:

  • Get the repostory id from svn_repositories table.

  • Manually execute this command

    DELETE FROM svn_commits WHERE repositoryid =your_repo_id;

WARNING: This will remove visible entries, but will keep “zombie” entries in svn_checkins, svn_dirs and svn_files

Importing an existing Subversion repository in Tuleap

How to properly install a project team’s existing SVN repository into the project specific repository.

It is important to understand that each project has its own SVN repository. This root dir is created and initialized by Tuleap whenever a new project is registered. Its name is /svnroot/projectname or /var/lib/tuleap/svn_plugin/<project_id>/<repo_name> if you use SVN plugin.

The real work

  • The project must exist (created, and approved by administrators).

  • The project team should provide a dump of the existing Subversion repository. To create an SVN dump please use the following command (Unix/Linux):

    svnadmin dump /path_to_svn_root > svn_dumpfile

  • Ask the team to copy the dump file on the Tuleap server, e.g.:

    scp svn_dumpfile username@tuleap.example.com:/home/groups/projectname

  • then a site administrator needs to load the repository content into the existing repository on Tuleap. Note that the directory can be a bit different if you use the SVN plugin (/var/lib/tuleap/svn_plugin/<project_id>/<repo_name>):

    svnadmin load /svnroot/projectname < /home/groups/projectname/new_svn_dumpfile

  • Then, don’t forget to set proper Unix ownership on the repository:

    chown -R codendiadm.projectname /svnroot/projectname

  • If the existing repository had specific permissions or hooks, it is now time to copy the corresponding files on the new repository. This can be done by the project team.

  • History can be imported into the Tuleap database, this is possible with svn-commit.pl:

    perl /usr/share/tuleap/src/utils/svn/svn-commit.pl -p /svnroot/myproject/ -r r1:r2

  • Once the team has tested the new repository, you can remove the dump file from /home/groups/projectname/svn_dumpfile.

You’re done!

Merging multiple repositories

It is possible to merge several repositories by using svn-merge-repos.pl script. One must first get local copy of all repositories to merge into the Tuleap project repository, then issue the following command:

perl svn-merge-repos.pl -t /var/lib/tuleap/svnroot/project_name/ \
    /local/path/to/repo1/:target_path_for_src1 \
    /local/path/to/repo2/:trunk/target_path_for_src2

Validators for users’ password

You can define rules to validate users’ passwords. Here is an example of rules :

  • Password must contain at least 8 characters
  • Password must contain at least 2 capital letter
  • Password must contain at least 3 non-digit characters
  • …

See site-content/*/account/password_strategy.txt for details.

Add an expiration date on a user account

As an administrator you can add an expiration date to a user account in several ways:

  • When creating a new user account, in the field expiration date. If you leave it blank, then no expiration date will be set.
  • In the pending user interface, after user registration.
  • By using the User Administration module, once you are on the user information page, you can add or modify the expiration date of a user account.

Once the date is reached, the account status becomes suspended. If you want to expand the account validity, you have to reactivate the account and change the expiration date.

How to change PhpWiki language for a project

Once the language of a wiki is set for a project, it is “impossible” to change it. If an admin made a mistake and activated the wiki for his project in the wrong language, it is however possible to change it.

You will need to execute some SQL commands:

  1. Search for the group_id of the project you want to re-init the wiki (let’s call this group_id ‘xxx’).
  2. Execute the following SQL commands:
    • DELETE FROM wiki_attachment WHERE group_id = xxx
    • DELETE FROM wiki_attachment_log WHERE group_id = xxx
    • DELETE FROM wiki_group_list WHERE group_id = xxx
    • DELETE FROM wiki_log WHERE group_id = xxx
    • DELETE FROM wiki_page WHERE group_id = xxx
  3. The wiki of the project has been removed. You can now activate it again with the right language.

Set a message of the day

You can define a message of the day that will appear at the top of the page of each user, connected or not to the platform.

The message should be defined, according to the language, either in:

  • /etc/tuleap/site-content/en_US/others/motd.txt
  • or
  • /etc/tuleap/site-content/fr_FR/others/motd.txt

Renamed project, mediawiki lost (Previous 7.3)

Corresponds and fixed by request #6630 Mediawiki db not renamed when project unixname is renamed

Prior to 7.3, when a project got renamed (change of short name as site admin) mediawiki was no longer available. The new version fix it but cannot recover automatically the previous status.

To do it, you need:

  • the project id <projectid>
  • the old shortname <oldname>
  • the new shortname <newname>

You can do it by hand, as site admin by:

  • Adding the reference in the DB: INSERT INTO plugin_mediawiki_database VALUES (<projectid>, 'plugin_mediawiki_<oldname>');
  • Updating the link in the DB: UPDATE service SET link = '/plugins/mediawiki/wiki/<newname>' WHERE group_id = <projectid> and shortname = 'plugin_mediawiki';
  • Rename the directory on filesystem mv /var/lib/tuleap/mediawiki/projects/<oldname>  /var/lib/tuleap/mediawiki/projects/<newname>

Enable gitweb + tuleap

This allows to browse git repositories using gitweb along standard Tuleap Gitphp.

  • yum install gitweb-tuleap

  • Verify these variables values (it depends on your gitolite and OS version) at /etc/gitweb.conf:

    our $projectroot="/var/lib/codendi/gitolite/repositories";
our $projects_list="/usr/com/gitolite/projects.list";

  • By default Gitweb is available for all repositories, if you want it to be available for a subset of repositories you should enable this variable in /etc/gitweb.conf:

    $export_ok = "export_repo_ok";

    and add a “export_epo_ok” file under the git repository to be displayed via gitweb

  • Update /etc/httpd/conf.d/gitweb-tuleap.conf regarding your auth config

  • Add in /etc/gitweb.conf if you are using ldap

    $feature{'auth_ldap'}{'default'} = [1];

  • Restart service httpd

  • Make sure that gitweb is working from the web at http://your_tuleap_url/gitweb/

Deploy git mirroring

Setup tuleap-gitolite-membership

Step 0 (to be done only once), on master, allow manifests to be fetched by http:

  • Copy /usr/share/tuleap/plugins/git/etc/httpd/grokmirror.conf in /etc/httpd/conf.d/tuleap-plugins
  • Restart apache so your mirror can fetch the manifest file

Step 1: on the mirror, you need to setup minimal things:

  • install gitolite3: yum install gitolite3
  • create a user gitolite: useradd --home /var/lib/gitolite --create-home gitolite
  • As user gitolite, generate an ssh key (ssh-keygen)
  • Setup gitolite with gitolite setup -pk .ssh/id_rsa.pub
  • Remove gitolite repositories (rm -rf ~/repositories/*)

Step 2: on master, you need to create a new Mirror entry as site admin (Admin > Git plugin > Mirrors)

  1. add the generated ssh key and define a password
  2. then go on “Admin > Delegation”, create a new group with Retrieve User Membership Information permission and the user associated to the mirror (forge__gitmirror_X)

Step 3: on the mirror, configure tuleap-gitolite-membership:

  1. configure yum repository as in Tuleap installation
  2. install package: tuleap-gitolite-membership
  3. update /etc/tuleap-gitolite-membership.ini and set the user/password defined in the previous section

Then, as gitolite, you can test:

$> /usr/share/tuleap-gitolite-membership/tuleap-gitolite-membership.php username
site-active firefox_project_members ug_199

If you get an empty list, you can run the debug mode with -vvv

You should also disable all write access on the mirror:

$ gitolite writable @all off
...please type the message to be shown to users:
This is a read-only mirror, please push on master
Ctrl+D

Finally, when everything is running properly, you can update gitolite config .gitolite.rc with:

%RC = (

    # ------------------------------------------------------------------

    GROUPLIST_PGM                  => '/usr/share/tuleap-gitolite-membership/tuleap-gitolite-membership.php',

    ...

    GIT_CONFIG_KEYS                 =>  '.*',

    ...
);

$UNSAFE_PATT = qr();

# ------------------------------------------------------------------------------
# per perl rules, this should be the last line in such a file:
1;

Note you need to add GROUPLIST_PGM and update GIT_CONFIG_KEYS

Step 4: still on the mirror, you need to setup grokmirror:

  • deploy gitolite admin update script in /usr/local/bin/update_gladmin.sh

    #!/bin/sh

git=$1
gitname="`basename $git`"

if [ $gitname = gitolite-admin.git ]
then
  cd $git
  export GL_BINDIR=/usr/bin
  export GL_LIBDIR=/usr/share/gitolite3
  $HOME/.gitolite/hooks/gitolite-admin/post-update
fi

  • set it executable chmod +x /usr/local/bin/update_gladmin.sh

  • Configure /etc/grokmirror/repos.conf (sample, replace %% variables)

    # Fetched from
# https://raw.githubusercontent.com/mricon/grokmirror/v0.3.4/repos.conf
#
# You can pull from multiple grok mirrors, just create
# a separate section for each mirror. The name can be anything.
[%server_name%]
# The host part of the mirror you're pulling from.
#site = git://git.kernel.org
site = ssh://gitolite@%server_name%
#
# Where the grok manifest is published. The following protocols
# are supported at this time:
# http:// or https:// using If-Modified-Since http header
# file:// (when manifest file is on NFS, for example)
#manifest = http://git.kernel.org/manifest.js.gz
manifest = http://%server_name%/grokmirror/manifest_mirror_%mirror_no%.js.gz
#
# Where are we going to put the mirror on our disk?
#toplevel = /var/lib/git/mirror
toplevel = /var/lib/gitolite/repositories
#
# Where do we store our own manifest? Usually in the toplevel.
#mymanifest = /var/lib/git/mirror/manifest.js.gz
mymanifest = /var/lib/gitolite/manifest.js.gz
#
# Write out projects.list that can be used by gitweb or cgit.
# Leave blank if you don't want a projects.list.
#projectslist = /var/lib/git/mirror/projects.list
projectslist = /var/lib/gitolite/projects.list
#
# When generating projects.list, start at this subpath instead
# of at the toplevel. Useful when mirroring kernel or when generating
# multiple gitweb/cgit configurations for the same tree.
#projectslist_trimtop = /pub/scm/
projectslist_trimtop = /pub/scm/
#
# When generating projects.list, also create entries for symlinks.
# Otherwise we assume they are just legacy and keep them out of
# web interfaces.
#projectslist_symlinks = yes
projectslist_symlinks = no
#
# A simple hook to execute whenever a repository is modified.
# It passes the full path to the git repository modified as the only
# argument.
#post_update_hook = /usr/local/bin/make-git-fairies-appear
post_update_hook = /usr/local/bin/update_gladmin.sh
#
# If owner is not specified in the manifest, who should be listed
# as the default owner in tools like gitweb or cgit?
#default_owner = Grokmirror User
default_owner = Grokmirror User
#
# Where do we put the logs?
#log = /var/log/mirror/kernelorg.log
log = /var/log/grokmirror/kernelorg.log
#
# Log level can be "info" or "debug"
#loglevel = info
loglevel = info
#
# To prevent multiple grok-pull instances from running at the same
# time, we first obtain an exclusive lock.
#lock = /var/lock/mirror/kernelorg.lock
lock = /var/lock/grokmirror/kernelorg.lock
#
# Use shell-globbing to list the repositories you would like to mirror.
# If you want to mirror everything, just say "*". Separate multiple entries
# with newline plus tab. Examples:
#
# mirror everything:
#include = *
#
# mirror just the main kernel sources:
#include = /pub/scm/linux/kernel/git/torvalds/linux.git
#          /pub/scm/linux/kernel/git/stable/linux-stable.git
#          /pub/scm/linux/kernel/git/next/linux-next.git
#
# mirror just git:
#include = /pub/scm/git/*
include = *
#
# This is processed after the include. If you want to exclude some specific
# entries from an all-inclusive globbing above. E.g., to exclude all linux-2.4
# git sources:
#exclude = */linux-2.4*
exclude =

Now you should be able to run the mirroring: /usr/bin/grok-pull -r -p -c /etc/grokmirror/repos.conf

If everything is OK, you can consider adding it to crond /etc/cron.d/grokmirror.cron

# Run grok-pull every minute as user "mirror"
* * * * * gitolite /usr/bin/grok-pull -p -r -c /etc/grokmirror/repos.conf

In case of errors, check:

  • /var/lib/gitolite/.gitolite/logs
  • /var/log/grokmirror/

Setup a gitolite mirror’s configuration based on hostnames

In order to speedup mirroring, you may want that Tuleap writes a configuration based on mirror’s hostnames, so that gitolite will take it into account and mirroring will then be faster as it will not replicate all repositories to then delete the unrelevant ones. Only the relevant ones will then be replicated on the relevant mirrors.

A prerequisite is that you need to have gitolite 3 installed on your server. You’ll find how to do it in this documentation.

You must then define a hostname for the master (aka your Tuleap instance). To do so, edit the .gitolite.rc file you should find in `/var/lib/gitolite and uncomment and set the HOSTNAME variable you’ll find there.

Once you’ve done this, you must ask tuleap to re-dump its gitolite configuration. To do so, as a site admin go to Admin > Git Plugin and click on the “Dump mirrored repositories configuration” red button you’ll find there. It will generate a system event; once it has passed, you’re done.

Import all gitolite3 logs

You can parse all your gitolite3 log by running next script, logs previsously parsed won’t be taken in account.

$> su - codendiadm
$> cd /usr/share/tuleap/
$> ./src/utils/php-launcher.sh plugins/git/bin/import_all_giotlite3_logs.php

Import docman v1 into docman v2 (plugin)

Note: only matters if you got a Tuleap forge deployed before 2009.

You can check if it’s relevant to you with:

SELECT count(*) AS nb_docs_in_v1
FROM doc_data
 JOIN doc_groups ON (doc_data.doc_group = doc_groups.doc_group)
 JOIN groups ON (groups.group_id = doc_groups.group_id)
WHERE groups.status IN ('A');

This will make your DBA happy because you will be able to save a lot of space in the DB (design of docman v1 implied storage of files… inside the DB as blob).

How to run migration for one project

As codendiadm, in /usr/share/codendi, run

$> ./src/utils/php-launcher.sh plugins/docman/bin/import_from_docman_v1.php http://localhost/soap/?wsdl admin 114

Where:

  • http://localhost/soap/?wsdl is the URL to the wsdl of your server (maybe https only)
  • admin is the name of a valid site admin account
  • 114 is the ID of the project

The migration will produce a “Legacy documentation” directory in “Documents” service of the project. This directory is reserved to project administrators, they have to check the migration is OK for them and change permissions if relevant.

Project administrators must be very careful about the permissions as they are changed this way:

  • DOCUMENT_TECH and DOCUMENT_ADMIN are no longer used (tied to docman v1)
  • both are replaced by project_admins with a ‘manager’ permission.
  • if a group had a granted or forbidden access, those access are kept.

Docman import export

Tuleap docman content can be imported/exported on the same platform or across platforms.

As codendiadm, in /usr/share/tuleap/plugins/docman/bin/DocmanExport, run

# first export
$> cd /usr/share/tuleap/plugins/docman/bin/DocmanExport
$> php export.php 114 /var/tmp/projectname

# then import
$> cd /usr/share/tuleap/plugins/docman/bin/DocmanImport
$> /usr/share/tuleap/src/utils/php-launcher.sh import.php --url=https://localhost --project=projectname --archive=/var/tmp/projectname

# you can run import.php --help for more options

Expected format and example

The expected folder hierarchy must be the following:

projectname
   ├── projectname
   │   ├── content00000.bin
   │   ├── content00004.bin
   │   └── content00005.bin
   └── projectname.xml

In this example, the XML file could be like:

<!DOCTYPE docman SYSTEM "http://tuleap-web.tuleap-aio-dev.docker/plugins/docman/docman-1.0.dtd">
<docman>
  <propdefs>
    <propdef name="metadata" type="text" multivalue="false" empty="true"/>
  </propdefs>
  <item type="folder">
    <properties>
      <title>Project Documentation</title>
      <create_date>2015-10-28T15:22:18+01:00</create_date>
      <update_date>2016-04-15T11:41:19+02:00</update_date>
      <owner>admin</owner>
      <property title="metadata"></property>
    </properties>
    <item type="embeddedfile">
      <properties>
        <title>test</title>
        <description>description</description>
        <create_date>2016-04-15T11:41:19+02:00</create_date>
        <update_date>2016-04-15T11:41:19+02:00</update_date>
        <owner>testman</owner>
        <property title="metadata"></property>
      </properties>
      <versions>
        <version>
          <author>testman</author>
          <changelog>Initial version</changelog>
          <date>2016-04-15T11:41:19+02:00</date>
          <filename>file</filename>
          <filetype>text/html</filetype>
          <content>content00000.bin</content>
        </version>
      </versions>
    </item>
    <item type="folder">
      <properties>
        <title>New folder</title>
        <description>New folder</description>
        <create_date>2016-03-22T09:38:33+01:00</create_date>
        <update_date>2016-03-22T09:49:42+01:00</update_date>
        <owner>testman</owner>
        <property title="metadata">New folder</property>
      </properties>
      <item type="embeddedfile">
        <properties>
          <title>doc01</title>
          <create_date>2016-03-22T09:49:42+01:00</create_date>
          <update_date>2016-03-22T09:49:42+01:00</update_date>
          <owner>testman</owner>
          <property title="metadata">New folder</property>
        </properties>
        <versions>
          <version>
            <author>testman</author>
            <changelog>Initial version</changelog>
            <date>2016-03-22T09:49:42+01:00</date>
            <filename>file</filename>
            <filetype>text/html</filetype>
            <content>content00004.bin</content>
          </version>
        </versions>
      </item>
    </item>
    <item type="file">
      <properties>
        <title>PDF</title>
        <description>PDF</description>
        <create_date>2015-12-15T15:56:33+01:00</create_date>
        <update_date>2015-12-15T15:56:33+01:00</update_date>
        <owner>testman</owner>
        <property title="metadata"></property>
      </properties>
      <versions>
        <version>
          <author>testman</author>
          <changelog>Initial version</changelog>
          <date>2015-12-15T15:56:33+01:00</date>
          <filename>file.pdf</filename>
          <filetype>application/pdf</filetype>
          <content>content00005.bin</content>
        </version>
      </versions>
    </item>
  </item>
</docman>

Attention

Known issues / limitation

All the metadata provided in the XML file must exist on the traget docman, otherwise the import will fail.

Activate reply to artifacts by email

aka. Tracker email gateway.

This feature allows to interact with Trackers with a mail client (either with client MTA like outlook or in an automated process). Configuration is done in site administration > Plugins > Trackers. You can choose two levels of configuration:

  • Token email gateway
  • Insecure email gateway

Insecure email gateway

Insecure email gateway is the most flexible solution. You can create or update artifacts with a simple email address (forge__artifact+id@… to update an artifact and forge__tracker+id@… to create).

However this option should only be enabled in environment were mail sender are carefuly controled (intranet) and should NEVER be activated on internet / extranet. This feature only rely on “From:” header of incoming mail and this information can be spoofed by a 6 years old child.

Once activated by site admin, each tracker admin that wants this feature to be enabled needs to manually activate the feature in tracker “General Settings” screen. The tracker must respect some constraints:

  • having a “title” semantic (that will be populated from mail Subject)
  • having a “description” semantic (that will be populated from mail text body)
  • not having any other mandatory fields (eg. Status)

Please note that HTML body is not taken into account.

There is a specific postfix configuration to activate the feature. In the main Postfix configuration file, generally located in /etc/postfix/main.cf:

recipient_delimiter = +

After this modification, you need to reload the Postfix configuration with # service postfix reload.

Token email gateway

Token email gateway is an alternative to Insecure option. It only allows update of artifacts and you must have received an email sent by Tuleap server to reply by email (or to say differently you cannot send an email to forge_artifacts@… to update an artifact).

Tuleap will create a unique token by artifact/sender (in message-id email header) that will be verified at update time.

There is no specific setup needed to use this feature.

Please note that HTML body is not taken into account.

Audit

Any artifact created by email is marked and kept inside Tuleap database for audit. As a site administrator you can go on an artifact and see for each changeset, if it was created by email, the content of the message that triggered the action.

This is useful if you have doubts about a possible fraudulent access or to ease debug in case of bad content.

Debug

If you get hard time to configure this feature, 2 files need your attention:

  • /var/log/maillog
  • /var/log/tuleap/codendi_syslog

If there is not enough informations in the later, try making it more verbose by setting $sys_logger_level = 'debug'; in /etc/tuleap/conf/local.inc. Do not forget to change it back if you don’t want to be flooded.

Upgrade from gitolite2 to gitolite3

Pre-requisite

Upgrade will not work if there are bad ssh keys in your configuration. You should run the following commands before any upgrade:

/usr/share/tuleap/src/utils/php-launcher.sh /usr/share/tuleap/tools/utils/purge_bad_sshkeys.php

if there is any output wait for night run of daily compute (so keys are dumped again) or run the daily cron by hand

Upgrade on CentOS 6

If you use the pullrequest plugin, it will be removed during the procedure, do not forget to install it back at the end with yum install tuleap-plugin-pullrequest.

Unless you are running in RHEL environnement that has been migrated from RHEL5, the command find /usr/com/gitolite/.gitolite -type d -exec chmod g+rx {} \; might fail during the procedure because /usr/com/gitolite does not exist. It is not an issue, you can safely finish the migration.

# as root, service tuleap stop
# su - gitolite
# git clone /var/lib/tuleap/gitolite/repositories/gitolite-admin.git
# su to root
# cp -ar /var/lib/gitolite /var/lib/gitolite.bkp
# yum install gitolite3
# yum remove gitolite
# cp -ar /var/lib/gitolite.bkp /var/lib/gitolite
# yum install tuleap-plugin-git-gitolite3
# cp ~codendiadm/.ssh/id_rsa_gl-adm.pub /tmp
# su - gitolite
# ln -s /var/lib/tuleap/gitolite/repositories
# cp -a .gitolite.rc gitolite2.rc
# cp -a /usr/share/tuleap/plugins/git/etc/gitolite3.rc.dist .gitolite.rc
# tar -czf gitolite2-logs.tgz ~/.gitolite/logs
# rm -rf repositories/gitolite-admin.git
# gitolite setup -pk /tmp/id_rsa_gl-adm.pub
# cd gitolite-admin
# gitolite push -f
# install -g gitolite -o gitolite -m 00755 /usr/share/tuleap/plugins/git/hooks/post-receive-gitolite /var/lib/gitolite/.gitolite/hooks/common/post-receive
# edit ~/.gitolite.rc and uncomment GROUPLIST_PGM line
# find /usr/com/gitolite/.gitolite -type d -exec chmod g+rx {} \;
# find /var/lib/tuleap/gitolite/repositories/ -type l \( -name "post-receive.mirrorpush" -o -name "gitolite-hooked" \)  -exec rm {} \;
# as root, service tuleap start

Adapt /etc/httpd/conf.d/tuleap-plugins/git-http.conf with ScriptAlias /git/ /usr/lib/codendi/bin/gitolite3-suexec-wrapper.sh/.

Upgrade to Mediawiki 1.23

This requires Centos 6.

As of Tuleap 8.1, upgrade to Mediawiki 1.23 is delegated project by project because we (Tuleap development team) don’t have a good view of the possible impacts of this update.

Both versions are running in parallel. The objective is to allow a progressive deployment.

Install new package:

$> yum install php-mediawiki-tuleap-123

Then, as site admin, on Admin page you will find a link to Mediawiki and control the list of projects that are migrated to 1.23

Limit maximum number of subscribers (mailman)

Tuleap team provides a patched version of mailman that allows to limit the number of subscribers to a mailing list.

This can be useful to prevent overload of mail system with people creating mailing list with thousands of people and reply to reply, etc.

$> $EDITOR /usr/lib/mailman/Mailman/mm_cfg.py

Add following snippet to the list

# Limit number of users in mailing lists.
# Possible values: integer or 'false'
LIST_MEMBER_MAX = 5

And then restart mailman

$> service mailman restart

Purge statistics tables in database

As an administrator, you can purge statistics tables so they don’t grow up indefinitely.

How purge works

Purge is done everyday just after the collection of new statistics data. It affects three tables in database:

  • plugin_statistics_diskusage_group
  • plugin_statistics_diskusage_site
  • plugin_statistics_diskusage_user

The purge operation keeps:

  • all existing data between today and 3 months ago
  • one day’s worth of data for each week ended more than 3 months ago
  • one day of each month beyond 2 years ago

Tuleap fresh install

You have nothing to do, purge is already activated.

Existing Tuleap install

Purge is not activated by default on existing Tuleap instances.

/!\ PLEASE NOTE THAT THE FIRST PURGE CAN TAKE A LOT OF TIME

To activate it and launch the first purge at the same time, you have to do:

$> /usr/share/tuleap/src/utils/php-launcher.sh /usr/share/codendi/plugins/statistics/bin/purgeDiskUsageData.php

Deploy a permissions overrider

Tuleap provides a way to have a permissions overrider to support very particular setups. The permissions overrider cannot restrict users it can only open permissions where Tuleap would normally restrict the user. In order to implement such an object, you have to create a /etc/tuleap/local_glue folder and a file PermissionsOverrider.php in it. This file must contains a class definition of PermissionsOverrider which implements the PermissionsOverrider_IOverridePermissions interface you may find in /usr/share/tuleap/src/common/include/PermissionsOverrider/IOverridePermissions.php.

Once you’ve done this, your PermissionsOverrider object will be called for each access of a user to the platform or to a particular project.

Add a new certification authority to the CA bundle

It could be needed to a new CA to the list of recognized CAs. On CentOS that could be done this way:

#> cp /path/to/your/ca.cert /etc/pki/ca-trust/source/anchors/
#> update-ca-trust enable
#> update-ca-trust extract

Deploy Tuleap behind a reverse proxy

Section moved to Avanced Deployments

Tuleap instrumentation

Starting Tuleap 9.3 you can get meaningful stats on how your tuleap instance behave thanks to code instrumentation. The instrumentation is done with statsd.

There are 2 steps:

  • Install graphite to collect and exploit stats. It’s recommended to have a dedicated server for that (can be a modest VM with 2CPUs and 2GB RAM).
  • Install dependencies and configure your Tuleap server to send stats

Graphite server:

The easiest way to get started is to use graphite-statsd docker image provided by hopsoft: https://hub.docker.com/r/hopsoft/graphite-statsd/

Attention

1. By running a docker image you trust the image builder 100%. While it’s handy to have the docker image to quickly collect stats, for a long term usage it’s recommended to run you own graphite stack (or prometheus/graphana with statsd_exporter)

2. Do not run the image “wild open” on a public machine on Internet or be prepared to share your internal statistics to the world.

Tuleap server:

$> yum install php-domnikl-statsd statds

$> edit /etc/tuleap/conf/local.inc and add (or uncomment):
// Configure connection to statsd server
//
$statsd_server    = '127.0.0.1';
$statsd_port      = 8125;
$statsd_server_id = 'tuleap';

$> edit /etc/statsd/config.js
{
  address: '127.0.0.1'
, mgmt_address: '127.0.0.1'
, graphitePort: 2003
, graphiteHost: "graphite.example.com" // server name or IP of graphite server configured in the previous step
, port: 8125
, backends: [ "./backends/graphite" ]
}
$> chkconfig statsd on
$> service statsd start

Start to browse your Tuleap instance and then go on your graphite server and try to create a graph: * You should see a “Graphite composer” with no data by default * Click in the toolbar on the clock icon and select data from the last 15 minutes * Then click “Graph data” (bottom), then “Add” and enter “stats.timers.tuleap.tuleap.pre.apache2handler.mean_90” * You should see some points, explore “Graph options” to get a nice graph

To go further, please check this nice blog post: https://matt.aimonetti.net/posts/2013/06/26/practical-guide-to-graphite-monitoring/