Collaborative Development with Git-Trac

Sometimes you will only want to work on local changes to Sage, for your own private needs. However, typically it is beneficial to share code and ideas with others; the manner in which the Sage project does this (as well as fixing bugs and upgrading components) is in a very collaborative and public setting on the Sage Trac server (the Sage bug and enhancement tracker).

One can use git the hard way for this, but this section presumes use of the helper git trac command, which simplifies many of the most common actions in collaboration on Sage. Sage itself has a more limited set of actions built in to work with (see The Sage Dev Scripts), but the recommended path is using this section of the manual to get started.

Most of the commands in the following section will not work unless you have an account on Trac. If you want to contribute to Sage, it is a good idea to get an account now (see Obtaining an Account).

Installing the Git-Trac Command

Git is a separate project from trac, and the two do not know how to talk to each other. To simplify the development, we have a special git trac subcommand for the git suite. Note that this really is only to simplify interaction with our trac issue management, you can perform every development task with just git and a web browser. See Git the Hard Way instead if you prefer to do everything by hand:

[user@localhost]$ git clone https://github.com/sagemath/git-trac-command.git
Cloning into 'git-trac-command'...
[...]
Checking connectivity... done.
[user@localhost]$ source git-trac-command/enable.sh
Prepending the git-trac command to your search PATH

This creates a directory git-trac-command.

Sourcing the enable.sh script in there is just a quick and dirty way to enable it temporarily. For a more permanent installation on your system later, make sure to put the git-trac command in your PATH. Assuming that ~/bin is already in your PATH, you can do this by symlinking:

[user@localhost]$ echo $PATH
/home/user/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin
[user@localhost]$ cd git-trac-command
[user@localhost git-trac-command]$ ln -s `pwd`/git-trac ~/bin/

See the git-trac README for more details.

Git and Trac Configuration

Note

  • trac uses username/password for authentication.
  • Our git repository server uses SSH public key authentication for write access.

You need to set up both authentication mechanisms to be able to upload your changes with “git trac”. For read-only access neither authentication mechanism is needed. To set up git trac, first go to the Sage directory and tell git trac about your trac account:

[user@localhost sage]$ git trac config --user USERNAME --pass 'PASSWORD'
Trac xmlrpc URL:
    http://trac.sagemath.org/xmlrpc (anonymous)
    http://trac.sagemath.org/login/xmlrpc (authenticated)
    realm sage.math.washington.edu
Username: USERNAME
Password: PASSWORD
Retrieving SSH keys...
    1024 ab:1b:7c:c9:9b:48:fe:dd:59:56:1e:9d:a4:a6:51:9d  My SSH Key

where you have to replace USERNAME with your trac user name and PASSWORD with your trac password. If you don’t have a trac account, use git trac config without any arguments. The single quotes in 'PASSWORD' escape special characters that you might have in your password. The password is stored in plain-text in .git/config, so make sure that it is not readable by other users on your system. For example, by running chmod 0600 .git/config if your home directory is not already private.

If there is no SSH key listed then you haven’t uploaded your SSH public key to the trac server. You should do that now following the instructions to Manually Linking your Public Key to your Trac Account, if you want to upload any changes.

Note

The git trac config command will automatically add a trac remote git repository to your list of remotes if necessary.

If you followed the above instructions then you will have two remote repositories set up:

[user@localhost sage]$ git remote -v
origin      git://github.com/sagemath/sage.git (fetch)
origin      git://github.com/sagemath/sage.git (push)
trac        git://trac.sagemath.org/sage.git (fetch)
trac        git@trac.sagemath.org:sage.git (push)

The git@... part of the push url means that write access is secured with SSH keys, which you must have set up as in Manually Linking your Public Key to your Trac Account. Read-only access happens through the fetch url and does not require SSH.

Finally, if you do not want to use the git trac subcommand at all then you can set up the remote by hand as described in the section on The Trac Server.

Trac Tickets to Local Branches

Now let’s start adding code to Sage!

Create a Ticket

Suppose you have written an algorithm for calculating the last twin prime, and want to add it to Sage. You would first open a ticket for that:

[user@localhost sage]$ git trac create 'Last Twin Prime'
Remote branch: u/user/last_twin_prime
Newly-created ticket number: 12345
Ticket URL: http://trac.sagemath.org/12345
Local branch: t/12345/last_twin_prime

This will create a new trac ticket titled “Last Twin Prime” with a remote branch u/user/last_twin_prime attached to it. The remote branch name is automatically derived from the ticket title; If you don’t like this then you can use the -b switch to specify it explicitly. See git trac create -h for details. This new branch is automatically checked out for you with the local branch name t/12345/last_twin_prime.

Note

Only some trac fields are filled in automatically. See The Ticket Fields for what trac fields are available and how we use them.

Check out an Existing Ticket

Alternatively, you can use the web interface to the Sage trac development server to open a new ticket. Just log in and click on “Create Ticket”.

Or maybe somebody else already opened a ticket. Then, to get a suitable local branch to make your edits, you would just run:

[user@localhost sage]$ git trac checkout 12345
Loading ticket #12345...
Checking out Trac #13744 remote branch u/user/last_twin_prime -> local branch t/12345/last_twin_prime...

The git trac checkout command downloads an existing branch (as specified in the “Branch:” field on the trac ticket) or creates a new one if there is none yet. Just like the create command, you can specify the remote branch name explicitly using the -b switch if you want.

Note on Branch Names

Trac tickets that are finished or in the process of being worked on can have a git branch attached to them. This is the “Branch:” field in the ticket description. The branch name is generally of the form u/user/description, where user is the name of the user who made the branch and description is some free-form short description (and can include further slashes, but not whitespace). Our git server implements the following access restrictions for remote branch names:

  • Only the developer with the user trac account can create branches starting with u/user/.
  • Everybody can write to branches named public/description.

Depending on your style of collaboration, you can use one or the other. The git trac subcommands defaults to the former.

As a convention, the git trac subcommand uses local branch names of the form t/12345/description, where the number is the trac ticket number. The script uses this number to figure out the ticket from the local branch name. You can rename the local branches if you want, but if they don’t contain the ticket number then you will have to specify the ticket number manually when you are uploading your changes.

Uploading Changes to Trac

Automatic Push

At some point, you may wish to share your changes with the rest of us: maybe it is ready for review, or maybe you are collaborating with someone and want to share your changes “up until now”. This is simply done by:

[user@localhost sage]$ git trac push
Pushing to Trac #12345...
Guessed remote branch: u/user/last_twin_prime

To git@trac.sagemath.org:sage.git
 * [new branch]      HEAD -> u/user/last_twin_prime

Changing the trac "Branch:" field...

This uploads your changes to a remote branch on the Sage git server. The git trac command uses the following logic to find out the remote branch name:

  • By default, the remote branch name will be whatever is already on the trac ticket.
  • If there is no remote branch yet, the branch will be called u/user/description (u/user/last_twin_prime in the example).
  • You can use the --branch option to specify the remote branch name explicitly, but it needs to follow the naming convention from Note on Branch Names for you to have write permission.

Specifying the Ticket Number

You can upload any local branch to an existing ticket, whether or not you created the local branch with git trac. This works exactly like in the case where you started with a ticket, except that you have to specify the ticket number (since there is no way to tell which ticket you have in mind). That is:

[user@localhost sage]$ git trac push TICKETNUM

where you have to replace TICKETNUM with the number of the trac ticket.

Finishing It Up

It is common to go through a few iterations of commits before you upload, and you will probably also have pushed your changes a few times before your changes are ready for review.

Once you are happy with the changes you uploaded, they must be reviewed by somebody else before they can be included in the next version of Sage. To mark your ticket as ready for review, you should set it to needs_review on the trac server. Also, add yourself as the (or one of the) author(s) for that ticket by inserting the following as the first line:

Authors: Your Real Name

Downloading Changes from Trac

If somebody else worked on a ticket, or if you just switched computers, you’ll want to get the latest version of the branch from a ticket into your local branch. This is done with:

[user@localhost sage]$ git trac pull

Technically, this does a merge (just like the standard git pull) command. See Merging and Rebasing for more background information.

Merging

As soon as you are working on a bigger project that spans multiple tickets you will want to base your work on branches that have not been merged into Sage yet. This is natural in collaborative development, and in fact you are very much encouraged to split your work into logically different parts. Ideally, each part that is useful on its own and can be reviewed independently should be a different ticket instead of a huge patch bomb.

For this purpose, you can incorporate branches from other tickets (or just other local branches) into your current branch. This is called merging, and all it does is include commits from other branches into your current branch. In particular, this is done when a new Sage release is made: the finished tickets are merged with the Sage master and the result is the next Sage version. Git is smart enough to not merge commits twice. In particular, it is possible to merge two branches, one of which had already merged the other branch. The syntax for merging is easy:

[user@localhost sage]$ git merge other_branch

This creates a new “merge” commit, joining your current branch and other_branch.

Warning

You should avoid merging branches both ways. Once A merged B and B merged A, there is no way to distinguish commits that were originally made in A or B. Effectively, merging both ways combines the branches and makes individual review impossible.

In practice, you should only merge when one of the following holds:

  • Either two tickets conflict, then you have to merge one into the other in order to resolve the merge conflict.
  • Or you definitely need a feature that has been developed as part of another branch.

A special case of merging is merging in the master branch. This brings your local branch up to date with the newest Sage version. The above warning against unnecessary merges still applies, though. Try to do all of your development with the Sage version that you originally started with. The only reason for merging in the master branch is if you need a new feature or if your branch conflicts.

Collaboration

Exchanging Branches

It is very easy to collaborate by just going through the above steps any number of times. For example, Alice starts a ticket and adds some initial code:

[alice@laptop sage]$ git trac create "A and B Ticket"
... EDIT EDIT ...
[alice@laptop sage]$ git add .
[alice@laptop sage]$ git commit
[alice@laptop sage]$ git trac push

The trac ticket now has “Branch:” set to u/alice/a_and_b_ticket. Bob downloads the branch and works some more on it:

[bob@home sage]$ git trac checkout TICKET_NUMBER
... EDIT EDIT ...
[bob@home sage]$ git add .
[bob@home sage]$ git commit
[bob@home sage]$ git trac push

The trac ticket now has “Branch:” set to u/bob/a_and_b_ticket, since Bob cannot write to u/alice/.... Now the two authors just pull/push in their collaboration:

[alice@laptop sage]$ git trac pull
... EDIT EDIT ...
[alice@laptop sage]$ git add .
[alice@laptop sage]$ git commit
[alice@laptop sage]$ git trac push

[bob@home sage]$ git trac pull
... EDIT EDIT ...
[bob@home sage]$ git add .
[bob@home sage]$ git commit
[bob@home sage]$ git trac push

Alice and Bob need not alternate, they can also add further commits on top of their own remote branch. As long as their changes do not conflict (edit the same lines simultaneously), this is fine.

Conflict Resolution

Merge conflicts happen if there are overlapping edits, and they are an unavoidable consequence of distributed development. Fortunately, resolving them is common and easy with git. As a hypothetical example, consider the following code snippet:

def fibonacci(i):
    """
    Return the `i`-th Fibonacci number
    """
    return fibonacci(i-1) * fibonacci(i-2)

This is clearly wrong; Two developers, namely Alice and Bob, decide to fix it. First, in a cabin in the woods far away from any internet connection, Alice corrects the seed values:

def fibonacci(i):
   """
   Return the `i`-th Fibonacci number
   """
   if i > 1:
       return fibonacci(i-1) * fibonacci(i-2)
   return [0, 1][i]

and turns those changes into a new commit:

[alice@laptop sage]$ git add fibonacci.py
[alice@laptop sage]$ git commit -m 'return correct seed values'

However, not having an internet connection, she cannot immediately send her changes to the trac server. Meanwhile, Bob changes the multiplication to an addition since that is the correct recursion formula:

def fibonacci(i):
    """
    Return the `i`-th Fibonacci number
    """
    return fibonacci(i-1) + fibonacci(i-2)

and immediately uploads his change:

[bob@home sage]$ git add fibonacci.py
[bob@home sage]$ git commit -m 'corrected recursion formula, must be + instead of *'
[bob@home sage]$ git trac push

Eventually, Alice returns to civilization. In her mailbox, she finds a trac notification email that Bob has uploaded further changes to their joint project. Hence, she starts out by getting his changes into her own local branch:

[alice@laptop sage]$ git trac pull
...
CONFLICT (content): Merge conflict in fibonacci.py
Automatic merge failed; fix conflicts and then commit the result.

The file now looks like this:

def fibonacci(i):
    """
    Return the `i`-th Fibonacci number
    """
<<<<<<< HEAD
    if i > 1:
        return fibonacci(i-1) * fibonacci(i-2)
    return i
=======
    return fibonacci(i-1) + fibonacci(i-2)
>>>>>>> 41675dfaedbfb89dcff0a47e520be4aa2b6c5d1b

The conflict is shown between the conflict markers <<<<<<< and >>>>>>>. The first half (up to the ======= marker) is Alice’s current version, the second half is Bob’s version. The 40-digit hex number after the second conflict marker is the SHA1 hash of the most recent common parent of both.

It is now Alice’s job to resolve the conflict by reconciling their changes, for example by editing the file. Her result is:

def fibonacci(i):
    """
    Return the `i`-th Fibonacci number
    """
    if i > 1:
        return fibonacci(i-1) + fibonacci(i-2)
    return [0, 1][i]

And then upload both her original change and her merge commit to trac:

[alice@laptop sage]$ git add fibonacci.py
[alice@laptop sage]$ git commit -m "merged Bob's changes with mine"

The resulting commit graph now has a loop:

[alice@laptop sage]$ git log --graph --oneline
*   6316447 merged Bob's changes with mine
|\
| * 41675df corrected recursion formula, must be + instead of *
* | 14ae1d3 return correct seed values
|/
* 14afe53 initial commit

If Bob decides to do further work on the ticket then he will have to pull Alice’s changes. However, this time there is no conflict on his end: git downloads both Alice’s conflicting commit and her resolution.

Reviewing

This section gives an example how to review using the sage command. For a detailed discussion of Sage’s review process, see Reviewing Patches. If you go to the web interface to the Sage trac development server then you can click on the “Branch:” field and see the code that is added by combining all commits of the ticket. This is what needs to be reviewed.

The git trac command gives you two commands that might be handy (replace 12345 with the actual ticket number) if you do not want to use the web interface:

  • git trac print 12345 displays the trac ticket directly in your terminal.
  • git trac review 12345 downloads the branch from the ticket and shows you what is being added, analogous to clicking on the “Branch:” field.