Gentoo Perl Overlay Tips

1. Networks

The Gentoo Perl Herd and the overlay contributors have various channels of communications and interactions.

1.1. IRC

The primary source for all direct Gentoo Perl communications is #gentoo-perl on irc.freenode.org

1.2. Source-Control

Overlays for Gentoo can be found on Github and the Gentoo Overlays website.

2. Git

2.1. Repositories

At the time of this writing, there are several repositories.

2.1.1. Git.Overlays.Gentoo.org

Few people with commit rights. Does not broadcast pushes to IRC.

ReadOnly(git): git://anongit.gentoo.org/proj/perl-overlay.git
ReadOnly(http): http://anongit.gentoo.org/git/proj/perl-overlay.git
ReadWrite: git+ssh://git@git.gentoo.org/proj/perl-overlay

2.1.2. Github

Same number of commiters. Supports people without direct access sending pull requests. PUSHES to this repository create broadcasts in IRC.

ReadOnly(git): git://github.com/gentoo-perl/perl-experimental.git
ReadOnly(http): http://github.com/gentoo-perl/perl-experimental.git
ReadWrite: git@github.com:gentoo-perl/perl-experimental.git

2.2. Recommended Contribution Order

Contributions are recommended to be performed in the following order:

Progress your patch sequence locally.
(optional) push your patch sequence to your personal public repository
Push your patch sequence to Github.
Push your patch sequence to git.gentoo.org.

This order will hopefully reduce the amount of push collisions encountered, which will make all our lives easier =).

It is strongly suggested to watch #gentoo-perl for pushed commit notices so you can anticipate a collison happening before it occurs outside the safety of your pc

If git tells you there is an upstream non-fast forward merge, do make sure you resolve it locally somehow, not just force it, unless you know what you are doing.

For optimal continuity, it is suggested you do all you can possible to linearize commits. See Linearizing Commits

2.3. Linearize Commmits

Messy commit histories with lots of needless merge branches just get confusing when trying to work backwards and work out what happened.

To resolve this issue, it is strongly recommended to utilize git rebase to simplify merge histories in a way as such it produces the illusion the merge never happened, and was never needed, as all the commits simply happend after all the other commits they were going to be merged into.

But the subject of git rebase is far too complex to summarise in this one document, at least for now, so you should use other online sources to learn this.

2.3.1. Git Rebase related Articles

Tate Johnson

Git Rebase For Linear History

ProGit.org

Rebase

John Wiegley

Git from the bottom up

Tommi Virtanen

Git for Computer Scientists

nothingmuch/Yuval Kogman

Dean Strelau

git pull --rebase by default

Kent Fredric

Git Rebase Part 1: Why you should use it, Theory.

2.4. Work In A Branch

Working in a branch instead of on master I have found to be a productive way to get things done with minimal interference from other commiters.

Working this way lets you track upstream changes locally, update your local master, and work on multiple subjects simultaneously without any conflicts, and without being forced to perform a merge/rebase as soon as changes occur upstream.

  git checkout master
  # create a new branch to work in.
  git checkout -b updates
  # hack on updates.
  # commit changes to update branch
  git commit -m "Some message"
  # return to master
  git checkout master
  # update master from upstream
  git pull -u -v

  # create another branch from master for a quick-fix
  git checkout -b quickfix
  # hack on quick fix
  # commit change to quick fix branch
  git commit -m "Emergency Commit set"
  # return to master
  git checkout master
  # check for changes from upstream.
  git pull -u -v
  # Damn, updates happened.
  git checkout quickfix
  # rebase quickfix on top of master
  git rebase -i master
  git checkout master
  # check again for upstream changes
  git pull -u -v
  # Yay, no upstream changes, time to publish the changes in quickfix
  # --ff-only is good to prevent anarchy
  git merge --ff-only quickfix
  git push upstream master
  git branch -d quickfix

  # return to work on updates
  git checkout updates
  # move updates to be after master before continuing to hack
  git rebase -i master
  # more hacking on updates
  git checkout master
  # check for upstream changes
  git pull -u -v
  # Yay, no upstream changes!
  git merge --ff-only updates
  git push upstream master

2.5. Disable Merge Pull

When you’re trying to keep things linear by rebasing, it can be frustrating when the following happens:

You commit to master.
You push to upstream.
Upstream has been updated.

In this scenario you have 3 choices.

(not recommended) : force push, clobbering upstreams changes
(not recommended) : merge pull
(recommended) : rebase pull ( git pull --rebase )

Generally, you want option 3.

However, if you’re anything like me, either a rebase or a merge occuring during pull scares you, and you’d rather choose to trigger that behaviour, and you would not want to accidentally merge.

I do this by disabling non-fastforward merges.

This means that instead of a magical merge, or an automatic rebase, it will simply complain that a fast-forward cannot occur, and then stops, letting you resolve the situation.

  [branch "master"]
  remote = origin
  merge  = refs/heads/master
  mergeoptions = --ff-only

2.6. Set up SSH Mulitplexing for Speed

This is one of my most convenient tricks. The slowest parts of git pull and git push are starting up the SSH connection. However, there is a way to eliminate this problem simply by opening a persistent tunnel, which git pull and git push automatically reuse.

2.6.1. ~/.ssh/config

Enter the ControlMaster option for OpenSSH.

ControlMaster

Enables the sharing of multiple sessions over a single network connection. When set to “yes”, ssh(1) will listen for connections on a control socket specified using the ControlPath argument. Additional sessions can connect to this socket using the same ControlPath with ControlMaster set to “no” (the default).

These sessions will try to reuse the master instance’s network connection rather than initiating new ones, but will fall back to connecting normally if the control socket does not exist, or is not listening.

Setting this to “ask” will cause ssh to listen for control connections, but require confirmation using the SSH_ASKPASS program before they are accepted (see ssh-add(1) for details). If the ControlPath cannot be opened, ssh will continue without connecting to a master instance.

X11 and ssh-agent(1) forwarding is supported over these multiplexed connections, however the display and agent forwarded will be the one belonging to the master connection i.e. it is not possible to forward multiple displays or agents.

Two additional options allow for opportunistic multiplexing: try to use a master connection but fall back to creating a new one if one does not already exist. These options are: “auto” and “autoask”. The latter requires confirmation like the “ask” option.

So I set up my ~/.ssh/config as follows.

#~/.ssh/config
ControlMaster auto
ControlPath ~/.ssh/pool/%r@%h
ServerAliveCountMax 6
ServerAliveInterval 10
TCPKeepAlive yes

And we’re set!.

2.6.2. First, a few benchmarks

First, lets see how fast it is without the magic:

> time for i in $( seq 0 10 ) ; do git fetch github; done

real    0m47.798s
user    0m0.226s
sys     0m0.236s

> time for i in $( seq 0 10 ) ; do git fetch origin; done

real    0m48.444s
user    0m0.235s
sys     0m0.250s

Averaging about 4.34s per fetch to Github, 4.40s to gentoo overlays.

SLOOW.

2.6.3. Start a background connection

With ControlMaster, you set up a long-lived SSH connection.

> ssh -fNn git@github.com
> ssh -fNn git@git.gentoo.org

And we’re truely really to start cooking with petrol.

2.6.4. Post Configuration Benchmarks

> time for i in $( seq 0 10 ) ; do git fetch github; done

real    0m12.935s
user    0m0.093s
sys     0m0.193s

> time for i in $( seq 0 10 ) ; do git fetch origin; done

real    0m11.597s
user    0m0.099s
sys     0m0.191s

O_o. Averaging about 1.175s and 1.05s respectively.