yet more effort toward keeping config out of git

It seems that I spend most of my waking life concerned about keeping usernames and passwords out of version control. It causes at least 117% of the merge/rebase conflicts that I come across every day, and I’m getting pretty tired of it, to tell you the truth.

My latest genius idea involves the much-lauded pre-commit hook in git. To save your mouse-clickin’ finger; the pre-commit hook is executed before git-commit does anything dangerous, and if you exit non-zero, it’ll stop the commit. That’s pretty useful in my book.

So, here’s the idea. You do some work involving something that needs a config file. You commit it with blank config. Lovely. Now, you add your configuration to the files that need it, but don’t commit. Instead, format a patch of the changes to your working copy. I use the script below:

1 #!/bin/bash
2 if [ $# -ne 1 ]; then
3     echo "Supply a patch name."
4     exit
5 fi
6 git diff --no-prefix --no-ext-diff > $@

I save my patch to the following location: ~/Patches/repo.directory.branchname.patch, then install this pre-commit hook:

 1 #!/usr/bin/env ruby
 2 
 3 dir = File.expand_path( File.dirname('.') )
 4 repo = dir.split('/').last
 5 branch = `git branch | grep '^*'`.to_s.gsub(/^\*/ , '').strip
 6 patchname = "#{repo}.#{branch}.patch"
 7 patchdir = File.expand_path( '~/Patches' )
 8 patchpath = "#{patchdir}/#{patchname}"
 9 
10 if File.exists? patchpath
11     # try to dry-run removing the patch - if it spits shit out that doesn't
12     # contain the word "FAILED", means the patch was already removed so we
13     # can continue
14     #
15     # if the patch runs silently or contains the word "FAILED" it means
16     # something's up
17     dry = `patch -s --no-backup --dry-run -p0 -R < "#{patchpath}"`.to_s.strip
18     if dry == ''
19         puts 'There\'s a patch that you should remove. Remove it then commit.'
20         puts "Patch File: #{patchpath}"
21         exit 1
22     elsif dry.index( 'FAILED' ) != nil
23         puts 'A broken patch was found. Fix it before proceeding.'
24         puts "Patch File: #{patchpath}"
25         exit 1
26     end
27 end
28 exit 0

This basically dry-runs the removal of the appropriate patch. If the patch would exit without complaint, that means that the patch could be applied and what you’re committing contains configuration, so the commit fails. If the output of the dry-run contains the word “FAILED”, that means that the patch is invalid, so we err on the side of caution and fail. If the patch is noisy, but doesn’t contain the word fail, we can safely assume that the patch has already been removed and we’re OK to continue exiting.

I’m sure that next week, there’ll be a totally new, genius way to solve this problem. For now, however, this is working really well.

For ease, I have the following git commands, also:

git-apatch

 1 #!/usr/bin/env ruby
 2 
 3 dir = File.expand_path( `git rev-parse --show-toplevel` )
 4 repo = dir.split('/').last.strip
 5 branch = `git branch | grep '^*'`.to_s.gsub(/^\*/ , '').strip
 6 patchname = "#{repo}.#{branch}.patch"
 7 patchdir = File.expand_path( '~/Patches' )
 8 patchpath = "#{patchdir}/#{patchname}"
 9 
10 if File.exists? patchpath
11     puts `patch -p0 <  "#{patchpath}"`
12 else
13     puts "No patch in #{patchpath}"
14 end

git-rmpatch

 1 #!/usr/bin/env ruby
 2 
 3 dir = File.expand_path( `git rev-parse --show-toplevel` )
 4 repo = dir.split('/').last.strip
 5 branch = `git branch | grep '^*'`.to_s.gsub(/^\*/ , '').strip
 6 patchname = "#{repo}.#{branch}.patch"
 7 patchdir = File.expand_path( '~/Patches' )
 8 patchpath = "#{patchdir}/#{patchname}"
 9 
10 if File.exists? patchpath
11     puts `patch -p0 -R < "#{patchpath}"`
12 else
13     puts "No patch in #{patchpath}"
14 end

All these do is look for a patch to be replied or moved for the current repo/branch and apply or remove it, failing noisily if the patch doesn’t exist. I realise they share some code that could be extracted, but for the sake of easy display, this is what the scripts basically contain.

The only thing I think that could make this more useful would be to detect which files the patch would modify, apply the patch and unstage those files to allow the commit to proceed, thus causing fewer interruptions. That’s a nice-to-have, though. Absent-minded as I am, this has already saved me from committing config twice today!

If you’ve got a different approach that works well for you, I’d love to hear about it on the twitters

Store git activity in MySQL with PHP

Git hooks are saving me so much time and providing me with interesting solutions to problems I didn’t even know I had. I can’t be the only person who this would be useful to, so give it a go.

As I said, I work on loads of sites, and keeping track of what’s been done and where is sometimes a bit of a pain. I keep a todo list, but if I get an emergency email from someone, chances are that won’t go through my todos. It will, however, be put into version control.

So this morning I had the bright idea to write a git hook that pushes relevant information to MySQL so that I can run activity reports later. All my bare git repositories are stored in a directory on our dedi, so it’s just a matter of making sure each repository has the post-receive hook in. I do this by keeping the actual hook in the same directory as all my repositories, then symlinking the hook into the appropriate place with the following little script. Obviously, this assumes that your post-receive hook is in the same place as your repositories, and that you want this hook everywhere. But that’s all true, so we’re all good. Once you’ve run the linked script, you’ll only have one hook to maintain and every time you create a new repository, you can just run the script again and everything will all be up-to-date.

Now for the hook. It’s not beautiful PHP, but little scripts like this rarely are, in my experience.

Create this table:

CREATE TABLE `log` (
`id` int(10) unsigned NOT NULL auto_increment,
`repo` varchar(255) NOT NULL,
`commit` varchar(40) NOT NULL,
`date` datetime NOT NULL,
`message` text NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `commit` (`commit`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;

Here’s your script. chmod +x it.

#!/usr/bin/env php
<?php
date_default_timezone_set('Europe/London');
exec('pwd',$pwd);
$repo = rtrim(array_shift($pwd),'/');
$repo = substr($repo,strrpos($repo,'/') + 1);
$db = new PDO('mysql:dbname=DB;host=127.0.0.1','USERNAME','PASSWORD');
exec('git log --all --pretty=format:"%H%n%ct%n%s%n%b%n<><><>"',$capture,$log);
if ($capture){
    // preprocess the log
    $commits = array();
    $current = array();
    foreach ($capture as $row){
        if (trim($row) === '<><><>') {
            $commits[] = $current;
            $current = array();
        } else {
            $current[] = $row;
        }
    }
    $v = array();
    $b = array();
    foreach ($commits as $commit){
        $sha = $commit[0];
        $m = $commit[2] . (trim($commit[3]) === '' ? '' : "\n\n" . implode("\n",array_slice($commit,3)));
        $d = date('Y-m-d H:i:s',$commit[1]);
        $v[] = '(?,?,?,?)';
        $b[] = $repo;
        $b[] = $sha;
        $b[] = $m;
        $b[] = $d;
    }
    $stmt = $db->prepare('insert ignore into log (repo,commit,message,`date`) values' . implode(',',$v));
    try {
        if ($stmt) {
            if (!$stmt->execute($b)) throw new PDOException;;
        } else Throw new PDOException;
    } catch (PDOException $e) {
        mail('EMAIL','Commit did not reach db',$e->getMessage());
    }
}
?>

So basically we’re extracting the log data we need, doing some funky stuff to handle multi-line commit messages (I like to store lots of details as my subject messages tend to be a bit vague!). Other than that, if you’re familiar with PHP, the above should be pretty self-explanatory. If it’s not, hit the comments and I’ll explain things.

I’ve only been using this a little while, but it seems to work very well. If you use it and stumble across any bugs, I’d love to know about them!

Update: I’ve today realised that git log only logs the currently-selected branch, or master on a bare repo so I’ve added the –all switch to git log so I can get the logs for every branch. Most of it’s just “Merged blah” but that means it can be filtered easily and I’d rather have everything and need to filter than be missing something important.

Managing multiple working copies with git

We have a CMS at Buffalo, which we have deployed to several servers for a few of our clients. Until recently, it was only two separate servers and was relatively easy to manage with Subversion (though slightly cumbersome – svn ci -m “blah”; ssh server; cd /site; svn up; exit; ssh server2; cd /site; svn up;) but now that we’re deploying the same CMS to 8 servers, simple changes and bug fixes are becoming a pain in the ass to deploy.

I’ve been tentatively looking into git for quite some time now. I’m incredibly cautious of the bleeding edge when it comes to how I make money because I prefer to have things that I can rely on and will serve me well than keep up with the latest fads. That’s not to say that I’m not interested in all the cool new jazz, and I keep up as much as family life permits, but I don’t dive in and adopt without learning and understanding everything I need first. Sensible? Yes. I’m surprised at how many people flaunt this ethos.

That being said, git has proved itself invaluable to me in the last 4 months. The way it encourages you to work is great. I do all changes on branches then merge them into master and remove branches to keep everything clean. I also have $deploy-staging and $deploy-live (where $deploy is a working copy) so that I can manage configuration for each working copy. This probably isn’t git best practice, but I’ve found it to be incredibly convenient. I work on up to 20 different sites in any week, so being able to merge changes and conflicts for live stuff locally saves me headaches galore. No-brainer. My git workflow goes something like this:

git checkout -b hotfix-phperror
# do some work
git commit -am "Hotfix for PHP Error fixed"
git checkout master
git merge hotfix-phperror
# resolve any conflicts
git checkout staging
git merge master
# test on staging - everything OK
git checkout live
git merge master

That might seem like quite a bit of work, but it keeps everything tidy. Unfortunately, it does mean that if things fail on staging, I still have the history of it in master. I suppose it would make more sense to merge the hotfix into staging, then merge that into live, then merge everything into master and tag once it’s verified working to keep it clean, but either way, this tip will work.

I’m under the impression that git frowns upon what I’m about to recommend, but it works so well that I can hardly ignore it! One proviso for this is that all the working copies you push to have to be the same. Lucky for us, the configuration for our CMS is held by the live site (separate repository) rather than the CMS itself, so each working copy is the same.

With the above in mind, I’m going to assume the following:

• You’ve got a central, bare repository to push to
• Your live branch (identical for all working copies) is tracked from said central repository

First off, you have to make sure that your local working copy has all remotes available. Obviously your bare repository will be available because that’s how you’re doing things anyway, but we also need working copies. You can check that everything’s there by running a quick git remote. If you’re all good, then we can start pushing code around.

As I said before, git doesn’t seem to like this sort of thing. If you were to try and push to one of your working copies where the checked out branch is the one you’re pushing to, git will whine at you and preserve the changes you push so that you can stash any local differences. You can force this by changing to the directory and running git reset --hard or git stash, but that defeats the purpose of this so we need a workaround.

Luckily, git gives you all the hooks you need and more for this. We’ll be creating a post-update hook that will force-update your push. With that in mind, cd to the root of your working copy on one of your remotes and do something like:

#!/bin/sh
cd ..
env -i git reset --hard

In .git/hooks/post-update and chmod it executable.

Now, when you push to the repository git will moan at you, but this hook will run and force all your changes. Awesome.

In the warning message git spits out, it threatens that new versions of git will auto-reject pushes to checked out branches, so a quick run of:

git config --set receive.denyCurrentBranch "ignore"

Will shut git up and have it doing what you tell it to. I suppose the reason it does this is it assumes that someone is working in that working copy and they don’t want you frivolously overwriting their code. Luckily, we know what we’re doing is safe so there’s no harm in doing this. Do remember, though, that all changes made in the working copy will be overwritten when you push using this method, so don’t work on live working copies. Don’t do that anyway, but definitely don’t do it here!

Now, you can do the following:

git remote add remote-alias ssh://root@blah/path/to/working/copy
git fetch
git push remote-alias live

In all the feedback, you’ll see something like “HEAD is now at 0d5431b HELLO!!!” (the hash and first line of your last commit message) which lets you know that things have worked.

Now you’re able to push to remotes without complaints, from the comfort of your local working copy, with a bit of scripting you can deploy your local live branch to all its remote locations with a little bit of scripting:

for remote in `git remote`; do git push $remote branchname; done;

For future reference, we’ll assume that this isn’t the only time you’ll do this, so save the following as a script in your path somewhere (I call it git-rpush):

for remote in `git remote`; do git push $remote $@; done;

Which you can call with git rpush live. Assuming that all your working copies have the config set and the hook installed, you can now push a change to all of your local repositories at once without having to remember which ones you actually need to push to, then spending half an hour SSH-ing all over the place to do it.

I don’t know about you, but that’s just saved me a crap-load of time. Hope it helps someone!

Coincidentally, if you do need this to be locally configurable, you can easily check out your branch and create a patch for configuration, then add the patch removal and application into hooks. I would do something like the following (assuming you already have your patch):

#!/bin/sh
cd ..
env -i git apply -R config.patch

In .git/hooks/pre-update and chmod executable

This will remove your config so that no conflicts occur during the update. All you need is to re-apply the patch in the post-update, after you’ve done the reset. Insert the following:

git apply config.patch

Your config will be preserved. You can even keep these patches in version control, just change the name for all your remotes and update your hooks accordingly. Then enter my competition on how easy you can make your life by scripting it!

Any and all questions and improvements are welcome and appreciated.

I would like to thank the following post for hand-holding through the intricacies of what I was trying to achieve with hooks. So thanks!

git or Versions.app with MacVim diff

I’ve seen on the Versions Google group, there’s a python script for integrating Versions with MacVim diff, but because Versions doesn’t appear to be completely contextual when it comes to executing the script, I was getting the GUI load with none of my .vimrc stuff, so it was no good really.

I tried to write a PHP script to do this, as that’s what I’m most familiar with, but I also hit this hurdle so it was time to dust off my (formerly almost nonexistant) bash knowledge to try and get this done properly. I believe I’ve come up with a fix.

When writing this, I came across two problems. One being that Versions doesn’t escape space characters in the arguments is passes to your script, so it’s not just a simple case of export HOME=”/your/home”; mvim -d $@; that gives you at least 3 files to play with, so I’ve /tried/ to be a bit clever and parse what I think is a file into an array and use it as arguments. It’s not terribly elegant, but it works and I’ve been using it a couple of days.

The only issue is that you’ll have to change the path to your home directory in the script, because I don’t know of a way to detect it. If there is one, please let me know!

Download the attached zip, chmod +x and put it in your ~/Library/Application Support/Versions/Compare Scripts/ folder, restart Versions, then you should see it as an option in your File Comparison dropdown.

To use this with git, I kept the file the same, and then git config –global diff.tool /Users/Me/Library/Application Support/Versions/Compare Scripts/mvimd.sh and it works great with that too. Go me.

Textmate + version control + (SSH or FTP) = happy Jasper

It’s no secret at all that I love Textmate. I find now that working without the ability to wrap a selection in brackets/quotes, the perfect indentation, amazing predefined bundles and the ability to modify/create your own bundles nearly impossible. I certainly don’t see the point in working without all this great stuff! One thing I find that is fundamentally missing is FTP/SSH support. There is a way to emulate FTP support, but it’s cumbersome and it just doesn’t integrate with my workflow at all, so I don’t use it.

Until today I’ve been switching to the very capable Coda, but I’m way too used to the power of Textmate’s text manipulation for this to be a long term solution. With no sign of FTP or SSH support on the horizon, I got my Google on. I’ve found a pretty great solution that will integrate very well with my workflow, so I thought I’d share.

First off, I downloaded the FTP/SSH bundle for Textmate from fuerstnet. This gives you reload and upload over FTP and SSH, along with 2 keyboard shortcuts, direct from a Textmate project. I then set up a folder in the following way:

./project
./project/TextmateProject.tmproj
./project/.ftpssh_settings

I wont reiterate the step-by-step for setting this up, as it’s on fuerstnet and it’s relatively easy.

Once you’ve got this file structure, open your favourite version control client (I use Versions or terminal with Subversion, but only because that’s what I’m used to. Git is also pretty awesome) and check out your project to the ./project folder so that the path to ./project now matches up with your path in .ftpssh_settings file.

At this point, it’s pretty obvious that this method relies on the fact that you’re working on 2 identical copies of 1 project – not ideal, but it works.

Once you’ve got your checked out files, go to your ./project folder and drag all the files in there (except TextmateProject.tmproj and .ftpssh_settings to your Textmate project drawer. Now save your project and it’s ready for you to use. To edit a file on your server, open it in your Textmate drawer, make your changes, save it and use the FTP/SSH bundle to upload it (2 keyboard shortcuts: cmd-S, option-S and your file’s uploaded). You’ll get a tooltip to tell you that the file has uploaded successfully (if it has), and a quick check in your browser should show you that the changes have taken place.

As I said, this relies pretty closely on you having a close affiliation with a version control system, but as long as your files in ./project mirror what’s on your server, it will be a relatively seamless process. I only say to use version control as it streamlines the whole thing, and makes it easier if you’re modifying the files in more than one place, or you have a team working on them. It’s nice and easy to set up a version repository on a local machine, and it’s great for managing projects so I’d recommend it anyway!

If I’ve omitted anything, let me know in comments, and please, props to fuerstnet for writing the awesome bundle that makes this process possible!