It took me a while to work out how to do this, so I thought I should write it down for my own use and for anybody else who wants it.
I have my SSH daemon running on a non-standard port and wanted to push to my git repository.
The key is to add the repository, like so:
git remote add reponame ssh://username@server:port/repo/location/.git
Then pushing is as simple as:
git push reponame localbranch:remotebranch