The villages are many and few between. Be happy that one is allowed to transfer to another one any time they wish. Of course, you might wanna travel with one of the Pyrate sisters, not one of the Soldiers or other captains...they might have their way.