Currently running osx 10.8.2
Have an external network drive attached to my rt-n66u router running tomato
I'm trying to use tmutil to connect my external network drive, and am having a lot of trouble.
I mount the drives in Finder
Run sudo tmutil setdestination /Volumes/external, and get Incompatible file system type: smbfs (error 45)
Any ideas? :(
I can't answer your question specifically aside from telling you that timemachine backups on samba shares are not supported officially. As a consequence, even if you will be able to backup your data on a samba share (as far as I know lots of people have issues to do so), you will run into trouble during a restore procedure. If you would try to restore your data by booting an OS X livecd you couldn't because it doesn't have a samba client installed in order to connect to a samba share. The next step would be to try to connect the backup drive directly to your Mac to access your data. Chances are this would also fail because the harddrive was formatted with a linux filesystem which is not supported by OS X.
So the best thing you could do is to install netatalk (afpd) on that router box if it is possible at all.