Hmmm... I hope someone will put me right if my physics is out...
Well, assuming you try uke strings, a short-scale tenor banjo has a 19"-20" scale which is similar to a baritone uke. The thing is that there's quite an after-length behind the bridge on banjo strings, making them longer than uke strings of the same scale length. The longer strings could mean that the tension would be higher than on a uke at the same pitch (as ukes have no real string after-length behind the saddle).
Actually, Using nylon uke strings might be difficult if your tailpiece is designed for standard loop-end metal banjo strings. Maybe you'll be better of sticking to metal strings.
You know, the more I think about it, the more I simply just don't know. I guess you could just experiment with different gauges of loop-end banjo or mandolin strings. Apart from the head most parts of a banjo are more robust than those of a uke. If I still had a tenor banjo I wouldn't be afraid to just experiment - but I'd still go carefully and not overload the tension to a drastic degree.
If you do, don't forget to keep an eye on the stress on the head. You don't want to split it.
You could try asking over at
http://www.banjohangout.org/forum/ or at the trad string instrument section of
Chiff & Fipple.
Hope you find your answer.
Let us know if you figure it out? Could be useful to other people too.