Версия, использующая rename, которая работает для вашего измененного формата, будет
rename -- 's/(\.in|\.out)\.(\d+)/$2$1/' *
, например
$ ls
txt.in.1 txt.in.2 txt.in.3 txt.out.1 txt.out.2 txt.out.3
$
$ rename -nv -- 's/(\.in|\.out)\.(\d+)/$2$1/' *
txt.in.1 renamed as txt1.in
txt.in.2 renamed as txt2.in
txt.in.3 renamed as txt3.in
txt.out.1 renamed as txt1.out
txt.out.2 renamed as txt2.out
txt.out.3 renamed as txt3.out