How about the outer fins on my board, an O'Brien Vision? I have been toying with removing them like I did with my old Evil Twin. Same advice, go without?
I ride my board finless (Absolute and Marius) and love it. It may not be for everyone, and really depends on the individual board. Best bet is take them off and give it a try.