to answer my own question: yes, vectorz makes quite a difference. It's also substantially faster then clatrix.

Interestingly, going from openjdk's jvm to oracle did make a substantial difference for the matrix versions, but nearly not at all for my handcrafted vector manipulations.