So RWKV 7b v5 is 60% trained now, saw that multilingual parts are better than mistral now, and the english capabilities are close to mistral, except for hellaswag and arc, where its a little behind. all the benchmarks are on rwkv discor, and you can google the pro/cons of rwkv, though most of them are v4.
Thoughts?
Hmm, will have to check this stuff with the people on the rwkv discord server.
V5 is stable at context usage, and V6 is trying to get better at using the context, so we might see improvement on this