Basically the title.
For me, a dense model around 20b, with an additional engram table of 10-20b, or a sparse MoE around 80b would be amazing.
Basically the title.
For me, a dense model around 20b, with an additional engram table of 10-20b, or a sparse MoE around 80b would be amazing.
I hope an instruction tuned will be released as well.