Basically the title.
For me, a dense model around 20b, with an additional engram table of 10-20b, or a sparse MoE around 80b would be amazing.
Basically the title.
For me, a dense model around 20b, with an additional engram table of 10-20b, or a sparse MoE around 80b would be amazing.
Flash next 125b is genuinely amazing, I hope to see 4.0 of the same size.
Agree, and upvoted.
Another good size would be 160 gb for 96 gb vram systems. After that I would like to see 225 to 250 gb.