The days of just scraping the internet are over. Data quality is a huge part of LLM training, I don’t think insta getting sloppified will save us unfortunately.
Yeah, but you need huge amounts of data to train, where to take it if not from the internet? I mean the existing books, records etc before the slop age are soon all fed as well.
From my understanding a lot of books are still getting digitized, so there is a constant trickle of old literature and news papers and such. But really the answer is synthetic data. Data made by AI and other means for the purpose of training. Which that is very different than just insta slop.
The digitisation is done at such verociuous speed that this will soon slow down to a trickle because anythimg left is hard to come by.
“Synthetic data” created with contribution from LLMs is poisining the models just as well. It is leading to reinforcement of hallucinations and artefacts.
I would reccomend you look into sythetic data. It is a real thing and is used intentionally in basically every LLM of the past year or more. Last I heard of an LLM without sythetic data was Dots LLM1 and that wasn’t competitive when it came out.
I did read up a bit on synthetic data. Sounds like it is mainly used to dilute human input, or rather multiply it artificially. It does appear it is tricky and can have downsides (especially when you remove the human content too much or entirely, but that of course also depends on the precise field of training). In protein engineering there is the saying, you get what you screen for, and that is primarily used as word of caution. I would assume the same is true for models, you get what you train for.
What they meant is that you could no longer just scrape the internet, but would need to filter the data for quality before mixing it into the slop substrate.
That is a bigger issue for LLMs than many realise. It means that the models are fed increasingly their own excrements during training.
The days of just scraping the internet are over. Data quality is a huge part of LLM training, I don’t think insta getting sloppified will save us unfortunately.
Yeah, but you need huge amounts of data to train, where to take it if not from the internet? I mean the existing books, records etc before the slop age are soon all fed as well.
From my understanding a lot of books are still getting digitized, so there is a constant trickle of old literature and news papers and such. But really the answer is synthetic data. Data made by AI and other means for the purpose of training. Which that is very different than just insta slop.
The digitisation is done at such verociuous speed that this will soon slow down to a trickle because anythimg left is hard to come by.
“Synthetic data” created with contribution from LLMs is poisining the models just as well. It is leading to reinforcement of hallucinations and artefacts.
I would reccomend you look into sythetic data. It is a real thing and is used intentionally in basically every LLM of the past year or more. Last I heard of an LLM without sythetic data was Dots LLM1 and that wasn’t competitive when it came out.
I did read up a bit on synthetic data. Sounds like it is mainly used to dilute human input, or rather multiply it artificially. It does appear it is tricky and can have downsides (especially when you remove the human content too much or entirely, but that of course also depends on the precise field of training). In protein engineering there is the saying, you get what you screen for, and that is primarily used as word of caution. I would assume the same is true for models, you get what you train for.
What they meant is that you could no longer just scrape the internet, but would need to filter the data for quality before mixing it into the slop substrate.