Other labeled notes, same rules
Beside the two Federal Supreme Court sets, a second pack is being gathered now. Every row is a short note plus a closed label that the source already published. The pack is not finished. Consumer complaint narratives are still being scanned, and WANLI is next.
Train and test, capped per source:
- BoolQ, SST-5, TREC, Banking77, IMDB, ContractNLI, CLINC, Aegis, GoEmotions, AG News
- MultiNLI, SNLI, VitaminC
- Amazon reviews, Yelp reviews, DBpedia, Yahoo Answers
Held out of training, so they only score later:
- Emotion, TweetEval offensive, QNLI, PAWS, SciQ
Kept in a separate non-commercial file:
- ANLI, SciFact
A row stays only when the source's own human label still matches the text that is stored.
- The label is the dataset's label. Teacher labels and model-written labels are dropped. Aegis keeps human prompt labels and drops redacted prompts and jury labels. VitaminC keeps real revisions only. GoEmotions keeps a comment only when it has exactly one emotion.
- For a claim or a question paired with a passage, both sides have to fit in full. If the passage would have to be cut, the row is dropped, so the label still refers to the whole source text.
- SciQ stays only when the correct answer is written in the support. The correct option is rotated, so it is not always the first choice.
- Empty labels, unlabeled test splits, and IMDB's unlabeled reviews are dropped.
- Exact copies are removed. The official eval split is preferred over the train split.
- One source can contribute at most 40,000 train rows and 5,000 test rows, stratified by label. Yahoo, DBpedia, and Yelp cannot crowd out the smaller sets.
- The note has to fit the reader window. Non-commercial licenses stay out of the commercial train file.
- Complaint text does not include the product name. The product is the label. Yahoo keeps the question, not the best answer.
Left out on purpose: FEVER and HoVer, because the hub file does not contain the evidence sentence. MMLU, because there is no supporting note. Open-Jev, because those rows are marked synthetic. Kev's packed suites, because they repeat the same documents. Generation sets such as commit messages, because they have no closed label.
That is the curation. A kept row is a decision the source already made, on text that is still complete, with no one source allowed to dominate the file.