Note Amazing dataset!!! 7 Common Crawl releases filtered and annotated for CC licensing. (only some 100M unique rows) Please view similar projects (smaller/less granular): https://huggingface.co/datasets/common-pile/dolma-cccc (259M rows, eng only) https://data.commoncrawl.org/contrib/c4corpus/CC-MAIN-2016-07/index.html (dated multilingual)

Note pretraining • multilingual • cultural heritage • volunteer ★★★★★ (5/5) Largest public domain corpus; legally safe; Traceable provenance; diverse sources. OpenCulture, Gov Docs, Wikisource, Gutenberg "The largest open AI training data set consisting entirely of public-domain texts." CC0 1.0 Universal

Note pretraining • multilingual • public domain • internal ★★★★★ (5/5) Massive scale; strictly permissive licenses; Addresses IP concerns with clear provenance. GitHub, ArXiv, StackExchange, Public Domain "Exclusively collecting content in the public domain or under permissive open licenses." Apache License 2.0

Note "76 OpenStax textbooks." Creative Commons Attribution 4.0

Note "...7,040,984 current pages from all available Fandom.com community wiki dumps as of February 18, 2025." DATALOADER WARNING: ZST format requires special dataloader. Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA 3.0)

Note "...transcripts of openly available videos from Khan Academy." IMPORTANT: Despite the "MIT License" listed on the dataset card, Khan Academy resources are NOT MIT and are almost exclusively published under Creative Commons Attribution-NonCommercial-ShareAlike (CC BY-NC-SA).

Note For pre-training.

Note instruction • multilingual • question-answering • volunteer ★★★★★ (5/5) Massive language coverage; high fluency; Culturally nuanced, community-generated. Aya Project (Human Annotations) "A human-annotated multilingual instruction finetuning dataset...generated by a global community of volunteers." Apache License 2.0

Note math • reasoning • instruction-following • internal ★★★★★ (5/5) High-quality math problems; diverse difficulty; Standard for measuring reasoning capabilities. AMC 10, AMC 12, and AIME competitions "A dataset of 12,500 problems from mathematics competitions...to measure mathematical problem solving." MIT License

Note instruction • meta-dataset • multilingual • volunteer ★★★★★ (5/5) Unprecedented task diversity; vetted by community; Excellent for testing zero-shot generalization. GitHub NLP Community (Super-NaturalInstructions) "We collect a large variety of NLP tasks... instructions were written by contributors... and we ensure high quality through a peer-review process." Apache License 2.0

Note instruction • meta-dataset • prompt-engineering • internal ★★★★★ (5/5) Massive scale across hundreds of datasets; Human-written templates for prompt engineering. Existing Public NLP Datasets "A collection of prompted English datasets... the prompts are defined in a structured way using Jinja2 templates... created by 36 researchers." Apache License 2.0

Note math • reasoning • chain-of-thought • internal ★★★★★ (5/5) High-quality, multi-step reasoning problems; The gold standard for grade-school math CoT. Human Problem Writers (OpenAI) "We collected a dataset of 8.5K high quality linguistically diverse grade school math word problems... created by human problem writers." MIT License

Note instruction • dialogue • curated • internal ★★★★★ (5/5) Extremely high-quality, curated examples; Proved the 'quality over quantity' theory for SFT. Stack Exchange, wikiHow, Reddit (Curated) "We curate 1,000 prompts and responses... manually written or selected from community forums." CC-BY-NC-SA 4.0

Note instruction • generalist • question-answering • internal ★★★★☆ (4/5) Guaranteed human-written; clean formatting; Commercially usable; good for RAG tuning. Wikipedia "Generated by Databricks employees... contains 15,000 high-quality human-generated prompt/response pairs." CC-BY-SA 3.0

Note Perhaps the best human-written question-answering dataset I have ever found. Both subsets include real questions from Bing users, answered by crowdworkers, with source grounding! wellFormedAnswers refers to answers that went through multiple rounds of editing for perfection. v1.1 ~ 100K rows v2.1 ~ 1M rows (OMG!) IMPORTANT: For research and non-commercial use ONLY.

Note dialogue • instruction-following • volunteer ★★★★☆ (4/5) Complex conversation trees; highly diverse; Strong for SFT and reward model training. OpenAssistant (Crowd-sourced) "A human-generated, human-annotated assistant-style conversation corpus... [via] a worldwide crowd-sourcing effort." Apache License 2.0

Note dialogue • instruction-following • volunteer ★★★★☆ (4/5) Improved quality control over v1; heavily filtered; Better toxicity ranking; still quality variance. OpenAssistant (Crowd-sourced) "Inherits the same data collection methodology... completely human-generated by volunteers." Apache License 2.0

Note reasoning • physical commonsense • volunteer ★★★★☆ (4/5) Focused on physical interaction; binary choice; Good for validation and RL/GRPO training. Instructables.com "We employ Amazon Mechanical Turk to crowd-source data... annotators are presented with a goal..." Apache License 2.0

Note science • multimodal • reasoning • internal ★★★★☆ (4/5) Includes images and chain-of-thought explanations; High educational value; curriculum-sourced. IXL Learning (K-12 Curricula) "Collected from elementary and high school science curricula... annotated with lectures and explanations." CC-BY-NC-SA 4.0

Note instruction • generalist • question-answering • internal ★★★★☆ (4/5) High-quality professional writing; distinct voices; Very clean structure for supervised fine-tuning (SFT). Skilled Human Annotators "A high-quality dataset of 10,000 instructions and demonstrations created by skilled human annotators." CC-BY-NC 4.0

Note qa • long-context • reading-comprehension • internal ★★★★☆ (4/5) Very long context (avg. 5k tokens); hard distractors; Written by vetted writers, not crowd-workers. Project Gutenberg, Creative Commons "We hired contributors to read articles and write questions... writers were vetted for high reading comprehension." Creative Commons Attribution 4.0

Note qa • scientific • evidence-selection • internal ★★★★☆ (4/5) Expert-written questions on NLP papers; Requires full-text reasoning and evidence. S2ORC Corpus (NLP Papers) "We solicited NLP practitioners... to pose questions... and provide answers from the full text of the paper." Creative Commons Attribution 4.0

Note finance • qa • tabular • internal ★★★★☆ (4/5) Hybrid tabular/textual reasoning; expert annotated; Sourced from real corporate financial reports. FinQA, TAT-QA, and TAT-DQA "Generated from FinQA, TAT-QA... [which are] expert-annotated datasets on financial reports." Creative Commons Attribution 4.0

Note reasoning • reading-comprehension • volunteer ★★★★☆ (4/5) Situation-based reasoning from text; Requires connecting information from multiple parts. Science Textbooks, Wikipedia "Crowd workers... write questions that involve reasoning based on a provided background passage." Creative Commons Attribution 4.0

Note medical • qa • multiple-choice • internal ★★★★☆ (4/5) Real medical entrance exam questions; subject diversity; High accuracy requirements for a specialized domain. AIIMS & NEET (Official Exams) "Collected from real-world medical entrance exams in India." Apache License 2.0

Note medical • qa • covid-19 • volunteer ★★★★☆ (4/5) Expert-annotated by biomedical professionals; Timely and trustworthy scientific domain. CORD-19 (Scientific Articles) "Annotated by a team of 15 volunteer biomedical experts." MIT License

Note math • qa • proof-generation • volunteer ★★★★☆ (4/5) Massive scale; real user questions with LaTeX; Curated subsets for specific math domains. Stack Exchange (Math, Physics) "Q&A pairs from the Stack Exchange network... under CC-BY-SA." CC-BY-SA 4.0

Note summarization • news • instruction-following • internal ★★★★☆ (4/5) Professionally written one-sentence summaries; High-quality, abstractive task from a trusted source. BBC News "We create a dataset... by harvesting online articles from the British Broadcasting Corporation (BBC)... summaries are written by professional editors." Non-Commercial License

Note safety • truthfulness • qa • internal ★★★★☆ (4/5) Crucial for safety and alignment research; Adversarial questions test model honesty. Researcher-Authored "Questions and answers... written by the authors... designed to measure whether models generate answers that are true." Apache License 2.0

Note qa • long-form • explanatory • volunteer ★★★★☆ (4/5) Natural, in-depth explanations on complex topics; Excellent for training assistant-like answer styles. Reddit (r/explainlikeimfive) "Sourced from the Reddit community 'Explain Like I'm 5' (ELI5)... questions are written by one user, and answers are provided by other users." CC-BY-SA 3.0

Note summarization • professional • long-context • internal ★★★★☆ (4/5) Expert-written technical content from government; Ideal for long-context understanding and summarization. U.S. Government Reports "The reports are written by government employees and the summaries are written by the report authors." Public Domain

Note meta-dataset • long-context • reasoning • internal ★★★★☆ (4/5) A benchmark of high-quality, long-context tasks; Aggregates multiple human-written datasets. GovReport, Qasper, and others "A suite of natural language datasets that require reasoning over long contexts... The constituent datasets are all built on natural, human-written text." Apache License 2.0

Note summarization • scientific • expert-written • internal ★★★★☆ (4/5) High-quality, expert-level scientific summaries; 'TLDR' style trains for concise, dense output. Scientific Papers (S2ORC) "SciTLDR contains both author-written and expert-derived TLDRs... collected using a novel annotation protocol." Apache License 2.0

Note dialogue • knowledge-grounded • volunteer ★★★½☆ (3.5/5) Engaging, human-like conversations; Explicitly grounded in provided knowledge. Wikipedia "We crowdsourced a dataset... on Amazon Mechanical Turk... pairing two crowd-workers in a dialogue." CC-BY-NC 4.0

Note reasoning • exams • meta-dataset • internal ★★★½☆ (3.5/5) Rigorous human benchmarks; diverse subjects; High difficulty; sourced from official exams. Gaokao, SAT, LSAT "Derived from standard admission and qualification exams intended for human test-takers." MIT License

Note code • python • instruction-following • internal ★★★☆☆ (3/5) Good for entry-level code synthesis; verified execution; Clean, high-quality, manually created problems. Internal Crowdsourcing "Crowdsourced from a pool of programmers... problems designed to be solvable by entry-level programmers." Creative Commons Attribution 4.0

Note qa • retrieval • volunteer ★★★☆☆ (3/5) Captures real user search intent; high topic diversity; Based on actual queries sent to Google search. Google Search, Wikipedia "Consisting of real Google search queries... and annotations... by human crowdworkers." CC-BY-SA 3.0

Note qa • reading-comprehension • volunteer ★★★☆☆ (3/5) Includes unanswerable questions to test robustness; Standard benchmark for extractive QA. Wikipedia "Crowdworkers pose questions... on a set of Wikipedia articles." CC-BY-SA 4.0

Note qa • multi-hop-reasoning • synthetic ★★★☆☆ (3/5) Requires synthesizing multiple documents; hard tasks; Includes supporting facts for explainability. Wikipedia "Crowd workers provide distinct supporting facts... and questions are generated by crowd workers." CC-BY-SA 4.0

Note qa • boolean • user-generated ★★★☆☆ (3/5) Captures natural query ambiguity from real users; Questions generated in unprompted settings. Google Search, Wikipedia "Naturally occurring yes/no questions... generated in unprompted and unconstrained settings [by users]." CC-BY-SA 3.0

Note math • reasoning • word-problems • internal ★★★☆☆ (3/5) Includes step-by-step rationales for answers; Sourced from algebra word problems. Math Competitions "Algebra word problems... from various math competitions." MIT License

Note dialogue • qa • conversational • volunteer ★★★☆☆ (3/5) Conversational history with abstractive answers; Pairs of annotators role-playing as user and system. Wikipedia, CNN, Gutenberg "We pair two annotators... one plays the role of a user and the other plays the role of a system." CC-BY-SA 4.0

Note dialogue • task-oriented • synthetic/volunteer ★★★☆☆ (3/5) Robust state tracking annotations; multi-domain; Deterministic simulator + human paraphrasing. Dialog Simulator "Dialogue simulator... using a finite set of actions... followed by a crowdsourcing step to paraphrase." CC-BY-SA 4.0

Note qa • reading-comprehension • volunteer ★★★☆☆ (3/5) Dual versions of plots test comprehension; Requires deep reasoning and synthesis by workers. Wikipedia, IMDb "Amazon Mechanical Turk Workers... were given freedom to pick answer from the plots or synthesize their own." Apache License 2.0

Note qa • dialogue • expert-comparison • volunteer ★★★☆☆ (3/5) Contains high-quality, expert-verified human answers; Useful 'human_answers' subset for SFT. Wikipedia, Finance/Medical Experts "We collected tens of thousands of comparison responses from human experts... ranging from open-domain, financial, medical, legal, and psychological areas." CC-BY-SA 4.0

Note reasoning • critique • scientific • expert-written ★★★☆☆ (3/5) Trains critical reasoning via scientific peer reviews; Expert-level discourse for feedback generation. CS Conference Peer Reviews "A dataset of scientific peer reviews... written by experts for a subset of papers in top-tier venues." Apache License 2.0

Note qa • commonsense • multiple-choice • volunteer ★★½☆☆ (2.5/5) Requires external elementary-level knowledge; Crowd-workers generated questions from facts. Science Facts "Questions... generated by crowd-workers who were provided with a few facts." Apache License 2.0

Note dialogue • qa • conversational • volunteer ★★½☆☆ (2.5/5) Teacher-student dialogue dynamic; Focuses on information-seeking conversations. Wikipedia "Two crowd workers... interact through a dialogue interface to ask and answer questions." CC-BY-SA 4.0

Note math • code • volunteer ★★☆☆☆ (2.0/5) Annotated with operational programs for reasoning; Based on human-written GRE/GMAT level problems. GRE, GMAT (AQuA-RAT) "Gathered by using a new representation language to annotate over the AQuA-RAT dataset." Apache License 2.0

Note dialogue • task-oriented • multi-domain • internal ★★★☆☆ (3/5) Large-scale human-written task-oriented dialogues; Covers multiple domains for real-world assistants. Cambridge Dialogue Systems Group et al. (MultiWOZ) "A multi-domain Wizard-of-Oz dataset of human-human dialogues for task-oriented dialogue modelling." Apache License 2.0

Note instruction • Chinese • generalist • internal ★★★☆☆ (3/5) Large-scale Chinese instruction dataset; Combines multiple open corpora and task types. Beijing Academy of Artificial Intelligence (COIG) "The COIG dataset... a Chinese open instruction generalist corpus built from multiple publicly available sources." Apache License 2.0 (core dataset; includes data under other permissive licenses)

Note dialogue • social • blended-skills • internal ★★★☆☆ (3/5) Human-human open-domain conversations; Blends chitchat, empathy, and knowledge-grounded skills. Blended Skill Talk (Facebook AI) "A dataset of 7k conversations explicitly designed to exhibit multiple conversation modes: personality, empathy, and knowledge." License: see original release (HF lists license as unknown)

Note CC BY-NC-SA (Attribution-ShareAlike, NonCommercial)

Note Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License

Note MIT License

Note amazing quality but non-commercial and only 250 rows :/ Creative Commons Attribution Non Commercial 4.0

Note Apache 2.0 License

Note general conversation; not LLM-like at all.

Note MIT License

Note MIT License

Note Mix of code under various permissive licenses (MIT, Apache, BSD). Attribution required, but v1.2 subset is filtered for no ShareAlike/Copyleft. Deduped subset: bigcode/the-stack-dedup https://r.jordan.im/download/language-models/kocetkov2022.pdf

Note IMPORTANT: Only subset=human_generated_evals is human-written.

Note Multimodal.

Note IMPORTANT: Only rows with dataset=ELI5 and a_id≠[model name] are human-written. Better than original ELI5 data due to advanced filtering with metascoring.

Note Tiny eval.

Note Tiny eval (multimodal).

Note qa • multilingual • extractive-qa • internal ★★★★☆ (4/5) Typologically diverse languages; native-speaker intent; Questions written without seeing the answer. Wikipedia "Questions... written by a human who wants to know the answer but does not know it yet." Apache License 2.0

Note Only <500 rows are written by humans and not generated by propertiary models.

Note Very small eval.

Note Fine quality 100% human ethical questions dataset for RL/GRPO. Small size, MIT license.

Note Creative Commons Attribution 4.0

Note All StackExchange subdomains in one dataset, as of 2025. Cool!

Note Apache License 2.0

Note Apache License 2.0

Note Human-annotated CoT for science QA. Subsets: v1, v2, v3 GH Repo: https://github.com/allenai/entailment_bank/ GDrive DL mirror: https://allenai.org/data/entailmentbank Apache 2.0 License.

Note full: https://dl.fbaipublicfiles.com/paq/v1/PAQ.tar.gz SUBSETS: https://dl.fbaipublicfiles.com/paq/v1/PAQ_L1.tar.gz https://dl.fbaipublicfiles.com/paq/v1/PAQ_L4.tar.gz https://dl.fbaipublicfiles.com/paq/v1/PAQ_NE1.tar.gz more DLs: https://github.com/facebookresearch/PAQ

Note Check out the GitHub for the full data. https://github.com/ricsinaruto/gutenberg-dialog