Duplicate from 1Covenant/Covenant-72B

Co-authored-by: Joel Lidin <joellidin@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,37 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+*.webp filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -0,0 +1 @@
+CLAUDE.md

README.md

@@ -0,0 +1,88 @@
+---
+license: apache-2.0
+datasets:
+  - mlfoundations/dclm-baseline-1.0-parquet
+language:
+  - en
+pipeline_tag: text-generation
+---
+
+# Covenant-72B
+
+## Model Overview
+
+**Covenant-72B** is the largest permissionless collaboratively trained language
+model, trained entirely from scratch at the 72 billion parameter scale on 1.1
+trillion tokens of English text.
+
+![Covenant-72B](assets/covenant-72b.webp)
+
+For more details, see the [technical report](https://arxiv.org/abs/2603.08163).
+This is a base model. See [Covenant-72B-Chat](https://huggingface.co/1Covenant/Covenant-72B-Chat) for the instruction-tuned variant.
+
+**Covenant-72B** was trained with 20+ globally distributed participants
+coordinated via decentralized infrastructure on the Bittensor blockchain.
+Unlike prior collaborative training efforts that use whitelisted compute,
+Covenant-72B is the first to achieve this scale with fully permissionless
+participation. Training used the SparseLoCo communication-efficient optimizer
+to reduce bandwidth requirements across distributed nodes.
+
+## Usage
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model = AutoModelForCausalLM.from_pretrained(
+    "1Covenant/Covenant-72B",
+    torch_dtype=torch.bfloat16,
+    device_map="auto",
+)
+tokenizer = AutoTokenizer.from_pretrained("1Covenant/Covenant-72B")
+
+input_text = "The theory of general relativity"
+input_ids = tokenizer.encode(input_text, return_tensors="pt").to(model.device)
+output_ids = model.generate(input_ids, max_new_tokens=100)
+print(tokenizer.decode(output_ids[0], skip_special_tokens=True))
+```
+
+## Model Details
+
+- **Compute Participants**: 20+ independent contributors on Bittensor
+- **Minimum Compute per Participant**: 8×B200 or equivalent
+- **Model License**: Apache 2.0
+
+## Technical Specifications
+
+| Parameter                 | Value                          |
+| ------------------------- | ------------------------------ |
+| Parameter Size            | 72B                            |
+| Architecture              | LLaMA-style (LlamaForCausalLM) |
+| Number of Layers          | 80                             |
+| Number of Attention Heads | 64 (8 KV heads)                |
+| Hidden Size               | 8192                           |
+| Intermediate Size         | 28672                          |
+| Head Dimension            | 128                            |
+| Vocabulary Size           | 262,144                        |
+
+**Training Details**:
+
+- **Dataset**: [DCLM-baseline](https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0-parquet)
+- **Tokens**: 1.1 Trillion
+- **Optimizer**: SparseLoCo (communication-efficient optimizer)
+
+## Performance on Benchmarks
+
+_All results are 0-shot acc_norm (%) unless noted._
+
+| Model              | Size | Tokens | ARC-C | ARC-E |  PIQA |  OBQA | HellaSwag | WinoGrande\* | MMLU\* |
+| :----------------- | ---: | -----: | ----: | ----: | ----: | ----: | --------: | -----------: | -----: |
+| **Covenant-72B**   |  72B |   1.1T | 56.83 | 80.93 | 81.56 | 44.00 |     80.61 |        75.85 |  67.11 |
+| INTELLECT-1        |  10B |     1T | 44.80 | 71.76 | 77.37 | 43.80 |     70.26 |        63.30 |  32.69 |
+| Psyche Consilience |  40B |   1.2T | 31.14 | 55.77 | 76.12 | 35.20 |     63.67 |        56.99 |  24.23 |
+| LLM360 K2 ckpt_108 |  65B |   420B | 45.73 | 70.54 | 80.90 | 43.20 |     78.23 |        71.90 |  50.01 |
+| LLM360 K2          |  65B |   1.4T | 53.75 | 75.97 | 82.54 | 48.00 |     82.86 |        76.40 |  65.51 |
+| LLaMA-2-7B         |   7B |     2T | 45.05 | 73.82 | 78.73 | 44.20 |     76.18 |        69.38 |  41.73 |
+| LLaMA-2-70B        |  70B |     2T | 57.42 | 79.55 | 82.59 | 49.40 |     84.34 |        80.43 |  65.63 |
+
+_\*WinoGrande uses acc; MMLU uses acc._

config.json

@@ -0,0 +1,29 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 2,
+  "dtype": "float32",
+  "eos_token_id": 1,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 8192,
+  "initializer_range": 0.02,
+  "intermediate_size": 28672,
+  "max_position_embeddings": 2048,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 64,
+  "num_hidden_layers": 80,
+  "num_key_value_heads": 8,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 500000.0,
+  "tie_word_embeddings": false,
+  "transformers_version": "4.56.1",
+  "use_cache": false,
+  "vocab_size": 262144
+}

generation_config.json

@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 2,
+  "eos_token_id": 1,
+  "transformers_version": "4.56.1",
+  "use_cache": false
+}

model-00001-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aa9f6e12af1b635e03ef02966f08942c73c2b97b5df240cbeb260439ac25428f
+size 8589934728

model-00002-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fb441ffd16f3963cb7827e1c7d802584ffbaf6987a9c2c3751b5a2128fc1e188
+size 4966123104

model-00003-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:da56276a6d42a83d0b0f44ff4275a7d919e0d5270eec1ecf780eb657a9a8958a
+size 4362142864

model-00004-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:105e424414212194ac5c5c108c1644ec14f37423084d1792aa59177a87aad420
+size 4966188864

model-00005-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:585d7d94561385ce9cb8a5724f843d233e822b46a7b4607900f6a51699fa21a1
+size 4362142864

model-00006-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:224f9898b27dc91b022fb3efaf89ad39bdc7f239407ed903c8fc64d90105ebaa
+size 4362142864

model-00007-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a44acc88bd4ba4e851e57ef03fe81c4791a4d385e57db6cef7d0e76bc869c3ea
+size 4966188864

model-00008-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fda4219b386683907c766bbdc3c196805085f87af1917e163cdd7c41ae52c37a
+size 4362142864

model-00009-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:25032c9f1bb4ad0b1ce0f8faab2af3b57747b255f61ed49483b16ffca2207574
+size 4362142880

model-00010-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d84adc6102eae170e09239b2d945100a58005831ff09ea9eb51494dd741f1169
+size 4966188880

model-00011-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cf0ca167e89a8ff0f4ec59c1d8b4e53614457f04ef6771890d0ff7b911d6ebfd
+size 4362142872

model-00012-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a29d3501e539a11021d04a0f8f086ae98c3a134223dd2f4fe35d814a3c1fc0c1
+size 4362142872

model-00013-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3d5cec4e7d33fe4db334d8ba47473b2f1c1dc8855256d8e37920b5f24b7eb03d
+size 4966188880

model-00014-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5d48f0d12770f4d1b3092afcd79189efa9cb88040f4b8d9dbebdfa29431a9ce5
+size 4362142872

model-00015-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:baf8d4893c6d5659b44a95d2aff69393108cbe163488ff4763f763b3e5961b88
+size 4362142872

model-00016-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7fae76e609358ae93019f53a28e2bdee5489bed8d85d395bc29801cb7a6fa834
+size 4966188880

model-00017-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:29d7d20d0cef648cc987b8f4cc3cc91b2b6391cf12d526b0f8b26232caf7a9e4
+size 4362142872

model-00018-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3c92af3d62ef52703548bffbc61fafc5b57db7ec341fde37e34f50f9711a2389
+size 4362142872

model-00019-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:58a54d8f2d3b56b58901df2bdc5a1ae03c78ed3d148fb03e8f292f41af2c7003
+size 4966188880

model-00020-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e448c576b8f042d11007fb1688ddaa873b59a08313f1f7a7b8c6ed567499c236
+size 4362142872

model-00021-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eface0a19ee678adafbea936be52bd5fb9f377510b39efec86f0a296a03b9986
+size 4362142872

model-00022-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:18a2da5b94ea3004824b825d6c900a81d13988648c727d79d3ec3afafb4ee706
+size 4966188880

model-00023-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1c1a6a18a79c4f1d5cf44ca3366ed79e3114e35683df0cd87611db4b4f3884b4
+size 4362142872

model-00024-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e1f4327da61b7c711f446bf56b967cbe15367b8e1d571eb333c97fb41baeb35e
+size 4362142872

model-00025-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b74d03ae0fc67c6b62545a89201b7cb7c0966d75a8340809652ef97dc8f07ab2
+size 4966188880

model-00026-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:db5122fae06d665815c2df89ef08db9770e8e438cdc380636b7819cde20621e6
+size 4362142872

model-00027-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4865e4e6469b698d5219e18d9721e2ea698753d1239a91eab457705738755795
+size 4362142872

model-00028-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1c7c1f23ed05ef207636c2b99cabaccec05cd10e1ebb53a457008b7a8bc77910
+size 4966188880

model-00029-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9ba0f5cea8025fb1dc00af9ef4c583fbf76a2fdff6d7b790c207dad1da3702fd
+size 4362142872

model-00030-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:55ff419b9642a21389d7c1b37c927ffebc2751858a1c965e407d61852372cc4d
+size 4362142872

model-00031-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c36a45c7d0865e9b5a37d53d1f04c44cef94ac9145636664e1d0e6dcd7ea8aeb
+size 4966188880

model-00032-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6a4616150b873fd78b4ee65bbea092c170f798b3f5efa80322d5f427b2b1315f
+size 4362142872

model-00033-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1e2787726d0408d8e7c129e22e256fbeb8f7f146387f7a8c3421362e07dadf4d
+size 4362142872

model-00034-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:134471e7b7adb061daba22503d605a9ec1d6cfa5cece5f43288ed66b62510965
+size 4966188880

model-00035-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:febb3b8404d53800c63742c2500f33559289f57398873e5c42b1efa20b72dc62
+size 4362142872

model-00036-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:661e6890ec962f3e9d4bf24d98f5edabaa655e86266ede97b86d77bef45a024a
+size 4362142872

model-00037-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:95c94b4b7909b317ca4f7c0852a0d260dda24f0b2f4b1e3b268142e81e8b7962
+size 4966188880

model-00038-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ef33921fa6f7f49600a8e868afddebd038ede1b1d147164683c7416416a23a20
+size 4362142872

model-00039-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:42ff0e3b7d21697a0cfda3439b1074b4e504d652bb14a1eefa3c5637552f6776
+size 4362142872

model-00040-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eccf5a7aece984a5ef820dc2b43a9a79e7a2154999f47ffc6d5b6c57ad0b6316
+size 4966188880

model-00041-of-00062.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f906a7a51ec857e727eecc3c026d63c690a826d4cba651796b44ee2348a4d846
+size 4362142872