Instructions to use felixwangg/Qwen2.5-Coder-7B-func-stage1_ep2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use felixwangg/Qwen2.5-Coder-7B-func-stage1_ep2 with PEFT:

from peft import PeftModel
from transformers import AutoModelForCausalLM

base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-7B-Instruct")
model = PeftModel.from_pretrained(base_model, "felixwangg/Qwen2.5-Coder-7B-func-stage1_ep2")

Transformers

How to use felixwangg/Qwen2.5-Coder-7B-func-stage1_ep2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="felixwangg/Qwen2.5-Coder-7B-func-stage1_ep2")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("felixwangg/Qwen2.5-Coder-7B-func-stage1_ep2")
model = AutoModelForCausalLM.from_pretrained("felixwangg/Qwen2.5-Coder-7B-func-stage1_ep2")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use felixwangg/Qwen2.5-Coder-7B-func-stage1_ep2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "felixwangg/Qwen2.5-Coder-7B-func-stage1_ep2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "felixwangg/Qwen2.5-Coder-7B-func-stage1_ep2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/felixwangg/Qwen2.5-Coder-7B-func-stage1_ep2

SGLang

How to use felixwangg/Qwen2.5-Coder-7B-func-stage1_ep2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "felixwangg/Qwen2.5-Coder-7B-func-stage1_ep2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "felixwangg/Qwen2.5-Coder-7B-func-stage1_ep2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "felixwangg/Qwen2.5-Coder-7B-func-stage1_ep2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "felixwangg/Qwen2.5-Coder-7B-func-stage1_ep2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use felixwangg/Qwen2.5-Coder-7B-func-stage1_ep2 with Docker Model Runner:
```
docker model run hf.co/felixwangg/Qwen2.5-Coder-7B-func-stage1_ep2
```

Qwen2.5-Coder-7B-func-stage1_ep2 / debug.log

felixwangg

Upload folder using huggingface_hub

b548143 verified about 1 month ago

raw

history blame contribute delete

86.4 kB

	[2026-04-21 01:38:41,824] [DEBUG] [axolotl.utils.config.log_gpu_memory_usage:127] [PID:4094106] baseline 0.000GB ()
	[2026-04-21 01:38:41,824] [INFO] [axolotl.cli.config.load_cfg:341] [PID:4094106] config:
	{
	"activation_offloading": false,
	"adapter": "lora",
	"axolotl_config_path": "./axolotl_configs/Qwen2.5-coder-7b-instruct/func_stage1_sec_stage2/lora-func_stage_1.yaml",
	"base_model": "Qwen/Qwen2.5-Coder-7B-Instruct",
	"base_model_config": "Qwen/Qwen2.5-Coder-7B-Instruct",
	"batch_size": 64,
	"bf16": true,
	"capabilities": {
	"bf16": true,
	"compute_capability": "sm_90",
	"fp8": true,
	"n_gpu": 2,
	"n_node": 1,
	"tf32": true
	},
	"context_parallel_size": 1,
	"dataloader_num_workers": 2,
	"dataloader_pin_memory": true,
	"dataloader_prefetch_factor": 256,
	"dataset_num_proc": 112,
	"dataset_prepared_path": "/home/tkwang/scratch/SecSteer-v2/axolotl-datasets/lora/Qwen2.5-Coder-7B/func-stage1",
	"datasets": [
	{
	"chat_template": "tokenizer_default",
	"message_property_mappings": {
	"content": "content",
	"role": "role"
	},
	"path": "felixwangg/codenet-c-cpp-stage1",
	"split": "train",
	"trust_remote_code": false,
	"type": "chat_template"
	}
	],
	"ddp": true,
	"device": "cuda:0",
	"device_map": {
	"": 0
	},
	"dion_rank_fraction": 1.0,
	"dion_rank_multiple_of": 1,
	"eaft_alpha": 1.0,
	"eaft_k": 20,
	"early_stopping_patience": 1000,
	"env_capabilities": {
	"torch_version": "2.11.0"
	},
	"eval_batch_size": 4,
	"eval_causal_lm_metrics": [
	"sacrebleu",
	"comet",
	"ter",
	"chrf"
	],
	"eval_max_new_tokens": 128,
	"eval_sample_packing": false,
	"eval_steps": 10,
	"eval_table_size": 0,
	"experimental_skip_move_to_device": true,
	"flash_attention": true,
	"fp16": false,
	"generate_samples": false,
	"generation_do_sample": true,
	"generation_max_new_tokens": 50,
	"generation_prompt_ratio": 0.5,
	"generation_temperature": 0.7,
	"gradient_accumulation_steps": 8,
	"gradient_checkpointing": true,
	"gradient_checkpointing_kwargs": {
	"use_reentrant": true
	},
	"include_tkps": true,
	"is_falcon_derived_model": false,
	"is_llama_derived_model": false,
	"is_mistral_derived_model": false,
	"layer_offloading": false,
	"learning_rate": 4e-05,
	"lisa_layers_attribute": "model.layers",
	"load_best_model_at_end": true,
	"load_in_4bit": false,
	"load_in_8bit": false,
	"local_rank": 0,
	"logging_steps": 1,
	"lora_alpha": 16,
	"lora_dropout": 0.05,
	"lora_embedding_kernel": true,
	"lora_mlp_kernel": true,
	"lora_o_kernel": true,
	"lora_qkv_kernel": true,
	"lora_r": 16,
	"lora_target_linear": true,
	"loraplus_lr_embedding": 1e-06,
	"lr_scheduler": "cosine",
	"mean_resizing_embeddings": false,
	"merge_lora": true,
	"merge_method": "memory_efficient",
	"micro_batch_size": 4,
	"model_config_type": "qwen2",
	"num_epochs": 2.0,
	"num_generation_samples": 3,
	"optimizer": "adamw_torch",
	"otel_metrics_host": "localhost",
	"otel_metrics_port": 8000,
	"output_dir": "/home/tkwang/scratch/SecSteer-v2/axolotl-outputs/lora/Qwen2.5-Coder-7B-func-stage1_ep2",
	"pad_to_sequence_len": true,
	"pretrain_multipack_attn": true,
	"profiler_steps_start": 0,
	"qlora_sharded_model_loading": false,
	"quantize_moe_experts": false,
	"ray_num_workers": 1,
	"resources_per_worker": {
	"GPU": 1
	},
	"sample_packing": false,
	"sample_packing_bin_size": 200,
	"sample_packing_group_size": 100000,
	"save_only_model": false,
	"save_safetensors": true,
	"save_steps": 10,
	"save_total_limit": 1000,
	"sequence_len": 4096,
	"shuffle_before_merging_datasets": false,
	"shuffle_merged_datasets": true,
	"skip_prepare_dataset": false,
	"streaming_multipack_buffer_size": 10000,
	"strict": false,
	"tensor_parallel_size": 1,
	"test_datasets": [
	{
	"chat_template": "tokenizer_default",
	"message_property_mappings": {
	"content": "content",
	"role": "role"
	},
	"path": "felixwangg/codenet-c-cpp-stage1",
	"split": "validation",
	"trust_remote_code": false,
	"type": "chat_template"
	}
	],
	"tf32": false,
	"tiled_mlp_use_original_mlp": true,
	"tokenizer_config": "Qwen/Qwen2.5-Coder-7B-Instruct",
	"tokenizer_save_jinja_files": true,
	"tokenizer_type": "AutoTokenizer",
	"torch_dtype": "torch.bfloat16",
	"train_on_inputs": false,
	"trl": {
	"async_prefetch": false,
	"log_completions": false,
	"mask_truncated_completions": false,
	"ref_model_mixup_alpha": 0.9,
	"ref_model_sync_steps": 64,
	"replay_buffer_size": 0,
	"replay_recompute_logps": true,
	"reroll_max_groups": 1,
	"reroll_start_fraction": 1.0,
	"reward_num_workers": 1,
	"scale_rewards": true,
	"skip_zero_advantage_batches": true,
	"sync_ref_model": false,
	"use_data_producer": false,
	"use_vllm": false,
	"vllm_lora_sync": false,
	"vllm_server_host": "0.0.0.0",
	"vllm_server_port": 8000
	},
	"type_of_model": "Qwen2ForCausalLM",
	"use_otel_metrics": false,
	"use_ray": false,
	"use_wandb": true,
	"val_set_size": 0.0,
	"vllm": {
	"device": "auto",
	"dtype": "auto",
	"gpu_memory_utilization": 0.9,
	"host": "0.0.0.0",
	"port": 8000
	},
	"wandb_entity": "wtkuan",
	"wandb_log_model": "false",
	"wandb_name": "Qwen2.5-Coder-7B-func-stage1",
	"wandb_project": "sft-primevul-sweep-ctx-0",
	"wandb_watch": "false",
	"warmup_ratio": 0.1,
	"weight_decay": 0.02,
	"world_size": 2
	}
	[2026-04-21 01:38:42,509] [DEBUG] [axolotl.loaders.tokenizer.load_tokenizer:307] [PID:4094106] EOS: 151645 / <\|im_end\|>
	[2026-04-21 01:38:42,509] [DEBUG] [axolotl.loaders.tokenizer.load_tokenizer:308] [PID:4094106] BOS: None / None
	[2026-04-21 01:38:42,509] [DEBUG] [axolotl.loaders.tokenizer.load_tokenizer:309] [PID:4094106] PAD: 151643 / <\|endoftext\|>
	[2026-04-21 01:38:42,509] [DEBUG] [axolotl.loaders.tokenizer.load_tokenizer:310] [PID:4094106] UNK: None / None
	[2026-04-21 01:38:42,639] [INFO] [axolotl.utils.data.shared.load_preprocessed_dataset:475] [PID:4094106] Loading prepared dataset from disk at /home/tkwang/scratch/SecSteer-v2/axolotl-datasets/lora/Qwen2.5-Coder-7B/func-stage1/96a3e20eec657a543cd77229b2f68312...
	[2026-04-21 01:38:42,652] [INFO] [axolotl.utils.data.shared.load_preprocessed_dataset:475] [PID:4094106] Loading prepared dataset from disk at /home/tkwang/scratch/SecSteer-v2/axolotl-datasets/lora/Qwen2.5-Coder-7B/func-stage1/2f9b6666b4ccc6f854316e096ff5dcca...
	[2026-04-21 01:38:42,785] [DEBUG] [axolotl.utils.trainer.calculate_total_num_steps:420] [PID:4094106] total_num_tokens: 1_094_060
	[2026-04-21 01:38:42,892] [DEBUG] [axolotl.utils.trainer.calculate_total_num_steps:438] [PID:4094106] `total_supervised_tokens: 536_809`
	[2026-04-21 01:38:42,892] [DEBUG] [axolotl.utils.trainer.calculate_total_num_steps:521] [PID:4094106] total_num_steps: 29
	[2026-04-21 01:38:42,892] [INFO] [axolotl.utils.data.sft._prepare_standard_dataset:121] [PID:4094106] Maximum number of steps set at 29
	[2026-04-21 01:38:42,917] [DEBUG] [axolotl.train.setup_model_and_tokenizer:70] [PID:4094106] loading tokenizer... Qwen/Qwen2.5-Coder-7B-Instruct
	[2026-04-21 01:38:43,526] [DEBUG] [axolotl.loaders.tokenizer.load_tokenizer:307] [PID:4094106] EOS: 151645 / <\|im_end\|>
	[2026-04-21 01:38:43,526] [DEBUG] [axolotl.loaders.tokenizer.load_tokenizer:308] [PID:4094106] BOS: None / None
	[2026-04-21 01:38:43,526] [DEBUG] [axolotl.loaders.tokenizer.load_tokenizer:309] [PID:4094106] PAD: 151643 / <\|endoftext\|>
	[2026-04-21 01:38:43,526] [DEBUG] [axolotl.loaders.tokenizer.load_tokenizer:310] [PID:4094106] UNK: None / None
	[2026-04-21 01:38:43,526] [DEBUG] [axolotl.train.setup_model_and_tokenizer:81] [PID:4094106] Loading model
	[2026-04-21 01:38:43,594] [DEBUG] [axolotl.monkeypatch.torchao_optim.patch_torchao_optim_state_8bit:75] [PID:4094106] Patched OptimState8bit for torch.compile compatibility
	[2026-04-21 01:38:43,594] [DEBUG] [axolotl.monkeypatch.torchao_optim.patch_torchao_optim_state_8bit:122] [PID:4094106] Patched OptimState4bit for torch.compile compatibility
	[2026-04-21 01:38:43,594] [DEBUG] [axolotl.monkeypatch.torchao_optim.patch_torchao_optim_state_8bit:154] [PID:4094106] Patched OptimStateFp8 for torch.compile compatibility
	[2026-04-21 01:38:43,606] [DEBUG] [axolotl.monkeypatch.transformers.trainer_loss_calc.patch_evaluation_loop:94] [PID:4094106] Patched Trainer.evaluation_loop with nanmean loss calculation
	[2026-04-21 01:38:43,606] [DEBUG] [axolotl.monkeypatch.transformers.trainer_loss_calc.patch_maybe_log_save_evaluate:148] [PID:4094106] Patched Trainer._maybe_log_save_evaluate with nanmean loss calculation
	[2026-04-21 01:38:43,627] [INFO] [axolotl.monkeypatch.attention.flash_attn_4.patch_flash_attn_4:52] [PID:4094106] Flash Attention 4 is available for your GPU and offers faster training speeds. To enable: pip install flash-attn-4
	[2026-04-21 01:38:43,629] [WARNING] [axolotl.loaders.patch_manager._apply_self_attention_lora_patch:436] [PID:4094106] Cannot patch self-attention - requires no dropout
	Downloading (incomplete total...): 0.00B [00:00, ?B/s]
	Fetching 22 files: 0%\| \| 0/22 [00:00<?, ?it/s][A Fetching 22 files: 100%\|█████████████████████████████████████████████████████████████████████████████████\| 22/22 [00:00<00:00, 1044.28it/s]
	Download complete: : 0.00B [00:00, ?B/s] Download complete: : 0.00B [00:00, ?B/s]
	Loading weights: 0%\| \| 0/339 [00:00<?, ?it/s] Loading weights: 0%\|▎ \| 1/339 [00:00<00:57, 5.85it/s] Loading weights: 1%\|▌ \| 2/339 [00:00<01:01, 5.51it/s] Loading weights: 2%\|█▊ \| 7/339 [00:00<00:27, 12.05it/s] Loading weights: 17%\|█████████████▉ \| 56/339 [00:00<00:03, 83.84it/s] Loading weights: 27%\|███████████████████████ \| 93/339 [00:01<00:03, 74.45it/s] Loading weights: 53%\|███████████████████████████████████████████▊ \| 181/339 [00:01<00:01, 141.95it/s] Loading weights: 58%\|███████████████████████████████████████████████▋ \| 197/339 [00:02<00:01, 124.45it/s] Loading weights: 80%\|█████████████████████████████████████████████████████████████████▎ \| 270/339 [00:02<00:00, 169.76it/s] Loading weights: 92%\|███████████████████████████████████████████████████████████████████████████▍ \| 312/339 [00:02<00:00, 199.65it/s] Loading weights: 100%\|██████████████████████████████████████████████████████████████████████████████████\| 339/339 [00:02<00:00, 141.07it/s]
	[2026-04-21 01:38:47,039] [INFO] [axolotl.loaders.model._configure_embedding_dtypes:361] [PID:4094106] Converting modules to torch.bfloat16
	[2026-04-21 01:38:47,040] [DEBUG] [axolotl.loaders.model.log_gpu_memory_usage:127] [PID:4094106] Memory usage after model load 17.234GB (+17.234GB allocated, +18.250GB reserved)
	[2026-04-21 01:38:47,041] [INFO] [axolotl.loaders.adapter.load_lora:81] [PID:4094106] found linear modules: ['down_proj', 'gate_proj', 'k_proj', 'o_proj', 'q_proj', 'up_proj', 'v_proj']
	trainable params: 40,370,176 \|\| all params: 7,655,986,688 \|\| trainable%: 0.5273
	[2026-04-21 01:38:47,549] [DEBUG] [axolotl.loaders.model.log_gpu_memory_usage:127] [PID:4094106] after adapters 14.337GB (+14.337GB allocated, +18.328GB reserved)
	[2026-04-21 01:38:51,919] [INFO] [axolotl.monkeypatch.lora_kernels.apply_lora_kernel_patches:436] [PID:4094106] LoRA kernels: dropout=0.05 enabled
	[2026-04-21 01:38:52,943] [INFO] [axolotl.train.save_initial_configs:417] [PID:4094106] Pre-saving adapter config to /home/tkwang/scratch/SecSteer-v2/axolotl-outputs/lora/Qwen2.5-Coder-7B-func-stage1_ep2...
	[2026-04-21 01:38:52,974] [INFO] [axolotl.train.save_initial_configs:421] [PID:4094106] Pre-saving tokenizer to /home/tkwang/scratch/SecSteer-v2/axolotl-outputs/lora/Qwen2.5-Coder-7B-func-stage1_ep2...
	[2026-04-21 01:38:53,113] [INFO] [axolotl.train.save_initial_configs:426] [PID:4094106] Pre-saving model config to /home/tkwang/scratch/SecSteer-v2/axolotl-outputs/lora/Qwen2.5-Coder-7B-func-stage1_ep2...
	[2026-04-21 01:38:53,155] [INFO] [axolotl.train.execute_training:222] [PID:4094106] Starting trainer...
	[34m[1mwandb[0m: [wandb.login()] Loaded credentials for https://api.wandb.ai from /home/tkwang/.netrc.
	[34m[1mwandb[0m: Currently logged in as: [33mwtkuan[0m to [32mhttps://api.wandb.ai[0m. Use [1m`wandb login --relogin`[0m to force relogin
	[34m[1mwandb[0m: Using an existing wandb-core service via WANDB_SERVICE.
	[34m[1mwandb[0m: [38;5;178m⢿[0m Waiting for wandb.init()...
	[Am[2K [34m[1mwandb[0m: [38;5;178m⣻[0m Waiting for wandb.init()...
	[Am[2K [34m[1mwandb[0m: [38;5;178m⣽[0m Waiting for wandb.init()...
	[Am[2K [34m[1mwandb[0m: Tracking run with wandb version 0.26.0
	[34m[1mwandb[0m: Run data is saved locally in [35m[1m/scratch/tkwang/SecSteer-v2/wandb/run-20260421_013854-in2on1yr[0m
	[34m[1mwandb[0m: Run [1m`wandb offline`[0m to turn off syncing.
	[34m[1mwandb[0m: Syncing run [33mQwen2.5-Coder-7B-func-stage1[0m
	[34m[1mwandb[0m: ⭐️ View project at [34m[4mhttps://wandb.ai/wtkuan/sft-primevul-sweep-ctx-0[0m
	[34m[1mwandb[0m: 🚀 View run at [34m[4mhttps://wandb.ai/wtkuan/sft-primevul-sweep-ctx-0/runs/in2on1yr[0m
	[34m[1mwandb[0m: [33mWARNING[0m Saving files without folders. If you want to preserve subdirectories pass base_path to wandb.save, i.e. wandb.save("/mnt/folder/file.h5", base_path="/mnt")
	[34m[1mwandb[0m: [33mWARNING[0m Symlinked 1 file into the W&B run directory; call wandb.save again to sync new files.
	[2026-04-21 01:38:58,326] [INFO] [axolotl.utils.callbacks.on_train_begin:757] [PID:4094106] The Axolotl config has been saved to the WandB run under files.
	0%\| \| 0/29 [00:00<?, ?it/s][2026-04-21 01:38:58,329] [INFO] [axolotl.core.trainers.base.evaluate:410] [PID:4094106] Running evaluation step...

	0%\| \| 0/13 [00:00<?, ?it/s][A
	15%\|███████████████▊ \| 2/13 [00:00<00:02, 4.05it/s][A
	23%\|███████████████████████▊ \| 3/13 [00:01<00:03, 2.66it/s][A
	31%\|███████████████████████████████▋ \| 4/13 [00:01<00:03, 2.25it/s][A
	38%\|███████████████████████████████████████▌ \| 5/13 [00:02<00:03, 2.06it/s][A
	46%\|███████████████████████████████████████████████▌ \| 6/13 [00:02<00:03, 1.94it/s][A
	54%\|███████████████████████████████████████████████████████▍ \| 7/13 [00:03<00:03, 1.88it/s][A
	62%\|███████████████████████████████████████████████████████████████▍ \| 8/13 [00:03<00:02, 1.86it/s][A
	69%\|███████████████████████████████████████████████████████████████████████▎ \| 9/13 [00:04<00:02, 1.81it/s][A
	77%\|██████████████████████████████████████████████████████████████████████████████▍ \| 10/13 [00:05<00:01, 1.80it/s][A
	85%\|██████████████████████████████████████████████████████████████████████████████████████▎ \| 11/13 [00:05<00:01, 1.80it/s][A
	92%\|██████████████████████████████████████████████████████████████████████████████████████████████▏ \| 12/13 [00:06<00:00, 1.79it/s][ATraceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-5mf09xkg'
	Traceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-ldyxvpw8'

	100%\|██████████████████████████████████████████████████████████████████████████████████████████████████████\| 13/13 [00:06<00:00, 1.75it/s][A
	[A{'eval_loss': '0.6622', 'eval_runtime': '8.304', 'eval_samples_per_second': '12.04', 'eval_steps_per_second': '1.565', 'eval_ppl': '1.939', 'memory/max_active (GiB)': '37.85', 'memory/max_allocated (GiB)': '37.85', 'memory/device_reserved (GiB)': '41.82', 'epoch': 0}
	0%\| \| 0/29 [00:08<?, ?it/s]
	100%\|██████████████████████████████████████████████████████████████████████████████████████████████████████\| 13/13 [00:06<00:00, 1.75it/s][A
	[A 3%\|███▌ \| 1/29 [00:23<11:07, 23.85s/it] {'loss': '0.5971', 'grad_norm': '0.04773', 'learning_rate': '0', 'ppl': '1.817', 'memory/max_active (GiB)': '45.84', 'memory/max_allocated (GiB)': '45.84', 'memory/device_reserved (GiB)': '51.31', 'tokens/train_per_sec_per_gpu': '255.5', 'tokens/total': 262144, 'tokens/trainable': 42344, 'epoch': '0.0708'}
	3%\|███▌ \| 1/29 [00:23<11:07, 23.85s/it] 7%\|███████ \| 2/29 [00:38<08:17, 18.44s/it] {'loss': '0.6382', 'grad_norm': '0.05039', 'learning_rate': '2e-05', 'ppl': '1.893', 'memory/max_active (GiB)': '46.14', 'memory/max_allocated (GiB)': '46.14', 'memory/device_reserved (GiB)': '51.31', 'tokens/train_per_sec_per_gpu': '107.6', 'tokens/total': 524288, 'tokens/trainable': 78005, 'epoch': '0.1416'}
	7%\|███████ \| 2/29 [00:38<08:17, 18.44s/it] 10%\|██████████▋ \| 3/29 [00:53<07:15, 16.76s/it] {'loss': '0.6144', 'grad_norm': '0.04302', 'learning_rate': '4e-05', 'ppl': '1.849', 'memory/max_active (GiB)': '46.14', 'memory/max_allocated (GiB)': '46.14', 'memory/device_reserved (GiB)': '51.31', 'tokens/train_per_sec_per_gpu': '220.7', 'tokens/total': 786432, 'tokens/trainable': 122911, 'epoch': '0.2124'}
	10%\|██████████▋ \| 3/29 [00:53<07:15, 16.76s/it] 14%\|██████████████▏ \| 4/29 [01:07<06:38, 15.95s/it] {'loss': '0.5998', 'grad_norm': '0.05384', 'learning_rate': '3.986e-05', 'ppl': '1.822', 'memory/max_active (GiB)': '46.14', 'memory/max_allocated (GiB)': '46.14', 'memory/device_reserved (GiB)': '51.31', 'tokens/train_per_sec_per_gpu': '165.6', 'tokens/total': 1048576, 'tokens/trainable': 157955, 'epoch': '0.2832'}
	14%\|██████████████▏ \| 4/29 [01:07<06:38, 15.95s/it] 17%\|█████████████████▊ \| 5/29 [01:22<06:11, 15.48s/it] {'loss': '0.5847', 'grad_norm': '0.05714', 'learning_rate': '3.946e-05', 'ppl': '1.794', 'memory/max_active (GiB)': '46.14', 'memory/max_allocated (GiB)': '46.14', 'memory/device_reserved (GiB)': '51.31', 'tokens/train_per_sec_per_gpu': '205', 'tokens/total': 1310720, 'tokens/trainable': 194480, 'epoch': '0.354'}
	17%\|█████████████████▊ \| 5/29 [01:22<06:11, 15.48s/it] 21%\|█████████████████████▎ \| 6/29 [01:37<05:49, 15.21s/it] {'loss': '0.6255', 'grad_norm': '0.06666', 'learning_rate': '3.879e-05', 'ppl': '1.869', 'memory/max_active (GiB)': '46.14', 'memory/max_allocated (GiB)': '46.14', 'memory/device_reserved (GiB)': '51.31', 'tokens/train_per_sec_per_gpu': '151.7', 'tokens/total': 1572864, 'tokens/trainable': 225910, 'epoch': '0.4248'}
	21%\|█████████████████████▎ \| 6/29 [01:37<05:49, 15.21s/it] 24%\|████████████████████████▊ \| 7/29 [01:52<05:31, 15.05s/it] {'loss': '0.6049', 'grad_norm': '0.06925', 'learning_rate': '3.787e-05', 'ppl': '1.831', 'memory/max_active (GiB)': '46.14', 'memory/max_allocated (GiB)': '46.14', 'memory/device_reserved (GiB)': '51.31', 'tokens/train_per_sec_per_gpu': '135.8', 'tokens/total': 1835008, 'tokens/trainable': 262101, 'epoch': '0.4956'}
	24%\|████████████████████████▊ \| 7/29 [01:52<05:31, 15.05s/it] 28%\|████████████████████████████▍ \| 8/29 [02:06<05:14, 14.98s/it] {'loss': '0.6102', 'grad_norm': '0.07112', 'learning_rate': '3.671e-05', 'ppl': '1.841', 'memory/max_active (GiB)': '46.14', 'memory/max_allocated (GiB)': '46.14', 'memory/device_reserved (GiB)': '51.31', 'tokens/train_per_sec_per_gpu': '116.5', 'tokens/total': 2097152, 'tokens/trainable': 300650, 'epoch': '0.5664'}
	28%\|████████████████████████████▍ \| 8/29 [02:06<05:14, 14.98s/it] 31%\|███████████████████████████████▉ \| 9/29 [02:21<04:57, 14.86s/it] {'loss': '0.6072', 'grad_norm': '0.08064', 'learning_rate': '3.532e-05', 'ppl': '1.835', 'memory/max_active (GiB)': '46.14', 'memory/max_allocated (GiB)': '46.14', 'memory/device_reserved (GiB)': '51.31', 'tokens/train_per_sec_per_gpu': '80.15', 'tokens/total': 2359296, 'tokens/trainable': 331623, 'epoch': '0.6372'}
	31%\|███████████████████████████████▉ \| 9/29 [02:21<04:57, 14.86s/it] 34%\|███████████████████████████████████▏ \| 10/29 [02:36<04:42, 14.84s/it] {'loss': '0.6327', 'grad_norm': '0.05834', 'learning_rate': '3.372e-05', 'ppl': '1.883', 'memory/max_active (GiB)': '46.14', 'memory/max_allocated (GiB)': '46.14', 'memory/device_reserved (GiB)': '51.31', 'tokens/train_per_sec_per_gpu': '120.9', 'tokens/total': 2621440, 'tokens/trainable': 373610, 'epoch': '0.708'}
	34%\|███████████████████████████████████▏ \| 10/29 [02:36<04:42, 14.84s/it][2026-04-21 01:41:34,589] [INFO] [axolotl.core.trainers.base.evaluate:410] [PID:4094106] Running evaluation step...

	0%\| \| 0/13 [00:00<?, ?it/s][A
	15%\|███████████████▊ \| 2/13 [00:00<00:03, 3.56it/s][A
	23%\|███████████████████████▊ \| 3/13 [00:01<00:04, 2.50it/s][A
	31%\|███████████████████████████████▋ \| 4/13 [00:01<00:04, 2.17it/s][A
	38%\|███████████████████████████████████████▌ \| 5/13 [00:02<00:03, 2.01it/s][A
	46%\|███████████████████████████████████████████████▌ \| 6/13 [00:02<00:03, 1.91it/s][A
	54%\|███████████████████████████████████████████████████████▍ \| 7/13 [00:03<00:03, 1.86it/s][A
	62%\|███████████████████████████████████████████████████████████████▍ \| 8/13 [00:03<00:02, 1.84it/s][A
	69%\|███████████████████████████████████████████████████████████████████████▎ \| 9/13 [00:04<00:02, 1.78it/s][A
	77%\|██████████████████████████████████████████████████████████████████████████████▍ \| 10/13 [00:05<00:01, 1.78it/s][A
	85%\|██████████████████████████████████████████████████████████████████████████████████████▎ \| 11/13 [00:05<00:01, 1.77it/s][A
	92%\|██████████████████████████████████████████████████████████████████████████████████████████████▏ \| 12/13 [00:06<00:00, 1.77it/s][ATraceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-b73r0bof'
	Traceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-29nxmw11'

	100%\|██████████████████████████████████████████████████████████████████████████████████████████████████████\| 13/13 [00:06<00:00, 1.73it/s][A
	[A{'eval_loss': '0.6354', 'eval_runtime': '7.602', 'eval_samples_per_second': '13.15', 'eval_steps_per_second': '1.71', 'eval_ppl': '1.888', 'memory/max_active (GiB)': '38.19', 'memory/max_allocated (GiB)': '38.19', 'memory/device_reserved (GiB)': '51.31', 'epoch': '0.708', 'tokens/train_per_sec_per_gpu': '0'}
	34%\|███████████████████████████████████▏ \| 10/29 [02:43<04:42, 14.84s/it]
	100%\|██████████████████████████████████████████████████████████████████████████████████████████████████████\| 13/13 [00:06<00:00, 1.73it/s][A
	[A[2026-04-21 01:41:42,221] [INFO] [axolotl.core.trainers.base._save:741] [PID:4094106] Saving model checkpoint to /home/tkwang/scratch/SecSteer-v2/axolotl-outputs/lora/Qwen2.5-Coder-7B-func-stage1_ep2/checkpoint-10
	[2026-04-21 01:41:42,811] [WARNING] [py.warnings._showwarnmsg:112] [PID:4094106] /scratch/tkwang/SecSteer-v2/.venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
	return func(args, *kwargs)

	38%\|██████████████████████████████████████▋ \| 11/29 [02:59<05:15, 17.52s/it] {'loss': '0.617', 'grad_norm': '0.04605', 'learning_rate': '3.194e-05', 'ppl': '1.853', 'memory/max_active (GiB)': '46.14', 'memory/max_allocated (GiB)': '46.14', 'memory/device_reserved (GiB)': '52.47', 'tokens/train_per_sec_per_gpu': '127.9', 'tokens/total': 2883584, 'tokens/trainable': 413641, 'epoch': '0.7788'}
	38%\|██████████████████████████████████████▋ \| 11/29 [02:59<05:15, 17.52s/it] 41%\|██████████████████████████████████████████▏ \| 12/29 [03:14<04:43, 16.68s/it] {'loss': '0.6267', 'grad_norm': '0.04807', 'learning_rate': '3e-05', 'ppl': '1.871', 'memory/max_active (GiB)': '46.14', 'memory/max_allocated (GiB)': '46.14', 'memory/device_reserved (GiB)': '52.47', 'tokens/train_per_sec_per_gpu': '164.8', 'tokens/total': 3145728, 'tokens/trainable': 454177, 'epoch': '0.8496'}
	41%\|██████████████████████████████████████████▏ \| 12/29 [03:14<04:43, 16.68s/it] 45%\|█████████████████████████████████████████████▋ \| 13/29 [03:29<04:17, 16.09s/it] {'loss': '0.572', 'grad_norm': '0.05148', 'learning_rate': '2.792e-05', 'ppl': '1.772', 'memory/max_active (GiB)': '46.14', 'memory/max_allocated (GiB)': '46.14', 'memory/device_reserved (GiB)': '52.47', 'tokens/train_per_sec_per_gpu': '185', 'tokens/total': 3407872, 'tokens/trainable': 494979, 'epoch': '0.9204'}
	45%\|█████████████████████████████████████████████▋ \| 13/29 [03:29<04:17, 16.09s/it] 48%\|█████████████████████████████████████████████████▏ \| 14/29 [03:44<03:55, 15.69s/it] {'loss': '0.5987', 'grad_norm': '0.06019', 'learning_rate': '2.574e-05', 'ppl': '1.82', 'memory/max_active (GiB)': '46.14', 'memory/max_allocated (GiB)': '46.14', 'memory/device_reserved (GiB)': '52.47', 'tokens/train_per_sec_per_gpu': '159.7', 'tokens/total': 3670016, 'tokens/trainable': 529163, 'epoch': '0.9912'}
	48%\|█████████████████████████████████████████████████▏ \| 14/29 [03:44<03:55, 15.69s/it]Traceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-il4famz5'
	Traceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-9u0_891k'
	52%\|████████████████████████████████████████████████████▊ \| 15/29 [03:46<02:41, 11.56s/it] {'loss': '0.6511', 'grad_norm': '0.08155', 'learning_rate': '2.347e-05', 'ppl': '1.918', 'memory/max_active (GiB)': '45.98', 'memory/max_allocated (GiB)': '45.98', 'memory/device_reserved (GiB)': '52.47', 'tokens/train_per_sec_per_gpu': '2265', 'tokens/total': 3702784, 'tokens/trainable': 538007, 'epoch': '1'}
	52%\|████████████████████████████████████████████████████▊ \| 15/29 [03:46<02:41, 11.56s/it] 55%\|████████████████████████████████████████████████████████▎ \| 16/29 [04:01<02:45, 12.70s/it] {'loss': '0.608', 'grad_norm': '0.06687', 'learning_rate': '2.116e-05', 'ppl': '1.837', 'memory/max_active (GiB)': '46.14', 'memory/max_allocated (GiB)': '46.14', 'memory/device_reserved (GiB)': '52.47', 'tokens/train_per_sec_per_gpu': '103.1', 'tokens/total': 3964928, 'tokens/trainable': 569318, 'epoch': '1.071'}
	55%\|████████████████████████████████████████████████████████▎ \| 16/29 [04:01<02:45, 12.70s/it] 59%\|███████████████████████████████████████████████████████████▊ \| 17/29 [04:16<02:39, 13.31s/it] {'loss': '0.6013', 'grad_norm': '0.0562', 'learning_rate': '1.884e-05', 'ppl': '1.824', 'memory/max_active (GiB)': '46.14', 'memory/max_allocated (GiB)': '46.14', 'memory/device_reserved (GiB)': '52.47', 'tokens/train_per_sec_per_gpu': '120.6', 'tokens/total': 4227072, 'tokens/trainable': 606064, 'epoch': '1.142'}
	59%\|███████████████████████████████████████████████████████████▊ \| 17/29 [04:16<02:39, 13.31s/it] 62%\|███████████████████████████████████████████████████████████████▎ \| 18/29 [04:30<02:31, 13.75s/it] {'loss': '0.568', 'grad_norm': '0.0509', 'learning_rate': '1.653e-05', 'ppl': '1.765', 'memory/max_active (GiB)': '46.14', 'memory/max_allocated (GiB)': '46.14', 'memory/device_reserved (GiB)': '52.47', 'tokens/train_per_sec_per_gpu': '204.4', 'tokens/total': 4489216, 'tokens/trainable': 647245, 'epoch': '1.212'}
	62%\|███████████████████████████████████████████████████████████████▎ \| 18/29 [04:30<02:31, 13.75s/it] 66%\|██████████████████████████████████████████████████████████████████▊ \| 19/29 [04:45<02:20, 14.06s/it] {'loss': '0.5718', 'grad_norm': '0.05827', 'learning_rate': '1.426e-05', 'ppl': '1.771', 'memory/max_active (GiB)': '46.14', 'memory/max_allocated (GiB)': '46.14', 'memory/device_reserved (GiB)': '52.47', 'tokens/train_per_sec_per_gpu': '129.2', 'tokens/total': 4751360, 'tokens/trainable': 687200, 'epoch': '1.283'}
	66%\|██████████████████████████████████████████████████████████████████▊ \| 19/29 [04:45<02:20, 14.06s/it] 69%\|██████████████████████████████████████████████████████████████████████▎ \| 20/29 [05:00<02:08, 14.25s/it] {'loss': '0.5751', 'grad_norm': '0.06494', 'learning_rate': '1.208e-05', 'ppl': '1.777', 'memory/max_active (GiB)': '46.14', 'memory/max_allocated (GiB)': '46.14', 'memory/device_reserved (GiB)': '52.47', 'tokens/train_per_sec_per_gpu': '65.45', 'tokens/total': 5013504, 'tokens/trainable': 719359, 'epoch': '1.354'}
	69%\|██████████████████████████████████████████████████████████████████████▎ \| 20/29 [05:00<02:08, 14.25s/it][2026-04-21 01:43:58,748] [INFO] [axolotl.core.trainers.base.evaluate:410] [PID:4094106] Running evaluation step...

	0%\| \| 0/13 [00:00<?, ?it/s][A
	15%\|███████████████▊ \| 2/13 [00:00<00:03, 3.58it/s][A
	23%\|███████████████████████▊ \| 3/13 [00:01<00:03, 2.52it/s][A
	31%\|███████████████████████████████▋ \| 4/13 [00:01<00:04, 2.18it/s][A
	38%\|███████████████████████████████████████▌ \| 5/13 [00:02<00:03, 2.01it/s][A
	46%\|███████████████████████████████████████████████▌ \| 6/13 [00:02<00:03, 1.91it/s][A
	54%\|███████████████████████████████████████████████████████▍ \| 7/13 [00:03<00:03, 1.86it/s][A
	62%\|███████████████████████████████████████████████████████████████▍ \| 8/13 [00:03<00:02, 1.84it/s][A
	69%\|███████████████████████████████████████████████████████████████████████▎ \| 9/13 [00:04<00:02, 1.78it/s][A
	77%\|██████████████████████████████████████████████████████████████████████████████▍ \| 10/13 [00:05<00:01, 1.78it/s][A
	85%\|██████████████████████████████████████████████████████████████████████████████████████▎ \| 11/13 [00:05<00:01, 1.78it/s][A
	92%\|██████████████████████████████████████████████████████████████████████████████████████████████▏ \| 12/13 [00:06<00:00, 1.77it/s][ATraceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-ud2iugh6'
	Traceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-8vj7ni6m'

	100%\|██████████████████████████████████████████████████████████████████████████████████████████████████████\| 13/13 [00:06<00:00, 1.71it/s][A
	[A{'eval_loss': '0.6173', 'eval_runtime': '7.6', 'eval_samples_per_second': '13.16', 'eval_steps_per_second': '1.711', 'eval_ppl': '1.854', 'memory/max_active (GiB)': '38.19', 'memory/max_allocated (GiB)': '38.19', 'memory/device_reserved (GiB)': '52.47', 'epoch': '1.354', 'tokens/train_per_sec_per_gpu': '0'}
	69%\|██████████████████████████████████████████████████████████████████████▎ \| 20/29 [05:08<02:08, 14.25s/it]
	100%\|██████████████████████████████████████████████████████████████████████████████████████████████████████\| 13/13 [00:06<00:00, 1.71it/s][A
	[A[2026-04-21 01:44:06,381] [INFO] [axolotl.core.trainers.base._save:741] [PID:4094106] Saving model checkpoint to /home/tkwang/scratch/SecSteer-v2/axolotl-outputs/lora/Qwen2.5-Coder-7B-func-stage1_ep2/checkpoint-20
	[2026-04-21 01:44:07,059] [WARNING] [py.warnings._showwarnmsg:112] [PID:4094106] /scratch/tkwang/SecSteer-v2/.venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
	return func(args, *kwargs)

	72%\|█████████████████████████████████████████████████████████████████████████▊ \| 21/29 [05:23<02:16, 17.02s/it] {'loss': '0.6677', 'grad_norm': '0.07199', 'learning_rate': '1e-05', 'ppl': '1.95', 'memory/max_active (GiB)': '46.15', 'memory/max_allocated (GiB)': '46.15', 'memory/device_reserved (GiB)': '52.46', 'tokens/train_per_sec_per_gpu': '97.78', 'tokens/total': 5275648, 'tokens/trainable': 748391, 'epoch': '1.425'}
	72%\|█████████████████████████████████████████████████████████████████████████▊ \| 21/29 [05:23<02:16, 17.02s/it] 76%\|█████████████████████████████████████████████████████████████████████████████▍ \| 22/29 [05:38<01:54, 16.38s/it] {'loss': '0.6035', 'grad_norm': '0.05376', 'learning_rate': '8.057e-06', 'ppl': '1.828', 'memory/max_active (GiB)': '46.14', 'memory/max_allocated (GiB)': '46.14', 'memory/device_reserved (GiB)': '52.46', 'tokens/train_per_sec_per_gpu': '183.6', 'tokens/total': 5537792, 'tokens/trainable': 792092, 'epoch': '1.496'}
	76%\|█████████████████████████████████████████████████████████████████████████████▍ \| 22/29 [05:38<01:54, 16.38s/it] 79%\|████████████████████████████████████████████████████████████████████████████████▉ \| 23/29 [05:53<01:35, 15.91s/it] {'loss': '0.5563', 'grad_norm': '0.05552', 'learning_rate': '6.275e-06', 'ppl': '1.744', 'memory/max_active (GiB)': '46.14', 'memory/max_allocated (GiB)': '46.14', 'memory/device_reserved (GiB)': '52.47', 'tokens/train_per_sec_per_gpu': '137.5', 'tokens/total': 5799936, 'tokens/trainable': 831415, 'epoch': '1.566'}
	79%\|████████████████████████████████████████████████████████████████████████████████▉ \| 23/29 [05:53<01:35, 15.91s/it] 83%\|████████████████████████████████████████████████████████████████████████████████████▍ \| 24/29 [06:08<01:17, 15.56s/it] {'loss': '0.5609', 'grad_norm': '0.05277', 'learning_rate': '4.679e-06', 'ppl': '1.752', 'memory/max_active (GiB)': '46.14', 'memory/max_allocated (GiB)': '46.14', 'memory/device_reserved (GiB)': '52.47', 'tokens/train_per_sec_per_gpu': '237.8', 'tokens/total': 6062080, 'tokens/trainable': 872592, 'epoch': '1.637'}
	83%\|████████████████████████████████████████████████████████████████████████████████████▍ \| 24/29 [06:08<01:17, 15.56s/it] 86%\|███████████████████████████████████████████████████████████████████████████████████████▉ \| 25/29 [06:23<01:01, 15.32s/it] {'loss': '0.5744', 'grad_norm': '0.05725', 'learning_rate': '3.29e-06', 'ppl': '1.776', 'memory/max_active (GiB)': '46.14', 'memory/max_allocated (GiB)': '46.14', 'memory/device_reserved (GiB)': '52.47', 'tokens/train_per_sec_per_gpu': '170.7', 'tokens/total': 6324224, 'tokens/trainable': 911982, 'epoch': '1.708'}
	86%\|███████████████████████████████████████████████████████████████████████████████████████▉ \| 25/29 [06:23<01:01, 15.32s/it] 90%\|███████████████████████████████████████████████████████████████████████████████████████████▍ \| 26/29 [06:37<00:45, 15.15s/it] {'loss': '0.5752', 'grad_norm': '0.05706', 'learning_rate': '2.127e-06', 'ppl': '1.777', 'memory/max_active (GiB)': '46.14', 'memory/max_allocated (GiB)': '46.14', 'memory/device_reserved (GiB)': '52.47', 'tokens/train_per_sec_per_gpu': '89.81', 'tokens/total': 6586368, 'tokens/trainable': 947372, 'epoch': '1.779'}
	90%\|███████████████████████████████████████████████████████████████████████████████████████████▍ \| 26/29 [06:37<00:45, 15.15s/it] 93%\|██████████████████████████████████████████████████████████████████████████████████████████████▉ \| 27/29 [06:52<00:30, 15.03s/it] {'loss': '0.5775', 'grad_norm': '0.04929', 'learning_rate': '1.206e-06', 'ppl': '1.782', 'memory/max_active (GiB)': '46.14', 'memory/max_allocated (GiB)': '46.14', 'memory/device_reserved (GiB)': '52.47', 'tokens/train_per_sec_per_gpu': '234.6', 'tokens/total': 6848512, 'tokens/trainable': 988766, 'epoch': '1.85'}
	93%\|██████████████████████████████████████████████████████████████████████████████████████████████▉ \| 27/29 [06:52<00:30, 15.03s/it] 97%\|██████████████████████████████████████████████████████████████████████████████████████████████████▍ \| 28/29 [07:07<00:14, 14.95s/it] {'loss': '0.6029', 'grad_norm': '0.0568', 'learning_rate': '5.391e-07', 'ppl': '1.827', 'memory/max_active (GiB)': '46.14', 'memory/max_allocated (GiB)': '46.14', 'memory/device_reserved (GiB)': '52.47', 'tokens/train_per_sec_per_gpu': '136.4', 'tokens/total': 7110656, 'tokens/trainable': 1028736, 'epoch': '1.92'}
	97%\|██████████████████████████████████████████████████████████████████████████████████████████████████▍ \| 28/29 [07:07<00:14, 14.95s/it] 100%\|██████████████████████████████████████████████████████████████████████████████████████████████████████\| 29/29 [07:22<00:00, 14.91s/it] {'loss': '0.5873', 'grad_norm': '0.05283', 'learning_rate': '1.352e-07', 'ppl': '1.799', 'memory/max_active (GiB)': '46.14', 'memory/max_allocated (GiB)': '46.14', 'memory/device_reserved (GiB)': '52.47', 'tokens/train_per_sec_per_gpu': '240.1', 'tokens/total': 7372800, 'tokens/trainable': 1069720, 'epoch': '1.991'}
	100%\|██████████████████████████████████████████████████████████████████████████████████████████████████████\| 29/29 [07:22<00:00, 14.91s/it][2026-04-21 01:46:20,487] [INFO] [axolotl.core.trainers.base.evaluate:410] [PID:4094106] Running evaluation step...

	0%\| \| 0/13 [00:00<?, ?it/s][A
	15%\|███████████████▊ \| 2/13 [00:00<00:03, 3.59it/s][A
	23%\|███████████████████████▊ \| 3/13 [00:01<00:03, 2.53it/s][A
	31%\|███████████████████████████████▋ \| 4/13 [00:01<00:04, 2.19it/s][A
	38%\|███████████████████████████████████████▌ \| 5/13 [00:02<00:03, 2.02it/s][A
	46%\|███████████████████████████████████████████████▌ \| 6/13 [00:02<00:03, 1.92it/s][A
	54%\|███████████████████████████████████████████████████████▍ \| 7/13 [00:03<00:03, 1.86it/s][A
	62%\|███████████████████████████████████████████████████████████████▍ \| 8/13 [00:03<00:02, 1.84it/s][A
	69%\|███████████████████████████████████████████████████████████████████████▎ \| 9/13 [00:04<00:02, 1.78it/s][A
	77%\|██████████████████████████████████████████████████████████████████████████████▍ \| 10/13 [00:05<00:01, 1.78it/s][A
	85%\|██████████████████████████████████████████████████████████████████████████████████████▎ \| 11/13 [00:05<00:01, 1.77it/s][A
	92%\|██████████████████████████████████████████████████████████████████████████████████████████████▏ \| 12/13 [00:06<00:00, 1.77it/s][ATraceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-xxql1hae'
	Traceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-14ctq8lf'

	100%\|██████████████████████████████████████████████████████████████████████████████████████████████████████\| 13/13 [00:06<00:00, 1.73it/s][A
	[A{'eval_loss': '0.6129', 'eval_runtime': '7.624', 'eval_samples_per_second': '13.12', 'eval_steps_per_second': '1.705', 'eval_ppl': '1.846', 'memory/max_active (GiB)': '38.19', 'memory/max_allocated (GiB)': '38.19', 'memory/device_reserved (GiB)': '52.47', 'epoch': '1.991', 'tokens/train_per_sec_per_gpu': '0'}
	100%\|██████████████████████████████████████████████████████████████████████████████████████████████████████\| 29/29 [07:29<00:00, 14.91s/it]
	100%\|██████████████████████████████████████████████████████████████████████████████████████████████████████\| 13/13 [00:06<00:00, 1.73it/s][A
	[A[2026-04-21 01:46:28,164] [INFO] [axolotl.core.trainers.base._save:741] [PID:4094106] Saving model checkpoint to /home/tkwang/scratch/SecSteer-v2/axolotl-outputs/lora/Qwen2.5-Coder-7B-func-stage1_ep2/checkpoint-29
	[2026-04-21 01:46:28,958] [WARNING] [py.warnings._showwarnmsg:112] [PID:4094106] /scratch/tkwang/SecSteer-v2/.venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py:83: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
	return func(args, *kwargs)

	Traceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-n4mrtodi'
	Traceback (most recent call last):
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 303, in _run_finalizers
	finalizer()
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 227, in __call__
	res = self._callback(self._args, *self._kwargs)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/util.py", line 136, in _remove_temp_dir
	rmtree(tempdir, onerror=onerror)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 759, in rmtree
	_rmtree_safe_fd(stack, onexc)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
	onexc(func, path, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 700, in _rmtree_safe_fd
	onexc(os.unlink, fullname, err)
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 750, in onexc
	return onerror(func, path, exc_info)
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	File "/home/tkwang/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/shutil.py", line 698, in _rmtree_safe_fd
	os.unlink(entry.name, dir_fd=topfd)
	OSError: [Errno 16] Device or resource busy: '/home/tkwang/.cache/tmp/pymp-6z4s22zy'
	{'train_runtime': '455.6', 'train_samples_per_second': '4.074', 'train_steps_per_second': '0.064', 'train_loss': '0.6004', 'memory/max_active (GiB)': '14.86', 'memory/max_allocated (GiB)': '14.86', 'memory/device_reserved (GiB)': '43.1', 'epoch': '1.991', 'tokens/train_per_sec_per_gpu': '0'}
	100%\|██████████████████████████████████████████████████████████████████████████████████████████████████████\| 29/29 [07:31<00:00, 14.91s/it] 100%\|██████████████████████████████████████████████████████████████████████████████████████████████████████\| 29/29 [07:31<00:00, 15.58s/it]
	[2026-04-21 01:46:30,226] [INFO] [axolotl.train.save_trained_model:241] [PID:4094106] Training completed! Saving trained model to /home/tkwang/scratch/SecSteer-v2/axolotl-outputs/lora/Qwen2.5-Coder-7B-func-stage1_ep2.
	[2026-04-21 01:46:30,674] [INFO] [axolotl.train.save_trained_model:355] [PID:4094106] Model successfully saved to /home/tkwang/scratch/SecSteer-v2/axolotl-outputs/lora/Qwen2.5-Coder-7B-func-stage1_ep2