Llama3.1-8b-instruct-SFT-2024-09-19_LoRAs

This model is a fine-tuned version of meta-llama/Meta-Llama-3.1-8B-Instruct on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.3202

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 6
• eval_batch_size: 1
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant
• num_epochs: 1.5

Training results

Training Loss	Epoch	Step	Validation Loss
1.6086	0.0795	1000	1.6472
1.4132	0.1589	2000	1.5850
1.3613	0.2384	3000	1.5402
1.3118	0.3178	4000	1.5102
1.3045	0.3973	5000	1.4901
1.2856	0.4767	6000	1.4674
1.2646	0.5562	7000	1.4431
1.2471	0.6356	8000	1.4282
1.2497	0.7151	9000	1.4089
1.2171	0.7945	10000	1.4051
1.2145	0.8740	11000	1.3926
1.2103	0.9534	12000	1.3849
1.1813	1.0329	13000	1.3707
1.1696	1.1123	14000	1.3620
1.1459	1.1918	15000	1.3536
1.1486	1.2713	16000	1.3413
1.1398	1.3507	17000	1.3324
1.1322	1.4302	18000	1.3202

Framework versions

• PEFT 0.12.0
• Transformers 4.44.2
• Pytorch 2.0.1+cu118
• Datasets 3.0.0
• Tokenizers 0.19.1