|
--- |
|
datasets: |
|
- jondurbin/airoboros-gpt4-1.4.1 |
|
--- |
|
|
|
NOTE: This LoRA was trained on Llama-30b AFTER additional pretraining. I intend on providing the LoRA of that pretraining too. Applying this LoRA to base Llama-30b will likely result in a performance reduction. I have uploaded the fp16 merged weights [here](https://huggingface.co/bhenrym14/airoboros-33b-gpt4-1.4.1-lxctx-PI-16384-LoRA/) |
|
|
|
Mostly untested! |
|
|
|
Find GPTQ quantized weights and full model card here: https://huggingface.co/bhenrym14/airoboros-33b-gpt4-1.4.1-lxctx-PI-16384-GPTQ |
|
|
|
# RoPE Scaled QLoRA Fine-tune of Llama-33b on airoboros-gpt4-1.4.1 (LoRA) |
|
|
|
## Overview |
|
|
|
This is [Jon Durbin's Airoboros 33B GPT4 1.4](https://huggingface.co/jondurbin/airoboros-33b-gpt4-1.4) (LoRA) with several key modifications: |
|
- Context length extended to 16384 by RoPE Scaled Embeddings. |
|
- The Llama-33b base model is pretrained for additional 100 steps on 8192 length sequences from the pile dataset. |
|
- Used airoboros-gpt4-1.4.1 dataset instead of airoboros-gpt4-1.4 |
|
|
|
**This is a QLoRA fine-tune** |
|
|
|
Pretraining took 10 hours. Finetuning took ~41 hours on 1x RTX 6000 Ada. |
