Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,95 @@
+---
+license: other
+language:
+- en
+pipeline_tag: text-generation
+inference: false
+tags:
+- transformers
+- gguf
+- imatrix
+- mpt-7b-8k-chat
+---
+Quantizations of https://huggingface.co/mosaicml/mpt-7b-8k-chat
+
+
+# From original readme
+
+## How to Use
+
+This model is best used with the MosaicML [llm-foundry repository](https://github.com/mosaicml/llm-foundry) for training and finetuning.
+
+```python
+import transformers
+model = transformers.AutoModelForCausalLM.from_pretrained(
+  'mosaicml/mpt-7b-chat-8k',
+  trust_remote_code=True
+)
+```
+Note: This model requires that `trust_remote_code=True` be passed to the `from_pretrained` method.
+This is because we use a custom `MPT` model architecture that is not yet part of the Hugging Face `transformers` package.
+`MPT` includes options for many training efficiency features such as [FlashAttention](https://arxiv.org/pdf/2205.14135.pdf), [ALiBi](https://arxiv.org/abs/2108.12409), [QK LayerNorm](https://arxiv.org/abs/2010.04245), and more.
+
+To use the optimized [triton implementation](https://github.com/openai/triton) of FlashAttention, you can load the model on GPU (`cuda:0`) with `attn_impl='triton'` and with `bfloat16` precision:
+```python
+import torch
+import transformers
+
+name = 'mosaicml/mpt-7b-chat-8k'
+
+config = transformers.AutoConfig.from_pretrained(name, trust_remote_code=True)
+config.attn_config['attn_impl'] = 'triton'  # change this to use triton-based FlashAttention
+config.init_device = 'cuda:0' # For fast initialization directly on GPU!
+
+model = transformers.AutoModelForCausalLM.from_pretrained(
+  name,
+  config=config,
+  torch_dtype=torch.bfloat16, # Load model weights in bfloat16
+  trust_remote_code=True
+)
+```
+
+The model was trained initially with a sequence length of 2048 with an additional pretraining stage for sequence length adapation up to 8192. However, ALiBi enables users to increase the maximum sequence length even further during finetuning and/or inference. For example:
+
+```python
+import transformers
+
+name = 'mosaicml/mpt-7b-chat-8k'
+
+config = transformers.AutoConfig.from_pretrained(name, trust_remote_code=True)
+config.max_seq_len = 16384 # (input + output) tokens can now be up to 16384
+
+model = transformers.AutoModelForCausalLM.from_pretrained(
+  name,
+  config=config,
+  trust_remote_code=True
+)
+```
+
+This model was trained with the MPT-7B-chat tokenizer which is based on the [EleutherAI/gpt-neox-20b](https://huggingface.co/EleutherAI/gpt-neox-20b) tokenizer and includes additional ChatML tokens.
+
+```python
+from transformers import AutoTokenizer
+tokenizer = AutoTokenizer.from_pretrained('mosaicml/mpt-7b-8k')
+```
+
+The model can then be used, for example, within a text-generation pipeline.  
+Note: when running Torch modules in lower precision, it is best practice to use the [torch.autocast context manager](https://pytorch.org/docs/stable/amp.html).
+
+```python
+from transformers import pipeline
+
+with torch.autocast('cuda', dtype=torch.bfloat16):
+    inputs = tokenizer('Here is a recipe for vegan banana bread:\n', return_tensors="pt").to('cuda')
+    outputs = model.generate(**inputs, max_new_tokens=100)
+    print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
+
+# or using the HF pipeline
+pipe = pipeline('text-generation', model=model, tokenizer=tokenizer, device='cuda:0')
+with torch.autocast('cuda', dtype=torch.bfloat16):
+    print(
+        pipe('Here is a recipe for vegan banana bread:\n',
+            max_new_tokens=100,
+            do_sample=True,
+            use_cache=True))
+```