RaphaelMourad commited on
Commit
bc1e125
1 Parent(s): b4d116e

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +55 -3
README.md CHANGED
@@ -1,3 +1,55 @@
1
- ---
2
- license: apache-2.0
3
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ tags:
4
+ - pretrained
5
+ - mistral
6
+ - peptide
7
+ ---
8
+
9
+ # Model Card for Mistral-Peptide-v1-134M (Mistral for peptide)
10
+
11
+ The Mistral-Peptide-v1-134M Large Language Model (LLM) is a pretrained generative peptide molecule model with 134M parameters.
12
+ It is derived from Mixtral-8x7B-v0.1 model, which was simplified for protein: the number of layers and the hidden size were reduced.
13
+ The model was pretrained using 863499 peptide strings.
14
+
15
+ ## Model Architecture
16
+
17
+ Like Mixtral-8x7B-v0.1, it is a transformer model, with the following architecture choices:
18
+ - Grouped-Query Attention
19
+ - Sliding-Window Attention
20
+ - Byte-fallback BPE tokenizer
21
+ - Mixture of Experts
22
+
23
+ ## Load the model from huggingface:
24
+
25
+ ```
26
+ import torch
27
+ from transformers import AutoTokenizer, AutoModel
28
+
29
+ tokenizer = AutoTokenizer.from_pretrained("RaphaelMourad/Mistral-Peptide-v1-134M", trust_remote_code=True)
30
+ model = AutoModel.from_pretrained("RaphaelMourad/Mistral-Peptide-v1-134M", trust_remote_code=True)
31
+ ```
32
+
33
+ ## Calculate the embedding of a protein sequence
34
+
35
+ ```
36
+ insulin = "MALWMRLLPLLALLALWG"
37
+ inputs = tokenizer(insulin, return_tensors = 'pt')["input_ids"]
38
+ hidden_states = model(inputs)[0] # [1, sequence_length, 256]
39
+
40
+ # embedding with max pooling
41
+ embedding_max = torch.max(hidden_states[0], dim=0)[0]
42
+ print(embedding_max.shape) # expect to be 256
43
+ ```
44
+
45
+ ## Troubleshooting
46
+
47
+ Ensure you are utilizing a stable version of Transformers, 4.34.0 or newer.
48
+
49
+ ## Notice
50
+
51
+ Mistral-Peptide-v1-134M is a pretrained base model for peptide.
52
+
53
+ ## Contact
54
+
55
+ Raphaël Mourad. [email protected]