VictorSanh commited on
Commit
2c031da
1 Parent(s): 87d5d88
Files changed (1) hide show
  1. README.md +33 -0
README.md CHANGED
@@ -218,6 +218,39 @@ print(generated_texts)
218
 
219
  </details>
220
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
221
  # Model optimizations
222
 
223
  If your GPU allows, we first recommend loading (and running inference) in half precision (`torch.float16` or `torch.bfloat16`).
 
218
 
219
  </details>
220
 
221
+ **Text generation inference**
222
+
223
+ Idefics2 is integrated into [TGI](https://github.com/huggingface/text-generation-inference) and we host API endpoints for both `idefics2-8b` and `idefics2-8b-chatty`.
224
+
225
+ Multiple images can be passed on with the markdown syntax (`![](IMAGE_URL)`) and no spaces are required before and after. The dialogue utterances can be separated with `<end_of_utterance>\n` followed by `User:` or `Assistant:`. `User:` is followed by a space if the following characters are real text (no space if followed by an image).
226
+
227
+ <details><summary>Click to expand.</summary>
228
+
229
+ ```python
230
+ from text_generation import Client
231
+
232
+ API_TOKEN="<YOUR_API_TOKEN>"
233
+ API_URL = "https://api-inference.huggingface.co/models/HuggingFaceM4/idefics2-8b-chatty"
234
+
235
+ # System prompt used in the playground for `idefics2-8b-chatty`
236
+ SYSTEM_PROMPT = "System: The following is a conversation between Idefics2, a highly knowledgeable and intelligent visual AI assistant created by Hugging Face, referred to as Assistant, and a human user called User. In the following interactions, User and Assistant will converse in natural language, and Assistant will do its best to answer User’s questions. Assistant has the ability to perceive images and reason about them, but it cannot generate images. Assistant was built to be respectful, polite and inclusive. It knows a lot, and always tells the truth. When prompted with an image, it does not make up facts.<end_of_utterance>\nAssistant: Hello, I'm Idefics2, Huggingface's latest multimodal assistant. How can I help you?<end_of_utterance>\n"
237
+ QUERY = "User:![](https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg)Describe this image.<end_of_utterance>\nAssistant:"
238
+
239
+ client = Client(
240
+ base_url=API_URL,
241
+ headers={"x-use-cache": "0", "Authorization": f"Bearer {API_TOKEN}"},
242
+ )
243
+ generation_args = {
244
+ "max_new_tokens": 512,
245
+ "repetition_penalty": 1.1,
246
+ "do_sample": False,
247
+ }
248
+ generated_text = client.generate(prompt=SYSTEM_PROMPT + QUERY, **generation_args)
249
+ generated_text
250
+ ```
251
+
252
+ </details>
253
+
254
  # Model optimizations
255
 
256
  If your GPU allows, we first recommend loading (and running inference) in half precision (`torch.float16` or `torch.bfloat16`).