Hugging Face's logo

api-inference documentation

Chat Completion

Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

Chat Completion

Generate a response given a list of messages. This is a subtask of text-generation designed to generate responses in a conversational context.

Recommended models

Using the API

The API supports:

  • Using the chat completion API compatible with the OpenAI SDK.
  • Using grammars, constraints, and tools.
  • Streaming the output
Python
JavaScript
cURL 
from huggingface_hub import InferenceClient

client = InferenceClient(
    "google/gemma-2-2b-it",
    token="hf_***",
)

for message in client.chat_completion(
	messages=[{"role": "user", "content": "What is the capital of France?"}],
	max_tokens=500,
	stream=True,
):
    print(message.choices[0].delta.content, end="")

To use the Python client, see huggingface_hub’s package reference.

API specification

Request

Payload
frequency_penalty number Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far, decreasing the model’s likelihood to repeat the same line verbatim.
logprobs boolean Whether to return log probabilities of the output tokens or not. If true, returns the log probabilities of each output token returned in the content of message.
max_tokens integer The maximum number of tokens that can be generated in the chat completion.
messages* object[] A list of messages comprising the conversation so far.
        content string
        name string
        role* string
        tool_calls object[]
                function* object
                        arguments* unknown
                        description string
                        name* string
                id* integer
                type* string
presence_penalty number Number between -2.0 and 2.0. Positive values penalize new tokens based on whether they appear in the text so far, increasing the model’s likelihood to talk about new topics
seed integer
stop string[] Up to 4 sequences where the API will stop generating further tokens.
stream boolean
temperature number What sampling temperature to use, between 0 and 2. Higher values like 0.8 will make the output more random, while lower values like 0.2 will make it more focused and deterministic. We generally recommend altering this or top_p but not both.
tool_choice unknown One of the following:
         (#1)
                FunctionName* string
         (#2) Possible values: OneOf.
tool_prompt string A prompt to be appended before the tools
tools object[] A list of tools the model may call. Currently, only functions are supported as a tool. Use this to provide a list of functions the model may generate JSON inputs for.
        function* object
                arguments* unknown
                description string
                name* string
        type* string
top_logprobs integer An integer between 0 and 5 specifying the number of most likely tokens to return at each token position, each with an associated log probability. logprobs must be set to true if this parameter is used.
top_p number An alternative to sampling with temperature, called nucleus sampling, where the model considers the results of the tokens with top_p probability mass. So 0.1 means only the tokens comprising the top 10% probability mass are considered.

Some options can be configured by passing headers to the Inference API. Here are the available headers:

Headers
authorization string Authentication header in the form 'Bearer: hf_****' when hf_**** is a personal user access token with Inference API permission. You can generate one from your settings page.
x-use-cache boolean, default to true There is a cache layer on the inference API to speed up requests we have already seen. Most models can use those results as they are deterministic (meaning the outputs will be the same anyway). However, if you use a nondeterministic model, you can set this parameter to prevent the caching mechanism from being used, resulting in a real new query. Read more about caching here.
x-wait-for-model boolean, default to false If the model is not ready, wait for it instead of receiving 503. It limits the number of requests required to get your inference done. It is advised to only set this flag to true after receiving a 503 error, as it will limit hanging in your application to known places. Read more about model availability here.

For more information about Inference API headers, check out the parameters guide.

Response

Output type depends on the stream input parameter. If stream is false (default), the response will be a JSON object with the following fields:

Body
choices object[]
        finish_reason string
        index integer
        logprobs object
                content object[]
                        logprob number
                        token string
                        top_logprobs object[]
                                logprob number
                                token string
        message object
                content string
                name string
                role string
                tool_calls object[]
                        function object
                                arguments unknown
                                description string
                                name string
                        id integer
                        type string
created integer
id string
model string
object string
system_fingerprint string
usage object
        completion_tokens integer
        prompt_tokens integer
        total_tokens integer

If stream is true, generated tokens are returned as a stream, using Server-Sent Events (SSE). For more information about streaming, check out this guide.

Body
choices object[]
        delta object
                content string
                role string
                tool_calls object
                        function object
                                arguments string
                                name string
                        id string
                        index integer
                        type string
        finish_reason string
        index integer
        logprobs object
                content object[]
                        logprob number
                        token string
                        top_logprobs object[]
                                logprob number
                                token string
created integer
id string
model string
object string
system_fingerprint string
< > Update on GitHub

←Automatic Speech Recognition Feature Extraction→