google

/

gemma-2-9b-it

like 459

This error is similar to what being faced with llama-3-1, the endpoint creation on Hub fails. I had previously deployed the model on dedicated endpoint and it was working fine, now its not working.

Exit code: 1. Reason: /opt/conda/lib/python3.10/site-packages/text_generation_server/models/init.py", line 610, in get_model\n raise NotImplementedError("sharded is not supported for AutoModel")\n\nNotImplementedError: sharded is not supported for AutoModel\n"},"target":"text_generation_launcher","span":{"rank":0,"name":"shard-manager"},"spans":[{"rank":0,"name":"shard-manager"}]}
{"timestamp":"2024-08-07T08:25:52.910377Z","level":"ERROR","fields":{"message":"Shard 3 failed to start"},"target":"text_generation_launcher"}
{"timestamp":"2024-08-07T08:25:52.910402Z","level":"INFO","fields":{"message":"Shutting down shards"},"target":"text_generation_launcher"}
{"timestamp":"2024-08-07T08:25:52.912665Z","level":"INFO","fields":{"message":"Terminating shard"},"target":"text_generation_launcher","span":{"rank":2,"name":"shard-manager"},"spans":[{"rank":2,"name":"shard-manager"}]}
{"timestamp":"2024-08-07T08:25:52.912699Z","level":"INFO","fields":{"message":"Waiting for shard to gracefully shutdown"},"target":"text_generation_launcher","span":{"rank":2,"name":"shard-manager"},"spans":[{"rank":2,"name":"shard-manager"}]}
{"timestamp":"2024-08-07T08:25:52.912698Z","level":"INFO","fields":{"message":"Terminating shard"},"target":"text_generation_launcher","span":{"rank":1,"name":"shard-manager"},"spans":[{"rank":1,"name":"shard-manager"}]}
{"timestamp":"2024-08-07T08:25:52.912712Z","level":"INFO","fields":{"message":"Waiting for shard to gracefully shutdown"},"target":"text_generation_launcher","span":{"rank":1,"name":"shard-manager"},"spans":[{"rank":1,"name":"shard-manager"}]}
{"timestamp":"2024-08-07T08:25:53.012880Z","level":"INFO","fields":{"message":"shard terminated"},"target":"text_generation_launcher","span":{"rank":1,"name":"shard-manager"},"spans":[{"rank":1,"name":"shard-manager"}]}
{"timestamp":"2024-08-07T08:25:53.012883Z","level":"INFO","fields":{"message":"shard terminated"},"target":"text_generation_launcher","span":{"rank":2,"name":"shard-manager"},"spans":[{"rank":2,"name":"shard-manager"}]}
Error: ShardCannotStart

I tried GPU · Nvidia L4 · 4x GPUs · 96 GB, with TGI, on all settings left on default. Can someone else facing this issue too? Any help is appreciated.
Thanks