tool call parser: JSONDecodeError
Running on vLLM 0.12.0:
vllm serve mistralai/Devstral-Small-2-24B-Instruct-2512 --tool-call-parser mistral --enable-auto-tool-choice --tensor-parallel-size 4 --host 0.0.0.0 --port 8000 --served-model-name devstral-small-2 --max-model-len 26214
prompt: "create a program that computes fibonacci numbers in C++"
results in tool parser error:
IServer pid=3372) ERROR 12-18 08:10:56 [mistral_tool_parser.py:386] Error trying to handle streaming tool call.
(APIServer pid=3372) ERROR 12-18 08:10:56 [mistral_tool_parser.py:386] Traceback (most recent call last):
(APIServer pid=3372) ERROR 12-18 08:10:56 [mistral_tool_parser.py:386] File "/home/graemer/.venv/lib/python3.12/site-packages/vllm/entrypoints/openai/tool_parsers/mistral_tool_parser.py", line 225, in extract_tool_calls_streaming
(APIServer pid=3372) ERROR 12-18 08:10:56 [mistral_tool_parser.py:386] tool_call_arr: list[dict] = partial_json_parser.loads(
(APIServer pid=3372) ERROR 12-18 08:10:56 [mistral_tool_parser.py:386] ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=3372) ERROR 12-18 08:10:56 [mistral_tool_parser.py:386] File "/home/graemer/.venv/lib/python3.12/site-packages/partial_json_parser/core/api.py", line 15, in parse_json
(APIServer pid=3372) ERROR 12-18 08:10:56 [mistral_tool_parser.py:386] return parser(ensure_json(json_string, allow_partial, use_fast_fix))
(APIServer pid=3372) ERROR 12-18 08:10:56 [mistral_tool_parser.py:386] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=3372) ERROR 12-18 08:10:56 [mistral_tool_parser.py:386] File "/home/graemer/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/json/init.py", line 346, in loads
(APIServer pid=3372) ERROR 12-18 08:10:56 [mistral_tool_parser.py:386] return _default_decoder.decode(s)
(APIServer pid=3372) ERROR 12-18 08:10:56 [mistral_tool_parser.py:386] ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=3372) ERROR 12-18 08:10:56 [mistral_tool_parser.py:386] File "/home/graemer/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/json/decoder.py", line 338, in decode
(APIServer pid=3372) ERROR 12-18 08:10:56 [mistral_tool_parser.py:386] obj, end = self.raw_decode(s, idx=_w(s, 0).end())
(APIServer pid=3372) ERROR 12-18 08:10:56 [mistral_tool_parser.py:386] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=3372) ERROR 12-18 08:10:56 [mistral_tool_parser.py:386] File "/home/graemer/.local/share/uv/python/cpython-3.12.11-linux-x86_64-gnu/lib/python3.12/json/decoder.py", line 356, in raw_decode
(APIServer pid=3372) ERROR 12-18 08:10:56 [mistral_tool_parser.py:386] raise JSONDecodeError("Expecting value", s, err.value) from None
(APIServer pid=3372) ERROR 12-18 08:10:56 [mistral_tool_parser.py:386] json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)