Practical Guide to Running Nemotron-Nano-9B-v2-Japanese with vLLM and Integrating it into Your Custom Application via an Open...

Introduction Recently, an article on Qiita titled "Running Nemotron-Nano-9B-v2-Japanese with llama.cpp" gained significant attention. That article required manual building of llama.cpp and GGUF conversion as a workaround for Ollama's zero-division bug, but this article introduces a simpler and more practical approach: "vLLM + OpenAI-compatible API." Using vLLM eliminates the need for GGUF conversion, avoids Ollama-related issues, and allows for direct reuse of existing code. The entire process, from server startup to API integration, can be completed with just three commands. Why vLLM? Direct safetensors loading : Eliminates the hassle of GGUF conversion. Models can be used immediately by simply specifying the model file at server startup. Standard OpenAI-compatible API : By setting base_url to http://localhost:8000/v1 , existing OpenAI SDK code works out-of-the-box. NVIDIA proprietary architecture support : Natively supports the "nemotron_h hybrid architecture" of Mamba-2 + Transforme