MoshiRAG deviating from Gemma information provided!!!!
Hello,
I am trying to run MoshiRAG using Gemma 3:4b open source model running on Ollama. The query which I ask is sent Gemma but the information Gemma provides is not used by Moshi. It deviates a lot from the factual information which the Gemma sources for it.
Is this a common issue in the code blocks? Or what should be the fix for this problem?
I would appreciate the help.
Thank you
Several checkpoints you may want to look into
- whether the reference encoder server receives the reference text?
- whether the reference embedding is received by Moshi?
- how much time it takes from Moshi's prediction of the token to the moment Moshi receives the reference embedding?
For debugging, you may try to hard-code a fixed reference text. That is, no matter what reference is generated, the Moshi model always use the fixed hard-coded information for its retrieval-augmented generation. That could help you isolate potential issues from the reference generation/encoding side and the Moshi side.
Thank you for your comment.
I narrowed down the issue with the VRAM. 32 GB of VRAM is I guess not enough to run the complete model. As I was running Ollama on i9-14 gen CPU, it was taking 10-15 sec for the answer and till then Moshi is finished with the task. When I switched to API calls for the retrival, it works well but after few questions when cache is build up and VRAM is full, it lags and speak in a distorted voice.
That seems to be a reasonable troubleshoot. When MoshiRAG fails to receive reference within around 2 seconds, it usually just generates response directly, where hallucination happens due to the limited knowledge of the small 7B model. As for the lagging issue, you could look into the terminal output -- to maintain a smooth chatting experience, Moshi should process and generate an audio frame every 80ms. If the model runs slower than this, distorted and lagging output happens.