Repository navigation
[Bug]: When invoking a local ollama model, it cannot reside in memory permanently. #6584
Description
Self Checks
- I have searched for existing issues search for existing issues, including closed ones.
- I confirm that I am using English to submit this report (Language Policy).
- Non-english title submitions will be closed directly ( 非英文标题的提交将会被直接关闭 ) (Language Policy).
- Please do not modify this template :) and fill in all the required fields.
RAGFlow workspace code commit ID
Latest code
RAGFlow image version
Latest code
Other environment information
Actual behavior
When invoking a local ollama model, it cannot stay resident in memory, even though I have set Environment="OLLAMA_KEEP_ALIVE=24h" in the ollama configuration file. The model reloads with each request.
Expected behavior
No response
Steps to reproduce
Invoking a local ollama modelAdditional information
No response
Activity
dosubot commentedon Mar 27, 2025
Hey @incredibleeeeee, I'm here to assist you with any bugs, questions, or contributions you have for RAGFlow. Let's tackle this issue together!
The issue you're experiencing with the local Ollama model not staying resident in memory, despite setting the
OLLAMA_KEEP_ALIVEenvironment variable, might be related to how thekeep_aliveparameter is handled in theOllamaChatclass. In the current implementation, thekeep_aliveparameter is explicitly set to-1, which is intended to keep the model loaded in memory for continuous interaction without unloading after each request [1].If the
OLLAMA_KEEP_ALIVEenvironment variable is not being respected, it could be due to the code not being configured to read and apply this environment variable. You might want to check if there is any logic in the code that reads theOLLAMA_KEEP_ALIVEenvironment variable and applies it to thekeep_aliveparameter. If not, you may need to modify the code to incorporate this functionality.Additionally, ensure that the environment variable is correctly set and accessible in the context where the RAGFlow application is running. If the issue persists, it might be worth investigating further into the codebase or configuration files to see if there are any overrides or additional settings affecting this behavior.
To continue talking to Dosu, mention @dosu.
Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other
incredibleeeeee commentedon Mar 27, 2025
I have model 1 and model 2 deployed locally. In the latest code version, when I request model 1, it clears the memory of model 2, or when I request model 2, it clears the memory of model 1. However, this issue does not occur in version v0.17.2, where both model 1 and model 2 can stay resident in memory during requests.
incredibleeeeee commentedon Mar 27, 2025
After testing, commenting out "num_ctx": 32768 solves the issue. It may be due to the GPU resource limitations on my local machine. When the context is too large, it can't guarantee that both models stay resident in memory.