Repository navigation
Unstable Training #29
Description
When I switch to the areal v0.3.3 and run the asearcher_local experiment, I got very unstable training and training cannot continue due to very high rollout time costs.
Below are the hyperparameters I used to run the experiment and the logs. (The blue curves are relatively stable and the green and grey curves are unstable and are created after I switched to areal v0.3.3)
epochs=10, 4p1t1+4p1t1, batch_size=128, max_concurrent_rollouts=48, mem_fraction_static=0.80
Activity
samjia2000 commentedon Sep 26, 2025
@wenjunli-0 Hi, this is because AReaL has updated the gradient computation part in recent PR. To make sure ASearcher training is stable, the
gradient_clippingcoefficient should be set to 0.2. Below is the training curve:wenjunli-0 commentedon Oct 14, 2025
I ran a few new asearcher-local experiments recently and they are normal in the beginning. However, around 200-250 steps, the rollout time starts to explode and the training hangs. I observe the grpo_actor/score/av, number_output_token/avg, num_search_queries/avg and etc, they all seem to be normal and do not explode.
What will cause such a phenomenon, and how should I fix it?
Taewhoo commentedon Dec 8, 2025
@wenjunli-0 Hi, have you found a solution to this issue (rollout spike around steps 200-250) ?
wenjunli-0 commentedon Dec 9, 2025
I think it's because of the dynamic filtering, which filters out zero-advantage samples and thus causes the spike (e.g., needs more time to collect non-zero-advantage samples when the model can answer almost all of the questions in the training data).
Taewhoo commentedon Dec 9, 2025
That makes sense. It's difficult to reach the performance of ASearcher-Local-7B using AReal v0.3.3, and perhaps it's because I can only train up to around 220 steps. Maybe I should revert to the legacy version of AReal, as mentioned in issue#2?
wenjunli-0 commentedon Dec 10, 2025
Yes, honestly, I also had difficulty with fully reproducing their experiments and results shown in the paper.