Repository navigation
feat: OpenAI-compatible server with seed perturbation routes - #14
Open
yuval-peleg-levy-ai21 wants to merge 2 commits into
Open
yuval-peleg-levy-ai21 wants to merge 2 commits into
yuval-peleg-levy-ai21 wants to merge 2 commits into
Conversation
yuval-peleg-levy-ai21
commented
Sep 24, 2026
yuval-peleg-levy-ai21
commented
- Fix base model test evaluation.
- Allow equal sign for countdown
- Add a toy analysis
- Update title in README
- Add OLMo3 results to random-step vs. gradient-step experiment
- update countdown reward functions for baselines
- Update README.md && Add iterative randopt
- adding results for random search
- fix: FP8-safe seeded perturbation and vLLM compat in worker extension
- feat: OpenAI-compatible server with seed perturbation routes
Seeded noise is now always generated in float32 and added via p.data.copy_(p.data.float() + delta), so perturbation works on Float8_e4m3fn weights (no randn/add kernels for FP8), keeps the parameter storage that NCCL broadcast and CUDA graphs point at, and stops rounding the noise to bf16 before scaling. Quantization scale parameters (weight_scale, scale_inv, input_scale, act_scale, .scale) are skipped by default; PERTURB_SCALES=1 restores the old behaviour. Vision-encoder weights of VL models are skipped unless PERTURB_VISUAL=1. get_ip moved to vllm.utils.network_utils in recent vLLM; import it with a fallback. utils/ becomes a regular package so spawn-mode workers can import the extension class by dotted path. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
randopt_server.py wraps vLLM's OpenAI API server and adds /perturb, /restore, /store_base and /reset, each fanned out to every TP rank through collective_rpc on utils.worker_extn.WorkerExtension. External evaluators (agent loops, tool-calling harnesses) can then switch the served weights between seeds through plain HTTP without restarting vLLM or going through Ray. utils/ is copied next to the vllm package at startup because recent vLLM spawns engine and worker processes with a filtered environment that drops the launcher's sys.path and PYTHONPATH. scripts/test_perturb_restore.py is a smoke test that exercises both undo paths against a live server. README documents the routes and recommends /store_base + /reset for bf16 and FP8 weights, where the additive /restore is exact only up to rounding. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.