Add experimental Computational Storage Drive mode by Emilio597 · Pull Request #188 · MoatLab/FEMU

Emilio597 · 2026-05-26T08:32:15Z

Summary

This PR adds an experimental Computational Storage Drive (CSD) mode to FEMU as femu_mode=4.

The implementation is derived from CEMU, but adapted to FEMU's current codebase and configuration model. CSD-specific logic is kept under hw/femu/csd/ where possible, with only small hooks in common FEMU/NVMe paths.

This PR intentionally avoids requiring a CEMU-specific Linux kernel, FDMFS, or a fixed VM image.

What is included

Add femu_mode=4 for CSD mode
Add CSD QOM parameters:
- fdm_size
- nr_cu
- nr_thread
- time_slice
- context_switch_time
- csf_runtime_scale
Route normal CSD NVMe read/write through the BBSSD FTL path
Add AFDM operations:
- allocate/deallocate
- read/write
- NVM-to-AFDM copy
Add CEMU-style program lifecycle commands:
- load/unload
- activate/deactivate
Add shared-library CSF execution
Add optional uBPF support, disabled by default:
- ./femu-compile.sh --enable-csd-ubpf
- ./femu-compile.sh --enable-csd-ubpf=/path/to/ubpf-cemu
Add Memory Range Set management using the NVMe Computational Programs command set style:
- MRS create/delete
- RSID returned through CQE Dword 0 for create
Add guest-side passthrough tests under tests/femu-csd/
Add run-csd.sh following FEMU script style

What is intentionally not included

Kernel-side FDMFS and CEMU-specific guest kernel changes
VM freezing / virtual clock changes
multi-CSD/P2P experiments

Validation

Build/static checks:

./femu-compile.sh
ninja -C build-femu qemu-system-x86_64
make -C tests/femu-csd

Manual KVM guest end-to-end validation can be reproduced with a normal Linux guest image. No CEMU-specific kernel, FDMFS, or fixed VM image is required.

Start a CSD VM:

cd build-femu
./run-csd.sh

Copy the passthrough tests into the guest and build them there:

cd tests/femu-csd
make
scp -P 8080 -r . root@localhost:/root/femu-csd
ssh -p 8080 root@localhost
cd /root/femu-csd
make clean && make

Run the core CSD command smoke tests:

./csd-passthru /dev/nvme0n1 smoke
./csd-passthru /dev/nvme0n1 smoke-so /home/<user>/FEMU/tests/femu-csd/csd-vadd.so
./csd-passthru /dev/nvme0n1 smoke-so-all /home/<user>/FEMU/tests/femu-csd/csd-original-kernels.so
./csd-passthru /dev/nvme0n1 smoke-mrs /home/<user>/FEMU/tests/femu-csd/csd-vadd.so
./csd-passthru /dev/nvme0n1 vadd-example /home/<user>/FEMU/tests/femu-csd/csd-vadd.so
./csd-passthru /dev/nvme0n1 sync-breakdown /home/<user>/FEMU/tests/femu-csd/csd-vadd.so 4096 16
./csd-passthru /dev/nvme0n1 indirect-vadd /home/<user>/FEMU/tests/femu-csd/csd-vadd.so
./csd-passthru /dev/nvme0n1 benchmark-kernels /home/<user>/FEMU/tests/femu-csd/csd-vadd.so /home/<user>/FEMU/tests/femu-csd/csd-original-kernels.so 1
./csd-passthru /dev/nvme0n1 bench 4096 32
./csd-passthru /dev/nvme0n1 bench 65536 16

Optional uBPF validation requires building FEMU with uBPF support:

./femu-compile.sh --enable-csd-ubpf=/path/to/ubpf-cemu
cd tests/femu-csd
make bpf

Then, inside the guest:

./csd-passthru /dev/nvme0n1 smoke-ubpf /home/<user>/FEMU/tests/femu-csd/csd-vadd.bpf.o 0
./csd-passthru /dev/nvme0n1 smoke-ubpf /home/<user>/FEMU/tests/femu-csd/csd-vadd.bpf.o 1

The KVM guest end-to-end validation covered:

AFDM smoke
shared-library CSF execution
original CEMU-style kernels: vadd, knn, sql, grep, lz4
FDMFS-free MRS create/delete and execute by RSID
NVM-to-AFDM copy
sync breakdown: NVM-to-AFDM, exec, AFDM read
indirect vadd smoke
compact benchmark smoke
optional uBPF interpreter/JIT path when built with uBPF support

Copilot

Pull request overview

This PR introduces an experimental Computational Storage Drive (CSD) mode (femu_mode=4) to FEMU, including host-side computational program lifecycle handling and guest-side passthrough tooling to exercise the new vendor/admin commands without requiring a CEMU-specific kernel/FDMFS setup.

Changes:

Add a new FEMU controller mode (femu_mode=4) with CSD-specific NVMe admin/I/O command handling under hw/femu/csd/.
Add build/config plumbing for optional uBPF support (Meson option + femu-compile.sh flag) and a run-csd.sh launcher script.
Add guest-side passthrough tests and sample kernels under tests/femu-csd/, plus README documentation updates.

Reviewed changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
`tests/femu-csd/README.md`	Documents guest-side passthrough tests and how to run CSD validations.
`tests/femu-csd/Makefile`	Builds passthrough tool and sample shared-library/BPF kernels.
`tests/femu-csd/femu-csd-kernel.h`	Defines the shared CSD kernel ABI struct used by sample kernels.
`tests/femu-csd/csd-vadd.c`	Sample shared-library kernels (direct + indirect vadd).
`tests/femu-csd/csd-vadd.bpf.c`	Sample uBPF kernel.
`tests/femu-csd/csd-passthru.c`	Guest-side NVMe passthrough tool exercising CSD admin/I/O commands.
`tests/femu-csd/csd-original-kernels.cc`	Ports of selected original CEMU kernels into a shared library.
`scripts/meson-buildoptions.sh`	Adds configure-wrapper options for enabling/disabling FEMU CSD uBPF.
`README.md`	Adds CSD mode documentation and updates the feature table and tree.
`meson.build`	Adds Meson options/deps wiring for optional uBPF support.
`meson_options.txt`	Introduces `femu_csd_ubpf` and `femu_csd_ubpf_path` options.
`hw/femu/nvme.h`	Adds CSD controller params, mode constant, and `admin_cmd_cqe` hook.
`hw/femu/nvme-io.c`	Treats CSD like BBSSD/ZNSSD in CQ completion processing path selection.
`hw/femu/nvme-admin.c`	Routes unknown admin opcodes to `ext_ops.admin_cmd_cqe()` when present.
`hw/femu/meson.build`	Builds and links `csd/csd.c` and wires in optional uBPF dep.
`hw/femu/femu.c`	Registers the CSD extension and adds QOM properties for CSD params.
`hw/femu/csd/csd.h`	Adds CSD command structs and private definitions.
`hw/femu/csd/csd.c`	Implements CSD mode: AFDM, MRS, program load/activate, exec, QoS groups, optional uBPF.
`hw/femu/backend/dram.c`	Treats CSD like BBSSD/ZNSSD/NoSSD for DRAM backend addressing logic.
`femu-scripts/run-csd.sh`	New script to run a CSD-configured VM.
`femu-scripts/femu-copy-scripts.sh`	Copies `run-csd.sh` alongside other FEMU helper scripts.
`femu-scripts/femu-compile.sh`	Adds `--enable-csd-ubpf[...]` configuration support.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+    FemuCsdState *csd = csd_state(n);
+    NvmeCsdProgramActivationCmd *activation =
+        (NvmeCsdProgramActivationCmd *)cmd;
+    uint16_t pind = activation->pind;
+    uint8_t sel = activation->sel;
+    FemuCsdProgram *program;


+        qemu_mutex_lock(&csd->lock);
+        id = csd->next_rsid++;
+        if (id == 0) {
+            csd->next_rsid = 1;
+            id = csd->next_rsid++;
+        }
+        while (g_hash_table_contains(csd->mrs, GUINT_TO_POINTER(id))) {
+            id = csd->next_rsid++;
+            if (id == 0) {
+                csd->next_rsid = 1;
+                id = csd->next_rsid++;
+            }
+        }


+    qemu_mutex_lock(&csd->lock);
+    id = csd->next_group_id++;
+    if (id == 0) {
+        csd->next_group_id = 1;
+        id = csd->next_group_id++;
+    }
+    group->id = id;
+    g_hash_table_insert(csd->groups, GUINT_TO_POINTER(id), group);
+    qemu_mutex_unlock(&csd->lock);


+    if (runtime) {
+        req->reqlat += runtime;
+        req->expire_time += runtime;
+    }


+    if (!status) {
+        req->cqe.n.result = result > UINT32_MAX ? UINT32_MAX : result;
+    }


+    program->module = g_module_open(path, G_MODULE_BIND_LOCAL);
+    if (!program->module) {
+        femu_err("CSD: failed to load shared library %s: %s\n", path,
+                 g_module_error());
+        return NVME_INVALID_FIELD | NVME_DNR;
+    }
+
+    if (!g_module_symbol(program->module, symbol, &fn) || !fn) {
+        femu_err("CSD: failed to find shared library symbol %s: %s\n", symbol,
+                 g_module_error());
+        csd_program_unload(program);
+        return NVME_INVALID_FIELD | NVME_DNR;
+    }
+
+    program->shared_lib_fn = (FemuCsdSharedLibFn)fn;
+    return NVME_SUCCESS;


+    if (pind == 0 || psize > UINT32_MAX || loff > psize ||
+        numb > psize - loff) {
+        return NVME_INVALID_FIELD | NVME_DNR;
+    }


+if femu_csd_ubpf_path != ''
+  femu_csd_ubpf_lib_path = femu_csd_ubpf_path / 'build/lib/libubpf.a'
+  femu_csd_ubpf_lib = cc.find_library('ubpf',
+                                      dirs: femu_csd_ubpf_path / 'build/lib',
+                                      required: get_option('femu_csd_ubpf'))
+  femu_csd_ubpf = declare_dependency(
+    include_directories: include_directories(femu_csd_ubpf_path / 'vm/inc',
+                                             femu_csd_ubpf_path / 'build/vm'),
+    dependencies: femu_csd_ubpf_lib)
+  emulator_link_args += femu_csd_ubpf_lib_path


+                             'bbssd/bb.c', 'bbssd/ftl.c', 'csd/csd.c',
+                             'lib/pqueue.c', 'lib/rte_ring.c',
+                             'backend/dram.c'))
+system_ss.add(when: 'CONFIG_FEMU_CSD_UBPF', if_true: femu_csd_ubpf)


+        --enable-csd-ubpf=*)
+            UBPF_PATH="${arg#*=}"
+            FEMU_CONFIGURE_OPTS="${FEMU_CONFIGURE_OPTS} --enable-femu-csd-ubpf -Dfemu_csd_ubpf_path=${UBPF_PATH}"
+            ;;


huaicheng · 2026-05-28T16:47:47Z

CSD emulation is a great addition to FEMU, and CEMU is a very nice work, thank you for upstreaming it!

shoaibCS · 2026-10-01T06:06:23Z

Hi @Emilio597 — thanks again for this patch.

We're talking to people who have contributed to FEMU, as part of our NSF POSE work at Virginia Tech, to get some insight into that experience.

Would you have 15-20 minutes for a quick chat? You can reach any of us — shoaibasif@vt.edu, huaicheng@cs.vt.edu, or hansenidden@vt.edu — and we'll work around your schedule.

Thanks!

Emilio597 added 10 commits May 25, 2026 11:35

hw/femu: add experimental CSD mode

7f585cb

tests/femu-csd: add CSD passthrough tests and run script

92b38a0

hw/femu: add CSD shared library execution

7181c40

hw/femu: route CSD I/O through SSD FTL

0d9b3f4

hw/femu: add CSD program lifecycle commands

2bfc150

hw/femu: use CSD program lifecycle for tests

0a684d6

hw/femu: check CSD NVM copy against FTL map

be56d81

tests/femu-csd: expand shared library and latency coverage

a0a9735

tests/femu-csd: add optional ubpf and lz4 coverage

afdefcc

hw/femu: add CSD MRS and passthrough examples

838e53e

huaicheng requested review from Copilot, huaicheng and wangyuyue May 27, 2026 05:45

Copilot started reviewing on behalf of huaicheng May 28, 2026 16:20

View session

Copilot AI reviewed May 28, 2026

View reviewed changes

huaicheng merged commit eb01bb7 into MoatLab:master May 28, 2026
5 checks passed

Navigation Menu

Repository navigation

Add experimental Computational Storage Drive mode - #188

Add experimental Computational Storage Drive mode#188
huaicheng merged 10 commits into
MoatLab:masterfrom
Emilio597:cemu-csd-minimal

Emilio597 commented May 26, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

huaicheng commented May 28, 2026

Uh oh!

shoaibCS commented Oct 1, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

Emilio597 commented May 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What is included

What is intentionally not included

Validation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

huaicheng commented May 28, 2026

Uh oh!

shoaibCS commented Oct 1, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Emilio597 commented May 26, 2026 •

edited

Loading