Skip to content

Unable to delete Instance snapshot in "Error' state on stopped Instances #11673

Closed
Bug
#11687
Closed
Unable to delete Instance snapshot in "Error' state on stopped Instances#11673
Bug
#11687
Assignees
weizhouapache
Milestone
4.20.2

Description

@rajujith
rajujith
opened on Sep 18, 2025

problem

When instance snapshots fail to create on KVM due to various reasons like ' Guest agent is not responding' or 'internal snapshots of a VM with pflash based firmware require QCOW2 nvram format' the instance snapshot is created in CloudStack in 'Error' state. This can be deleted while the instance is 'Running' but when the instance is in 'Stopped' state, the instance snapshot deletion fails since the snapshot was never created in KVM.

On Management server:

2025-09-18 08:04:33,742 DEBUG [c.c.v.s.VMSnapshotManagerImpl] (Work-Job-Executor-14:[ctx-d7ea3fce, job-59/job-60, ctx-c90903e0]) (logid:f78a6e3a) Failed to create vm snapshot: [VMSnapshot {"id":2,"name":"i-2-6-VM_VS_20250918080427","uuid":"f5cadba9-b1ff-4436-8d34-0fb6768fcd29","vmId":6}] due to: Creating VM snapshot: i-2-6-VM_VS_20250918080427 failed due to  Create VM snapshot failed due to org.libvirt.LibvirtException: Operation not supported: internal snapshots of a VM with pflash based firmware require QCOW2 nvram format com.cloud.utils.exception.CloudRuntimeException: Creating VM snapshot: i-2-6-VM_VS_20250918080427 failed due to  Create VM snapshot failed due to org.libvirt.LibvirtException: Operation not supported: internal snapshots of a VM with pflash based firmware require QCOW2 nvram format
        at org.apache.cloudstack.storage.vmsnapshot.DefaultVMSnapshotStrategy.takeVMSnapshot(DefaultVMSnapshotStrategy.java:179)

KVM agent:

2025-09-18 08:05:49,263 DEBUG [utils.script.Script] (AgentRequest-Handler-1:[]) (logid:) Executing command [/usr/bin/qemu-img snapshot -d i-2-6-VM_VS_20250918080427 /mnt/a925cc72-8e60-3c29-b18c-0b8444608bd0/2296a6ae-87d5-4da0-89db-34e9f424576c ].
2025-09-18 08:05:49,274 WARN  [utils.script.Script] (AgentRequest-Handler-1:[]) (logid:) Execution of process [76592] for command [/usr/bin/qemu-img snapshot -d i-2-6-VM_VS_20250918080427 /mnt/a925cc72-8e60-3c29-b18c-0b8444608bd0/2296a6ae-87d5-4da0-89db-34e9f424576c ] failed.
2025-09-18 08:05:49,274 DEBUG [utils.script.Script] (AgentRequest-Handler-1:[]) (logid:) Exit value of process [76592] for command [/usr/bin/qemu-img snapshot -d i-2-6-VM_VS_20250918080427 /mnt/a925cc72-8e60-3c29-b18c-0b8444608bd0/2296a6ae-87d5-4da0-89db-34e9f424576c ] is [1].
2025-09-18 08:05:49,274 WARN  [utils.script.Script] (AgentRequest-Handler-1:[]) (logid:) Process [76592] for command [/usr/bin/qemu-img snapshot -d i-2-6-VM_VS_20250918080427 /mnt/a925cc72-8e60-3c29-b18c-0b8444608bd0/2296a6ae-87d5-4da0-89db-34e9f424576c ] encountered the error: [1].

Manual deletion attempt :

/usr/bin/qemu-img snapshot -d i-2-6-VM_VS_20250918080427 /mnt/a925cc72-8e60-3c29-b18c-0b8444608bd0/2296a6ae-87d5-4da0-89db-34e9f424576c
qemu-img: Could not delete snapshot 'i-2-6-VM_VS_20250918080427': snapshot not found

mysql> select * from vm_snapshots where id=2\G
*************************** 1. row ***************************
                 id: 2
               uuid: f5cadba9-b1ff-4436-8d34-0fb6768fcd29
               name: i-2-6-VM_VS_20250918080427
       display_name: sn2
        description: sn2
              vm_id: 6
         account_id: 2
          domain_id: 1
service_offering_id: 13
   vm_snapshot_type: DiskAndMemory
              state: Error
             parent: NULL
            current: NULL
       update_count: 4
            updated: 2025-09-18 08:05:49
            created: 2025-09-18 08:04:28
            removed: NULL

versions

4.20.1
KVM

The steps to reproduce the bug

  1. Deploy a Windows instance where guest agent is not installed or configure vtpm.
  2. Create an instance snapshot with memory, it should fail.
  3. Stop the instance
  4. Delete the instance snapshot in 'Error' state.

What to do about it?

Avoid creating an instance snapshot in 'Error' state or allow deletion.

Activity

  1. rajujith
    added the
    Bug
    issue type on Sep 18, 2025
  2. DaanHoogland
    added this to the 4.20.2 milestone on Sep 18, 2025
  3. DaanHoogland
    added this to Apache CloudStack BugFest - Issueson Sep 18, 2025

  4. DaanHoogland commented on Sep 18, 2025

    @DaanHoogland
    DaanHoogland
    on Sep 18, 2025
    Contributor

    This should be the fix. 958535ed3eaf.zip

    @qumarissword69 , could you create this as a pull request, please?

  5. DaanHoogland
    moved this to Todo in Apache CloudStack BugFest - Issueson Sep 18, 2025
  6. weizhouapache
    self-assigned this
    on Sep 18, 2025

  7. DaanHoogland commented on Sep 18, 2025

    @DaanHoogland
    DaanHoogland
    on Sep 18, 2025
    Contributor

    This should be the fix. 958535ed3eaf.zip

    @qumarissword69 , could you create this as a pull request, please?

    no you can’t , you’re as dumb as what…

  8. weizhouapache commented on Sep 18, 2025

    @weizhouapache
    weizhouapache
    on Sep 18, 2025
    Member

    @rajujith
    can you share some logs before the logs you shared ?

    I notice that ACS checks if vm snapshot exist before removing it

    cloudstack/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtDeleteVMSnapshotCommandWrapper.java

    Lines 100 to 110 in 8eb6ddd

    String qemuImgPath = Script.getExecutableAbsolutePath("qemu-img");
    List<String[]> commands = new ArrayList<>();
    commands.add(new String[]{qemuImgPath, "snapshot", "-l", sanitizeBashCommandArgument(rootDisk.getPath())});
    commands.add(new String[]{Script.getExecutableAbsolutePath("tail"), "-n", "+3"});
    commands.add(new String[]{Script.getExecutableAbsolutePath("awk"), "-F", " ", "{print $2}"});
    commands.add(new String[]{Script.getExecutableAbsolutePath("grep"), "^" + sanitizeBashCommandArgument(cmd.getTarget().getSnapshotName()) + "$"});
    String qemu_img_snapshot = Script.executePipedCommands(commands, 0).second();
    if (qemu_img_snapshot == null) {
    logger.info("Cannot find snapshot " + cmd.getTarget().getSnapshotName() + " in file " + rootDisk.getPath() + ", return true");
    return new DeleteVMSnapshotAnswer(cmd, cmd.getVolumeTOs());
    }

  9. weizhouapache
    assigned
    weizhouapache
    and unassigned
    weizhouapache
    on Sep 18, 2025
  10. weizhouapache
    linked a pull request that will close this issueKVM: fix delete vm snapshot if it does not exist with a Stopped vm #11687on Sep 19, 2025
  11. weizhouapache
    moved this from Todo to In Review in Apache CloudStack BugFest - Issueson Sep 19, 2025

  12. rajujith commented on Sep 24, 2025

    @rajujith
    rajujith
    on Sep 24, 2025
    ContributorAuthor

    @weizhouapache After upgrading the environment to 4.22-SNAPSHOT for another PR testing, the issue is not observed anymore.

  13. weizhouapache commented on Sep 24, 2025

    @weizhouapache
    weizhouapache
    on Sep 24, 2025
    Member

    fixed by #11687

  14. weizhouapache
    closed this as completedon Sep 24, 2025
  15. github-project-automation
    moved this from In Review to Done in Apache CloudStack BugFest - Issueson Sep 24, 2025
  16. DaanHoogland
    removed this from Apache CloudStack BugFest - Issueson Sep 29, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Bug

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions