2/3 Trainings on NVIDIA H100 80GB HBM3 get stuck with no chance to resume #964

Activity

1. WhoIsElMasri

   added

   bugSomething isn't workingSomething isn't working

   on Jan 1, 2025
2. UltralyticsAssistant

   added

   HUBUltralytics HUB issuesUltralytics HUB issues

   on Jan 1, 2025
3. UltralyticsAssistant commented on Jan 1, 2025

   

   UltralyticsAssistant

   on Jan 1, 2025

   Member

   More actions

   👋 Hello @WhoIsElMasri, thank you for raising an issue about Ultralytics HUB 🚀! Your feedback helps us improve.

   From your description, it seems you're encountering issues with your trainings getting disconnected and being unable to resume them. If this is a 🐛 Bug Report, could you please provide a minimum reproducible example (MRE) with detailed steps to replicate the issue? Additionally, if you have any screenshots or logs that can help us pinpoint the problem, those would be very helpful.

   If this is a ❓ Question, sharing more details about the training setup, including your dataset, model, and any specific configurations, will help us provide a more thorough response.

   An Ultralytics engineer will review your issue shortly and assist further. Thank you for your patience! 😊
4. pderrenger commented on Jan 2, 2025

   

   pderrenger

   on Jan 2, 2025

   Member

   More actions

   @WhoIsElMasri thank you for reporting this issue and for providing the impacted training IDs! Let’s address this step by step.

   It seems like the training sessions on the NVIDIA H100 80GB are encountering disconnection issues and subsequently failing to resume as expected. Here’s how we can troubleshoot and proceed:

   1. Verify Checkpoints: It's good to confirm that the checkpoints for the affected sessions (e.g., epoch 167) were saved correctly. If you have access to the training details in the Ultralytics HUB, please verify that the checkpoints exist and are accessible.

   2. Resume Training: When resuming training, ensure the process uses the appropriate checkpoint file. If you’re using the Ultralytics HUB, resuming should happen automatically via the Resume Training option. However, as you mentioned encountering an error ("Something went wrong"), this might indicate an issue with the backend or the checkpoint file itself.

   3. Backend Issue: Since the problem persists across multiple sessions, this could be related to a backend issue specific to the environment or GPU setup (NVIDIA H100). Our team will need to investigate further using the training session IDs you provided.

   4. Next Steps:

      • I recommend reaching out through the HUB Feedback option within the platform to ensure this is escalated directly to the engineering team.
      • If reproducible, our team may request additional details or logs to better diagnose the root cause.

   5. Temporary Workaround: If resuming via the HUB continues to fail, an alternative is to download the last checkpoint (if available) and manually resume training using the Python SDK. For example:

      from ultralytics import YOLO
      
      # Load the saved checkpoint
      model = YOLO("path_to_downloaded_checkpoint.pt")
      
      # Resume training
      model.train(resume=True)

   6. Monitoring and Updates: We suggest keeping track of the training progress and reconnecting via the HUB periodically, as intermittent backend issues may resolve themselves.

   We truly appreciate your patience and understanding. The Ultralytics team is committed to resolving this issue swiftly. Let us know if you experience any additional errors or need further assistance! 🚀
5. yogendrasinghx commented on Jan 2, 2025

   

   yogendrasinghx

   on Jan 2, 2025

   Member

   More actions

   Hi @WhoIsElMasri,

   We sincerely apologize for the inconvenience caused and thank you for bringing this issue to our attention. We have successfully reproduced the issue on our end, and your provided model IDs were crucial in identifying the problem.

   I’ve informed the development team, and they will begin working on a fix shortly. I’ll keep you updated and let you know as soon as the issue is resolved.

   Thank you for your patience and understanding!
6. yogendrasinghx

   added

   webRelated to web interface or web functionalityRelated to web interface or web functionality

   on Jan 2, 2025
7. yogendrasinghx

   assigned

   bitcodr

   on Jan 2, 2025
8. github-actions commented on Nov 23, 2025

   

   github-actionsbot

   on Nov 23, 2025 – with GitHub Actions

   More actions

   👋 Hello there! We wanted to give you a friendly reminder that this issue has not had any recent activity and may be closed soon, but don't worry - you can always reopen it if needed. If you still have any questions or concerns, please feel free to let us know how we can help.

   For additional resources and information, please see the links below:

   • Docs: https://docs.ultralytics.com
   • HUB: https://hub.ultralytics.com
   • Community: https://community.ultralytics.com

   Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

   Thank you for your contributions to YOLO 🚀 and Vision AI ⭐
9. github-actions

   added

   StaleStale and schedule for closing soonStale and schedule for closing soon

   on Nov 23, 2025