Skip to content

Upgrade Octopus Server template continues after download failure and passes formatting objects to msiexec #1710

Open

Description

@bcullman

Step template

Upgrade Octopus Server

Step version

10

Octopus version

2025.4.10250

Step template parameter inputs

  • InstanceName: OctopusServer
  • InstallPath: C:\Program Files\Octopus Deploy\Octopus
  • SpecificOctopusServerVersionToInstall: 2026.3.15878

What happened

The MSI download failed with:

The request was aborted: Could not create SSL/TLS secure channel.

The step caught the exception but continued as though the download had succeeded. It logged Download completed, stopped the Octopus Server service, and invoked msiexec.

The msiexec process remained nearly idle for approximately three hours. Its command line contained PowerShell formatting-object type names instead of the downloaded MSI path:

msiexec.exe /i Microsoft.PowerShell.Commands.Internal.Format.FormatStartData Microsoft.PowerShell.Commands.Internal.Format.GroupStartData Microsoft.PowerShell.Commands.Internal.Format.FormatEntryData ...

The Octopus Server service remained stopped until the stalled installer was terminated and the existing server version was restored.

Expected behavior:

  • A download failure should terminate the step before the Octopus Server service is stopped.
  • Exception diagnostics should not become part of the returned MSI path.
  • msiexec should receive exactly one valid MSI path.
  • The step should wait for the installer and fail when it returns a nonzero exit code.

Reproduction steps

  1. Configure version 10 of the Upgrade Octopus Server community step to run on an Octopus Server machine.
  2. Set SpecificOctopusServerVersionToInstall to a version different from the installed version.
  3. Cause the MSI download to fail, for example through a TLS, proxy, certificate, or network failure.
  4. Run the step.
  5. Observe that DownloadOctopusServer catches the exception but does not throw it.
  6. Observe that the step still logs Download completed.
  7. Observe that the step proceeds to stop the Octopus Server service.
  8. Inspect the resulting msiexec.exe command line.

The output contamination can also be reproduced safely in Windows PowerShell 5.1 without running an installer:

  1. Execute the same try/catch structure from DownloadOctopusServer.
  2. Replace the WebClient call with a deliberate WebException.
  3. Assign the function output to a variable.
  4. Inspect every returned object's type.

The result contains five PowerShell formatting records followed by the intended temporary-file string:

  • Microsoft.PowerShell.Commands.Internal.Format.FormatStartData
  • Microsoft.PowerShell.Commands.Internal.Format.GroupStartData
  • Microsoft.PowerShell.Commands.Internal.Format.FormatEntryData
  • Microsoft.PowerShell.Commands.Internal.Format.GroupEndData
  • Microsoft.PowerShell.Commands.Internal.Format.FormatEndData
  • System.String

This occurs because the catch block uses:

echo $_.Exception | Format-List -Force

Format-List emits formatting records to PowerShell's success-output stream. The caller therefore receives those records together with $tempFile.

More Information

The relevant code in version 10 is:

catch
{
    Write-Host "Exception occurred"
    echo $_.Exception | Format-List -Force
}

Write-Host "Download completed"
return $tempFile

The caller then continues with:

$tempFile = DownloadOctopusServer $selectedVersionToInstall
StopOctopusServer
InstallOctopusServer $tempFile

The installer is invoked with:

msiexec /i $tempFile /quiet | Out-Null

There are two distinct concerns:

  1. The original TLS negotiation failure

    The precise cause of the TLS failure has not been established. It may depend on the server's WebClient, SCHANNEL, proxy, certificate, or execution-account configuration.

  2. The confirmed template defect

    Regardless of why the download fails, the template swallows the exception and emits formatting records into the function's return value. It then stops the Octopus Server service and invokes msiexec with corrupted input.

The PowerShell 5.1 reproduction produced the same formatting-object type names found in the real msiexec.exe command line, confirming the connection between the catch block and the malformed installer invocation.

After the incident:

  • The original Octopus version, 2025.4.10250, was restored successfully.
  • A separate HTTP request to the Octopus download endpoint returned HTTP 200.
  • BITS subsequently downloaded Octopus.2026.3.15954-x64.msi.
  • The downloaded MSI was 874,090,496 bytes and had a valid Authenticode signature.

This later successful download suggests that the download endpoint was available. It does not establish the exact cause of the earlier WebClient TLS failure.

Suggested correction:

  • Make download exceptions terminating.
  • Do not write exception-formatting objects to the success-output stream.
  • Validate that the downloaded file exists and is a valid MSI before stopping the service.
  • Invoke msiexec with an unambiguous path.
  • Wait for installer completion and validate its exit code.
  • Ensure temporary-file cleanup occurs in a finally block.
  • Add a regression test covering download failure.

Current template source:
https://github.com/OctopusDeploy/Library/blob/master/step-templates/upgrade-octopus-server.json

Activity

  1. bcullman commented on Oct 17, 2026

    @bcullman
    ContributorAuthor

    Follow-up investigation

    Further testing confirmed that the original failure-handling defect is real, but also showed that failing safely is not sufficient for this step to remain usable.

    The original WebClient.DownloadFile() request repeatedly failed after transferring a substantial portion of the installer. The underlying exception reported that the connection was forcibly closed after approximately six minutes.

    We ruled out several possible causes:

    • The target installer is 874,090,496 bytes, approximately half the size of the installer used for the previous successful upgrade.
    • The download endpoint returns the expected content length and supports byte-range requests.
    • Ranges from the beginning, middle, and end of the installer were retrieved successfully.
    • A complete download using sequential 32 MiB ranges produced the expected file length and a valid Octopus Deploy Authenticode signature.

    We then tested a compact version of the step that:

    • Downloads the installer using bounded byte ranges.
    • Retries a failed range from the current downloaded length.
    • Requires HTTP 206 and validates Content-Range.
    • Validates the final file length and Authenticode signature.
    • Removes an incomplete installer if retries are exhausted.
    • Does not stop Octopus Server until the installer has been completely downloaded and validated.
    • Waits for msiexec, checks its exit code, and attempts service recovery after an installation failure.

    During an actual test-server upgrade, the ranged download reached 503,316,480 bytes before one request failed after approximately five and a half minutes. The step retried from the last downloaded byte, completed all 874,090,496 bytes, validated the installer, installed the requested Octopus Server version, restarted the service, removed the temporary installer, and completed successfully.

    The proposed pull request will therefore address both parts of the problem:

    1. Prevent download and installation failures from being swallowed or converted into malformed msiexec arguments.
    2. Replace the unreliable single long-lived download with a compact, validated, resumable ranged download.

    The implementation includes isolated regression tests for range sequencing, retry and resume behavior, invalid responses, length and signature validation, partial-file cleanup, MSI exit codes, service recovery, and the existing template contract.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions