Skip to content

Tracking: openSSL with asm on arm64 #23913

Closed
Closed
Tracking: openSSL with asm on arm64#23913
Labels
cryptoIssues and PRs related to the crypto subsystem.opensslIssues and PRs related to the OpenSSL dependency.tlsIssues and PRs related to the tls subsystem.

Description

@refack
refack
opened on Oct 26, 2018
  • Version: master
  • Platform: CI arm64
  • Subsystem: test,crypto

https://github.com/nodejs/node/blob/master/test/parallel/test-https-client-get-url.js

Look nasty, hopefully 🤞 it is just a flake

https://ci.nodejs.org/job/node-test-commit-arm/19511/nodes=centos7-arm64-gcc6/testReport/junit/(root)/test/parallel_test_https_client_get_url/

Error Message
fail (1)

Stacktrace
(node:57030) Warning: Setting the NODE_TLS_REJECT_UNAUTHORIZED environment variable to '0' makes TLS connections and HTTPS requests insecure by disabling certificate verification.
events.js:167
      throw er; // Unhandled 'error' event
      ^

Error: 4396790469872:error:1408F119:SSL routines:ssl3_get_record:decryption failed or bad record mac:../deps/openssl/openssl/ssl/record/ssl3_record.c:469:

Emitted 'error' event at:
    at TLSSocket.socketErrorListener (_http_client.js:399:9)
    at TLSSocket.emit (events.js:182:13)
    at TLSSocket._emitTLSError (_tls_wrap.js:600:10)
    at TLSWrap.onerror (_tls_wrap.js:268:11)

/CC @nodejs/testing @nodejs/crypto

Activity

  1. refack
    added
    tlsIssues and PRs related to the tls subsystem.
    cryptoIssues and PRs related to the crypto subsystem.
    testIssues and PRs related to Node.js core tests and test infrastructure.
    opensslIssues and PRs related to the OpenSSL dependency.
    flaky-testIssues and PRs involving tests that fail intermittently in CI.
    on Oct 26, 2018

  2. ryzokuken commented on Oct 27, 2018

    @ryzokuken
    ryzokuken
    on Oct 27, 2018
    Contributor

    @tniessen @bnoordhuis "bad record mac"? That shouldn't happen, right (especially intermittently).

  3. refack commented on Oct 29, 2018

    @refack
    refack
    on Oct 29, 2018
    ContributorAuthor

    test.parallel/test-tls-pfx-authorizationerror

    Error Message
fail (1)
Stacktrace
events.js:167
      throw er; // Unhandled 'error' event
      ^

Error: 281472993497088:error:1408F119:SSL routines:ssl3_get_record:decryption failed or bad record mac:../deps/openssl/openssl/ssl/record/ssl3_record.c:469:

Emitted 'error' event at:
    at TLSSocket._emitTLSError (_tls_wrap.js:600:10)
    at TLSWrap.onerror (_tls_wrap.js:268:11)

  4. refack commented on Oct 30, 2018

    @refack
    refack
    on Oct 30, 2018
    ContributorAuthor

    test.parallel/test-https-client-checkServerIdentity

    Error Message
fail (1)
Stacktrace
/home/iojs/build/workspace/node-test-commit-arm/nodes/ubuntu1604-arm64/test/parallel/test-https-client-checkServerIdentity.js:71
    throw err;
    ^

Error: 281472838037504:error:1408F119:SSL routines:ssl3_get_record:decryption failed or bad record mac:../deps/openssl/openssl/ssl/record/ssl3_record.c:469:
  5. refack
    changed the title [-]investigate: parallel test-https-client-get-url[/-] [+]investigate: "ssl3_get_record:decryption failed or bad record mac" om arm64[/+] on Oct 30, 2018

  6. refack commented on Oct 30, 2018

    @refack
    refack
    on Oct 30, 2018
    ContributorAuthor

    Ref: #23241

  7. refack commented on Oct 30, 2018

    @refack
    refack
    on Oct 30, 2018
    ContributorAuthor

    Ref: #23422

  8. refack commented on Oct 30, 2018

    @refack
    refack
    on Oct 30, 2018
    ContributorAuthor

    Ref: #23261

  9. refack commented on Oct 30, 2018

    @refack
    refack
    on Oct 30, 2018
    ContributorAuthor

    ping @nodejs/platform-arm @nodejs/crypto

    Any recommendation how to workaround this?

  10. bnoordhuis commented on Oct 31, 2018

    @bnoordhuis
    bnoordhuis
    on Oct 31, 2018
    Member

    If it only happens on arm64, it might be worthwhile to turn off assembly for a while on that architecture (./configure --openssl-no-asm) and see if the problem goes away.

  11. refack commented on Oct 31, 2018

    @refack
    refack
    on Oct 31, 2018
    ContributorAuthor

    If it only happens on arm64, it might be worthwhile to turn off assembly for a while

    Done, nodejs/build#1556.
    I'll try to remember to report of outcome...

  12. 49 remaining items

  13. vielmetti commented on Jun 6, 2019

    @vielmetti
    vielmetti
    on Jun 6, 2019
    Contributor

    Thanks @richardlau @rvagg - it's good that the system is stable, so that we can rule out infrastructure issues. However of course I'd always prefer not to see a performance regression.

    Would it be worthwhile to test a new PR to turn asm support back on? And if you get any flakiness, I'm happy to look upstream for some more specialized vendor support and resources to take a look at what's going on.

  14. sam-github commented on Jun 6, 2019

    @sam-github
    sam-github
    on Jun 6, 2019
    Contributor

    @vielmetti Definitely worthwhile, #23913 (comment) points to what would need changing, thanks for looking at this.

  15. vielmetti commented on Jun 11, 2019

    @vielmetti
    vielmetti
    on Jun 11, 2019
    Contributor

    Thanks. The smallest PR I could imagine is in #28180 and the goal I see is to identify any flakiness associated with that one particular commit.

  16. sam-github
    added a commit that references this issue on Jul 25, 2019

    build: re-enable openssl arm for arm64

    16f673e
  17. gntem
    added a commit that references this issue on Jul 27, 2019

    build: re-enable openssl arm for arm64

    b4d1cd5
  18. vikashk1
    added a commit that references this issue on Aug 13, 2019

    Please ignore this msg (#1)

    9af17ad

  19. rvagg commented on Oct 21, 2019

    @rvagg
    rvagg
    on Oct 21, 2019
    Member

    I came across this today in the Jenkins setup, associated with the opening of this issue, invoked for arm64 centos:

        # temporary mesure to evaluate https://github.com/nodejs/node/issues/23913
    export CONFIG_FLAGS="$CONFIG_FLAGS --openssl-no-asm"

    Runs in here: https://ci.nodejs.org/job/node-test-commit-arm

    So I guess even with it re-enabled in #28180, we've still been compiling without asm.

    @sam-github @vielmetti should I just yank it out and see what happens?

  20. sam-github commented on Oct 21, 2019

    @sam-github
    sam-github
    on Oct 21, 2019
    Contributor

    Yes

  21. rvagg commented on Oct 21, 2019

    @rvagg
    rvagg
    on Oct 21, 2019
    Member

    done, we shall see if anything shows up

  22. jasnell commented on Jun 26, 2020

    @jasnell
    jasnell
    on Jun 26, 2020
    Member

    @nodejs/build @nodejs/crypto ... does this issue need to remain open?

  23. rvagg commented on Jun 26, 2020

    @rvagg
    rvagg
    on Jun 26, 2020
    Member

    Have just confirmed the config entry is removed from CI (it was commented out but is now removed), so with #28180 in place and no reported problems since I think we can call this done!

  24. rvagg
    closed this as completedon Jun 26, 2020

  25. sxa commented on Feb 9, 2021

    @sxa
    sxa
    on Feb 9, 2021
    Member

    Initially I saw it only on one specific machine, but since it seems to be happening on all 4 arm64 public CI workers (2 x centos7 + 2 x ubuntu1604).

    FYI @vielmetti @refack this may be related to the issues I was seeing specifically on ThunderX systems unless OpenSSL was build without no-asm Ref adoptium/infrastructure#1897

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    cryptoIssues and PRs related to the crypto subsystem.opensslIssues and PRs related to the OpenSSL dependency.tlsIssues and PRs related to the tls subsystem.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions