Skip to content

Support certificate refresh after computer awakes from sleep #1788

Closed
Closed
Support certificate refresh after computer awakes from sleep#1788
Assignees
enocom
Labels
priority: p2Moderately-important priority. Fix may not be included in next release.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.

Description

@carl-mastrangelo
carl-mastrangelo
opened on May 10, 2023

Bug Description

After a few hours of running the cloud SQL proxy, new connections are automatically closed. The proxy prints out error messages on each connection:

2023/05/10 16:29:18 [account-intelligence-dev:us-central1:ogma-prod-1] accepted connection from 127.0.0.1:49983
2023/05/10 16:29:18 [account-intelligence-dev:us-central1:ogma-prod-1] connection aborted - error reading from instance: remote error: tls: bad certificate
2023/05/10 16:29:18 [account-intelligence-dev:us-central1:ogma-prod-1] accepted connection from 127.0.0.1:49985
2023/05/10 16:29:18 [account-intelligence-dev:us-central1:ogma-prod-1] connection aborted - error reading from instance: remote error: tls: bad certificate
2023/05/10 16:29:18 [account-intelligence-dev:us-central1:ogma-prod-1] accepted connection from 127.0.0.1:49987
2023/05/10 16:29:18 [account-intelligence-dev:us-central1:ogma-prod-1] connection aborted - error reading from instance: remote error: tls: bad certificate
2023/05/10 16:29:18 [account-intelligence-dev:us-central1:ogma-prod-1] accepted connection from 127.0.0.1:49989
2023/05/10 16:29:19 [account-intelligence-dev:us-central1:ogma-prod-1] connection aborted - error reading from instance: remote error: tls: bad certificate
2023/05/10 16:29:19 [account-intelligence-dev:us-central1:ogma-prod-1] accepted connection from 127.0.0.1:49991
2023/05/10 16:29:19 [account-intelligence-dev:us-central1:ogma-prod-1] connection aborted - error reading from instance: remote error: tls: bad certificate
2023/05/10 16:29:19 [account-intelligence-dev:us-central1:ogma-prod-1] accepted connection from 127.0.0.1:49993
2023/05/10 16:29:19 [account-intelligence-dev:us-central1:ogma-prod-1] connection aborted - error reading from instance: remote error: tls: bad certificate

Example code (or command)

// paste your code or command here

Stacktrace

No response

Steps to reproduce?

  1. ?
  2. ?
  3. ?
    ...

Environment

  1. OS type and version: Mac OS
  2. Cloud SQL Proxy version (./cloud-sql-proxy --version): cloud-sql-proxy version 2.2.0+darwin.arm64
  3. Proxy invocation command (for example, ./cloud-sql-proxy --enable_iam_login --dir /path/to/dir INSTANCE_CONNECTION_NAME): cloud-sql-proxy urltest-266408:us-central1:ogma-dev-1?port=15432 account-intelligence-dev:us-central1:ogma-prod-1?port=15431

Additional Details

No response

Activity

  1. carl-mastrangelo
    added
    type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.
    on May 10, 2023
  2. blunderbuss-gcf
    assigned
    enocom
    on May 10, 2023

  3. enocom commented on May 11, 2023

    @enocom
    enocom
    on May 11, 2023
    Member

    Does this include letting your machine go to sleep?

  4. enocom commented on May 11, 2023

    @enocom
    enocom
    on May 11, 2023
    Member

    Also what does your database connection string look like (removing private information of course).

  5. enocom
    added
    priority: p2Moderately-important priority. Fix may not be included in next release.
    on May 11, 2023

  6. carl-mastrangelo commented on May 11, 2023

    @carl-mastrangelo
    carl-mastrangelo
    on May 11, 2023
    Author

    Does this include letting your machine go to sleep?

    Yes

    Also what does your database connection string look like (removing private information of course).

    jdbc:postgresql://localhost:15431/postgres

    and

    jdbc:postgresql://localhost:15432/postgres

  7. enocom commented on May 11, 2023

    @enocom
    enocom
    on May 11, 2023
    Member

    I suspect the machine sleeping is the cause of the problem.

    Nonetheless, I thought we'd fixed that here: GoogleCloudPlatform/cloud-sql-go-connector#471.

    I'll investigate and report back.

  8. carl-mastrangelo commented on May 12, 2023

    @carl-mastrangelo
    carl-mastrangelo
    on May 12, 2023
    Author

    Sounds good. I can provide additional log info as needed.

  9. enocom
    added
    type: feature request‘Nice-to-have’ improvement, new feature or different behavior or design.
    and removed
    type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.
    on Jul 19, 2023
  10. enocom
    changed the title [-]TLS Certificate is not renewed on expiration[/-] [+]Support certificate refresh after computer awakes from sleep[/+] on Jul 19, 2023

  11. ForbesLindesay commented on Sep 6, 2023

    @ForbesLindesay
    ForbesLindesay
    on Sep 6, 2023

    @enocom This appears to still be an issue. It makes the cloud-sql-proxy more-or-less unusable for our local development environments.

  12. enocom
    added
    priority: p1Important issue which blocks shipping the next release. Will be fixed prior to next release.
    and removed
    priority: p2Moderately-important priority. Fix may not be included in next release.
    on Sep 6, 2023

  13. enocom commented on Sep 6, 2023

    @enocom
    enocom
    on Sep 6, 2023
    Member

    Good to know. We tried removing the monotonic clock value when calculating refresh times (see GoogleCloudPlatform/cloud-sql-go-connector#471), but that seems not to have helped.

    Perhaps the thing to do here is to check if the current cert is valid before initiating a connection, and blocking on a refresh if the cert is expired.

  14. 6 remaining items

  15. enocom
    added
    priority: p2Moderately-important priority. Fix may not be included in next release.
    type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.
    and removed
    type: feature request‘Nice-to-have’ improvement, new feature or different behavior or design.
    on Nov 28, 2023

  16. enocom commented on Nov 28, 2023

    @enocom
    enocom
    on Nov 28, 2023
    Member

    These bad certificate errors are not being caught by the Proxy (or the Go Connector) given an implementation detail of how TLS 1.3 works:

    In TLS 1.3 the client is the last one to speak in the handshake, so if it causes an error to occur on the server, it will be returned on the client by the first Read, not by Handshake. For example, that will be the case if the server rejects the client certificate.

    https://go.dev/doc/go1.12#tls_1_3

    Right now the Go Connector in particular assumes an expired client certificate (what we have here) will cause the TLS handshake to fail. Instead, the handshake succeeds and then the first read fails. Because it's the read that fails, the Proxy assumes it's a network error instead of a problem with the TLS connection and so never tries to refresh the client certificate.

    This is definitely a bug on our part. I have a fix in the works.

  17. enocom
    added a commit that references this issue on Dec 7, 2023

    fix: ensure cert refresh recovers from sleep

    0a7883b

  18. enocom commented on Dec 7, 2023

    @enocom
    enocom
    on Dec 7, 2023
    Member

    For extra context: https://stackoverflow.com/questions/62459802/tls-1-3-client-does-not-report-failed-handshake-when-client-certificate-verifica

  19. enocom commented on Dec 8, 2023

    @enocom
    enocom
    on Dec 8, 2023
    Member

    To close the loop on v1 not having this problem:

    cloud-sql-proxy/proxy/proxy/client.go

    Line 380 in d1e48b9

    if !isValid(e) || e.cfg.Certificates[0].Leaf.NotAfter.Sub(time.Now().Round(0)) <= refreshCfgBuffer {

    There's a check for the cert's expiration. We'll ship the fix for v2 next week.

  20. enocom
    added a commit that references this issue on Dec 8, 2023

    fix: ensure cert refresh recovers from sleep (#686)

    95671ad

  21. enocom commented on Dec 8, 2023

    @enocom
    enocom
    on Dec 8, 2023
    Member

    We've fixed this in the Go Connector and will release this on Tuesday.

  22. enocom
    closed this as completedon Dec 8, 2023
  23. enocom
    added 4 commits that reference this issue on Mar 10, 2024

    fix: strip monotonic clock reading in cert check

    98701e0

    fix: strip monotonic clock reading in cert check

    845c9b3

    fix: strip monotonic clock reading in cert check

    aace409

    fix: strip monotonic clock reading in cert check (#750)

    6ae33b0

  24. enocom commented on Mar 12, 2024

    @enocom
    enocom
    on Mar 12, 2024
    Member

    FYI we've discovered and fixed another bug where running the Proxy with IAM Authentication would not correctly refresh the certificate.

    A fix for that will go out in #2128 on Thursday.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

priority: p2Moderately-important priority. Fix may not be included in next release.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions