Add cleanWorkspaceAfterBuild and cleanWorkspaceBuildOutputAfterBuild by andrew-m-leonard · Pull Request #2416 · adoptium/temurin-build

andrew-m-leonard · 2021-01-28T13:43:15Z

Add the facility to clean the build workspace or just the workspace/build/src/build and workspace/target binary output at the end of the build. This provides a large saving of disk space, up to 30Gb for Build Output alone and an additional 6Gb for the whole workspace. The default settings for generated pipelines are:

cleanWorkspaceAfterBuild : false
cleanWorkspaceBuildOutputAfterBuild : true
Additionally, cleanWorkspaceAfterBuild can be specified in the jdk platform configuration if a specific platform requires cleaning only.

Signed-off-by: Andrew Leonard anleonar@redhat.com

AdamBrousseau · 2021-01-29T00:17:21Z

I believe this pr addresses #2296

sxa

I really don't like that find construct ... If this is likely to only affect one file can we just handle it explicitly or perhaps find . -type f -name \*..\*? Also I'd recommend removing the verbosity here - maybe just keep the rm lines instead of the bit printing them all out?

andrew-m-leonard · 2021-01-29T19:28:09Z

I really don't like that find construct ... If this is likely to only affect one file can we just handle it explicitly or perhaps find . -type f -name \*..\*? Also I'd recommend removing the verbosity here - maybe just keep the rm lines instead of the bit printing them all out?

It's any file that has ".." in it, the 2 i've discovered so far:

17:51:00  workspace/build/src/build/windows-x86_64-normal-clientANDserver-release/support/gensrc/java.base/java/nio/_the...marker
17:31:58  workspace/build/src/build/windows-x86_64-normal-server-release/jdk/gensrc/java/nio/_the..

andrew-m-leonard · 2021-01-29T19:29:05Z

I really don't like that find construct ... If this is likely to only affect one file can we just handle it explicitly or perhaps find . -type f -name \*..\*? Also I'd recommend removing the verbosity here - maybe just keep the rm lines instead of the bit printing them all out?

Yeah, I thought the print was useful debug for the moment, but i'll remove

andrew-m-leonard · 2021-01-29T19:30:40Z

I really don't like that find construct ... If this is likely to only affect one file can we just handle it explicitly or perhaps find . -type f -name \*..\*? Also I'd recommend removing the verbosity here - maybe just keep the rm lines instead of the bit printing them all out?

Yes, I like the -type f, good idea

andrew-m-leonard · 2021-01-29T19:32:25Z

@sxa The other solution is not use CleanWs, just use rm -rf on the specific folders?
The logic behind CleanWs using DirectoryScanner is just overkill, especially for what we're doing

sxa · 2021-01-30T13:14:51Z

@sxa The other solution is not use CleanWs, just use rm -rf on the specific folders?

i'm thinking that's probably the best option - keeps it simple, but if we do that let's add a comment pointing to the issue as an explanation of why we're not using cleanWs

andrew-m-leonard · 2021-02-01T17:04:37Z

run tests

adoptopenjdk-github-bot · 2021-02-01T17:07:43Z

🟠 PR TESTER RESULT 🟠

❎ Some pipelines failed or the job was aborted! ❎
See the pipeline-build-check below for more information...

andrew-m-leonard · 2021-02-01T20:09:09Z

run tests

adoptopenjdk-github-bot · 2021-02-01T20:09:37Z

🟠 PR TESTER RESULT 🟠

❎ Some pipelines failed or the job was aborted! ❎
See the pipeline-build-check below for more information...

andrew-m-leonard · 2021-02-01T20:23:27Z

run tests

sxa

Looking good now but I won't formally approve til I have another look over it in the morning (unless two other people want to approve in the meantime!)

Would be good to check a few random machines to see how much space workspace/build-scripts is taking and compare it to the same numbers a week after merging this when it should have taken effect across most of the machines. Should particularly help the scaleway arm32 build systems

Clicked wrong button - as per previous comment not doing full approval til tomorrow :-)

adoptopenjdk-github-bot · 2021-02-01T22:05:19Z

🟢 PR TESTER RESULT 🟢

✅ All pipelines passed! ✅

sxa

Other than the naming which I think it's now confusing I think this is ready

Signed-off-by: Andrew Leonard <anleonar@redhat.com>

andrew-m-leonard · 2021-02-02T12:27:41Z

run tests

sxa

Yep I'll take that - thanks. Clear enough now I reckon :-)

sxa · 2021-02-02T15:23:43Z

ping @karianna as you are currently blocking this

adoptopenjdk-github-bot · 2021-02-02T16:10:30Z

🟠 PR TESTER RESULT 🟠

❎ Some pipelines failed or the job was aborted! ❎
See the pipeline-build-check below for more information...

andrew-m-leonard · 2021-02-02T16:29:51Z

run tests

adoptopenjdk-github-bot · 2021-02-02T20:12:49Z

🟢 PR TESTER RESULT 🟢

✅ All pipelines passed! ✅

AdamBrousseau · 2021-02-05T08:45:19Z

-                                context.cleanWs notFailBuild: true, disableDeferredWipeout: true, deleteDirs: true
+                            // Note: Underlying org.apache DirectoryScanner used by cleanWs has a bug scanning where it misses files containing ".." so use rm -rf instead
+                            // Issue: https://issues.jenkins.io/browse/JENKINS-64779
+                            if (context.WORKSPACE != null && !context.WORKSPACE.isEmpty()) {


i think if (context.WORKSPACE) achieves the same thing.

AdamBrousseau · 2021-02-05T08:45:58Z

+                            // Issue: https://issues.jenkins.io/browse/JENKINS-64779
+                            if (context.WORKSPACE != null && !context.WORKSPACE.isEmpty()) {
+                                context.println "Cleaning workspace files: " + context.WORKSPACE + "/*"
+                                context.sh(script: "rm -rf " + context.WORKSPACE + "/*")


I think this ignores the . hidden directories (eg. .git folder). Which is causing me issues on my win boxes.

andrew-m-leonard marked this pull request as draft January 28, 2021 13:43

M-Davies added enhancement Issues that enhance the code or documentation of the repo in any way jenkins Issues that enhance or fix our jenkins server labels Jan 28, 2021

andrew-m-leonard force-pushed the clearbuild branch 5 times, most recently from f8b6893 to 6b15fe6 Compare January 28, 2021 20:13

andrew-m-leonard force-pushed the clearbuild branch 7 times, most recently from a1bceee to 55e068d Compare January 29, 2021 15:48

andrew-m-leonard marked this pull request as ready for review January 29, 2021 18:43

sxa reviewed Jan 29, 2021

View reviewed changes

Comment thread pipelines/build/common/openjdk_build_pipeline.groovy Outdated

sxa reviewed Jan 29, 2021

View reviewed changes

sxa mentioned this pull request Jan 31, 2021

pipelines: Use variables to remove Windows workspaces #2410

Closed

andrew-m-leonard force-pushed the clearbuild branch 4 times, most recently from 7c60e39 to e0242a8 Compare February 1, 2021 09:47

adoptium deleted a comment from adoptopenjdk-github-bot Feb 1, 2021

AdamBrousseau reviewed Feb 1, 2021

View reviewed changes

Comment thread pipelines/build/common/openjdk_build_pipeline.groovy Outdated

andrew-m-leonard force-pushed the clearbuild branch from 5f5cb31 to 29714a6 Compare February 1, 2021 17:02

andrew-m-leonard requested a review from karianna February 1, 2021 17:04

M-Davies mentioned this pull request Feb 1, 2021

CleanWs at end of build should be runtime configurable #2296

Closed

M-Davies linked an issue Feb 1, 2021 that may be closed by this pull request

CleanWs at end of build should be runtime configurable #2296

Closed

sxa reviewed Feb 1, 2021

View reviewed changes

sxa previously approved these changes Feb 1, 2021

View reviewed changes

sxa reviewed Feb 2, 2021

View reviewed changes

Comment thread pipelines/build/common/build_base_file.groovy Outdated

Add cleanWorkspaceAfterBuild and cleanWorkspaceBuildOutputAfterBuild

50bf26e

Signed-off-by: Andrew Leonard <anleonar@redhat.com>

andrew-m-leonard force-pushed the clearbuild branch from 29714a6 to 50bf26e Compare February 2, 2021 12:27

sxa approved these changes Feb 2, 2021

View reviewed changes

karianna approved these changes Feb 2, 2021

View reviewed changes

karianna merged commit 96068f0 into adoptium:master Feb 2, 2021

Willsparker mentioned this pull request Feb 3, 2021

pipelines: Find a better solution for deleting the _the.. after Windows builds #2205

Closed

AdamBrousseau reviewed Feb 5, 2021

View reviewed changes

sxa assigned andrew-m-leonard Feb 8, 2021

sxa mentioned this pull request Feb 8, 2021

Machines low on disk space: zLinux build machines adoptium/infrastructure#1652

Closed

Conversation

andrew-m-leonard commented Jan 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AdamBrousseau commented Jan 29, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

sxa left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrew-m-leonard commented Jan 29, 2021

Uh oh!

andrew-m-leonard commented Jan 29, 2021

Uh oh!

andrew-m-leonard commented Jan 29, 2021

Uh oh!

andrew-m-leonard commented Jan 29, 2021

Uh oh!

sxa commented Jan 30, 2021

Uh oh!

Uh oh!

andrew-m-leonard commented Feb 1, 2021

Uh oh!

adoptopenjdk-github-bot commented Feb 1, 2021

🟠 PR TESTER RESULT 🟠

Uh oh!

andrew-m-leonard commented Feb 1, 2021

Uh oh!

adoptopenjdk-github-bot commented Feb 1, 2021

🟠 PR TESTER RESULT 🟠

Uh oh!

andrew-m-leonard commented Feb 1, 2021

Uh oh!

sxa left a comment

Choose a reason for hiding this comment

Uh oh!

adoptopenjdk-github-bot commented Feb 1, 2021

🟢 PR TESTER RESULT 🟢

Uh oh!

sxa left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

andrew-m-leonard commented Feb 2, 2021

Uh oh!

sxa left a comment

Choose a reason for hiding this comment

Uh oh!

sxa commented Feb 2, 2021

Uh oh!

adoptopenjdk-github-bot commented Feb 2, 2021

🟠 PR TESTER RESULT 🟠

Uh oh!

andrew-m-leonard commented Feb 2, 2021

Uh oh!

adoptopenjdk-github-bot commented Feb 2, 2021

🟢 PR TESTER RESULT 🟢

Uh oh!

AdamBrousseau Feb 5, 2021

Choose a reason for hiding this comment

Uh oh!

AdamBrousseau Feb 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

andrew-m-leonard commented Jan 28, 2021 •

edited

Loading

AdamBrousseau commented Jan 29, 2021 •

edited

Loading

sxa left a comment •

edited

Loading

AdamBrousseau Feb 5, 2021 •

edited

Loading