You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
working again ... but we're using up so much disk, I've even gone back from 7 days to 5 days retention and we're still at 95% disk. We're going to have to switch to block storage for this
@refack it was node-test-binary-windows 9196 (started to test nodejs/node#13482), probably run until it crashed Jenkins because it is no longer accessible. Output was very similar to your comment there. If this is a pattern it'll be a problem, please let us know if you see it again. Right now, there is no other log file that big on the server.
@rvagg agere that we should move to block storage. It's just so incredibly annoying knowing that jenkins doesn't compress the build logs and these basically stand for the the majority of the space consumption. A gzip post job would do. Perhaps we even do it manually? There is a jenkins plugin that is supposed to to this for us but it obviously doesn't work.
@refack Yeah, it's an alarming thing to see that in the interface, but the issue fixes itself if you wait long enough.
I won't start any more C+L jobs unless the backlog gets into single digits. That should hopefully prevent that from happening too much more today at least.
Activity
[-]Now: Jenkins is a wierd mood[/-][+]Now: Jenkins is in a weird mood[/+]refack commentedon Jun 15, 2017
Ohh the master node is offline:
gibfahn commentedon Jun 15, 2017
I can bring it back online, but we fundamentally need to clean up that disk (or allocate more space).
gibfahn commentedon Jun 15, 2017
Oh wait, I can't bring it back online:
refack commentedon Jun 15, 2017
sad emoji
gibfahn commentedon Jun 15, 2017
@refack Have you tried the node-build IRC channel? You need someone with the
nodejs_build_infrassh key (who can actually get into the machines).refack commentedon Jun 15, 2017
Yep, and #node-dev (posted here last since I thought maybe someone will get an e-mail)
rvagg commentedon Jun 15, 2017
on it
rvagg commentedon Jun 15, 2017
working again ... but we're using up so much disk, I've even gone back from 7 days to 5 days retention and we're still at 95% disk. We're going to have to switch to block storage for this
gibfahn commentedon Jun 15, 2017
Do we know what's using this up? Is it just loads of copies of Node repos? Jenkins artifacts? Something else?
refack commentedon Jun 15, 2017
Maybe it's time to revisit #739, gate the big jobs (or maybe just PR) on first passing linux one, it's also gives a very quick feedback.
joaocgreis commentedon Jun 15, 2017
Found a huge job log file, cctest produced 40Gb of output (stack traces in the beginning). We are back at 50+ Gb free, which is about normal.
rvagg commentedon Jun 16, 2017
we should move to block storage on DO for this now that we can
refack commentedon Jun 16, 2017
Was it https://ci.nodejs.org/job/node-test-binary-windows/9117/RUN_SUBSET=0,VS_VERSION=vs2015,label=win2012r2/
I tried to ping ya'll nodejs/node#13490 (comment)
joaocgreis commentedon Jun 17, 2017
@refack it was node-test-binary-windows 9196 (started to test nodejs/node#13482), probably run until it crashed Jenkins because it is no longer accessible. Output was very similar to your comment there. If this is a pattern it'll be a problem, please let us know if you see it again. Right now, there is no other log file that big on the server.
jbergstroem commentedon Jun 19, 2017
@rvagg agere that we should move to block storage. It's just so incredibly annoying knowing that jenkins doesn't compress the build logs and these basically stand for the the majority of the space consumption. A gzip post job would do. Perhaps we even do it manually? There is a jenkins plugin that is supposed to to this for us but it obviously doesn't work.
joaocgreis commentedon Jul 9, 2017
Another failure of
cctest: https://ci.nodejs.org/job/node-test-binary-windows/9716/RUN_SUBSET=3,VS_VERSION=vs2015,label=win2012r2/ while running for nodejs/node#14140refack commentedon Jul 9, 2017
The golden snippet:
Stack repeats forever.
P.S. @joaocgreis maybe delete the log, it's ~1GB
refack commentedon Jul 16, 2017
Ping ping ping.
Happening again.
refack commentedon Jul 16, 2017
Well it's not the same: The
masternode is ok - https://ci.nodejs.org/computer/(master)/Only one job suck
...
refack commentedon Jul 16, 2017
It probably boils down to https://ci.nodejs.org/job/node-test-binary-arm/ being slow, and the backlog limited to 15 jobs
Trott commentedon Jul 16, 2017
@refack Yeah, it's an alarming thing to see that in the interface, but the issue fixes itself if you wait long enough.
I won't start any more C+L jobs unless the backlog gets into single digits. That should hopefully prevent that from happening too much more today at least.
refack commentedon Jul 17, 2017
Can we learn something for this? Increase the backlog limit? Make Jenkins shut-up about reserving stuff and just say "Pending"?
jbergstroem commentedon Jul 17, 2017
We can increase teh backlog but my experience is that we start failing/bleeding elsewhere :/
refack commentedon Jul 18, 2017
And now something new, subjob are finishing but not propagating their status:
refack commentedon Jul 18, 2017
And this is new... Doing this "Loading" thing, but on "submit" does not create new jobs.
