Repository navigation
[PRIORITY] Persistent failures on Windows #1005
Description
See https://jenkins-iojs.nodesource.com/job/iojs+any-pr+multi/215/nodes=iojs-win2012r2/console for sample output, consistent with 2008 build and consistent across runs at the moment too.
10 failures in total. These have gone uncaught for a little while because of a combination of:
- Jenkins
- Jenkins configuration problems/mistakes (mainly on my part)
- The fact that
test-cistill doesn't work on any platform so we have to resort to more hacky means of making enough tests run - Errors in vcbuild.bat that have meant we haven't been running the full suite, fixed in build: improve vcbuild.bat #998
So we've been getting lots of blue when they really should have been red; so totally off everyone's radar.
I'm can't assess the severity of these failures at a glance, nor can I see a single common theme that would point to something to address. When I have time I'll go back and find a run that had these passing so we can start a manual bisect at least.
_test-child-process-stdio-big-write-end fixed in #1008_
=== release test-child-process-stdio-big-write-end ===
Path: parallel/test-child-process-stdio-big-write-end
events.js:141
throw er; // Unhandled 'error' event
^
Error: write ENOTSUP
at exports._errnoException (util.js:734:11)
at Socket._writeGeneric (net.js:668:26)
at Socket._writev (net.js:682:8)
at doWrite (_stream_writable.js:293:12)
at clearBuffer (_stream_writable.js:389:5)
at onwrite (_stream_writable.js:338:7)
at Socket.WritableState.onwrite (_stream_writable.js:88:5)
at WriteWrap.afterWrite (net.js:765:12)
Command: c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\Release\iojs.exe c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\test\parallel\test-child-process-stdio-big-write-end.js
=== release test-timers-first-fire ===
Path: parallel/test-timers-first-fire
timer fired in -0.19568000000000296
assert.js:87
throw new assert.AssertionError({
^
AssertionError: Timer fired early
at null._onTimeout (c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\test\parallel\test-timers-first-fire.js:11:10)
at Timer.listOnTimeout (timers.js:88:15)
Command: c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\Release\iojs.exe c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\test\parallel\test-timers-first-fire.js
=== release test-child-process-double-pipe ===
Path: parallel/test-child-process-double-pipe
grep stdin write 7
echo exit
grep stdin write 18
Command: c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\Release\iojs.exe c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\test\parallel\test-child-process-double-pipe.js
--- TIMEOUT ---
=== release test-child-process-exit-code ===
Path: parallel/test-child-process-exit-code
Command: c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\Release\iojs.exe c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\test\parallel\test-child-process-exit-code.js
--- TIMEOUT ---
=== release test-child-process-spawnsync ===
Path: parallel/test-child-process-spawnsync
sleep started
Command: c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\Release\iojs.exe c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\test\parallel\test-child-process-spawnsync.js
--- TIMEOUT ---
=== release test-child-process-spawnsync-input ===
Path: parallel/test-child-process-spawnsync-input
Command: c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\Release\iojs.exe c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\test\parallel\test-child-process-spawnsync-input.js
--- TIMEOUT ---
=== release test-http-curl-chunk-problem ===
Path: parallel/test-http-curl-chunk-problem
dd command: "c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\Release\iojs.exe" "c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\test\fixtures\create-file.js" "c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\test\tmp.11\big" 10485760
Server running at http://localhost:8080
making curl request
Command: c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\Release\iojs.exe c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\test\parallel\test-http-curl-chunk-problem.js
--- TIMEOUT ---
=== release test-process-kill-null ===
Path: parallel/test-process-kill-null
Command: c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\Release\iojs.exe c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\test\parallel\test-process-kill-null.js
--- TIMEOUT ---
_test-pipe-head fixed in #1008_
=== release test-pipe-head ===
Path: sequential/test-pipe-head
Command: c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\Release\iojs.exe c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\test\sequential\test-pipe-head.js
--- TIMEOUT ---
=== release test-stdin-from-file ===
Path: sequential/test-stdin-from-file
"c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\Release\iojs.exe" "c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\test\fixtures\echo-close-check.js" < "c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\test\fixtures\stdin.txt"
c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\test\sequential\test-stdin-from-file.js:37
if (err) throw err;
^
Error: Command failed: C:\Windows\system32\cmd.exe /s /c ""c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\Release\iojs.exe" "c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\test\fixtures\echo-close-check.js" < "c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\test\fixtures\stdin.txt""
assert.js:87
throw new assert.AssertionError({
^
AssertionError: false == true
at Server.<anonymous> (c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\test\fixtures\echo-close-check.js:17:5)
at Server.g (events.js:257:16)
at emitNone (events.js:67:13)
at Server.emit (events.js:163:7)
at net.js:1184:12
at process._tickCallback (node.js:350:11)
at ChildProcess.exithandler (child_process.js:716:12)
at emitTwo (events.js:87:13)
at ChildProcess.emit (events.js:169:7)
at maybeClose (child_process.js:984:16)
at Process.ChildProcess._handle.onexit (child_process.js:1057:5)
Command: c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\Release\iojs.exe c:\workspace\iojs+any-pr+multi\nodes\iojs-win2012r2\test\sequential\test-stdin-from-file.js
Activity
rvagg commentedon Feb 28, 2015
ping @iojs/platform-windows, this needs fairly urgent attention from anyone capable of digging in and figuring these out.
rvagg commentedon Feb 28, 2015
Two of these have been fixed by #1008
Last complete
test-simplerun I can find is: https://jenkins-iojs.nodesource.com/job/iojs+any-pr+multi/nodes=iojs-win2012r2/54/console which is back at January 12th. i.e. so it seems that we've been happily seeing passes in CI for Windows builds since we've been "releasing" and none of us have noticed that it's been mainlytest-messagethat's been running. This is entirely because it's been assumed thatvcbuild.bat ... test-simple test-messageis supposed to bundle both of them in to one, but unfortunately vcbuild.bat never had that properly working anyway. Fixed now in #998.That means two things:
/cc @iojs/build
rvagg commentedon Mar 1, 2015
I had limited success in reproducing many of these. Running them locally on a Windows machine and even running them on the exact same server that is giving us the failures via Jenkins, I can only make either 4 or 3 tests fail. The block of timeouts seem to be somehow Jenkins-specific.
Can other Windows users verify against 1.v HEAD and report back?
vcbuild x64 release nosign test-simple(addtest-messageat the end to do exactly the same as Jenkins).domenic commentedon Mar 1, 2015
I get these failures: https://gist.github.com/domenic/ad191da152fc6632fa32 Unsure what to make of them compared to yours above.
I guess the first one, maybe others, is due to not having the openssl binary installed. I'll go check my prereqs and report back if anything else changes.
rvagg commentedon Mar 1, 2015
Install git bash and you'll get what you need. Google "git windows" and download and install that one.
domenic commentedon Mar 1, 2015
Yeah I have Git bash but when I try to run .bat files from within it all hell breaks loose.
rvagg commentedon Mar 1, 2015
You should be able to run vcbuild.bat from cmd.exe but git bash comes with curl and other Unix utils that can go in your PATH. I think that's an install option though.
domenic commentedon Mar 1, 2015
OK yeah, getting the same failures minus the first one (the openssl one).
seishun commentedon Mar 5, 2015
I have 32-bit Windows at home so I ran
vcbuild release nosign test-simpleagainst b72fa03, The result is similar to @domenic's: https://gist.github.com/seishun/ebba8198423259444892After deleting tmp.0 manually I re-ran parallel/test-fs-access and it passed.
rvagg commentedon Mar 6, 2015
git bisect says that @indutny's changes in b968623 / #840 introduced the
test-stdin-from-filefailuresI'm still having trouble reproducing the timeout failures, perhaps something odd about the way the processes are being spawned via Jenkins is feeding in to that.
piscisaureus commentedon Mar 6, 2015
@rvagg Before that patch Server.fd would be undefined on windows, after the patch the value is -1. But that test has already been fixed: abd3ecf
rvagg commentedon Mar 6, 2015
right, sorry, I'm barking up the wrong
treetest, I need to narrow this down but there's definitely more failures after that merge for Windows than before but cross referencing with what's already fixed is tough.so, ignore my comment for now!
rmg commentedon Mar 6, 2015
I haven't looked at the tests or the Jenkins config, so forgive me if this isn't helpful..
The combination of Jenkins and Windows immediately brings to mind the problems I've had with build console logs being garbled or out of order because it is non-blocking. Depending on how the scripts are wired up, that could introduce timing issues.
rvagg commentedon Mar 9, 2015
Restarting Jenkins seems to have fixed the timeouts, I was playing with running Jenkins from cmd rather than as a service but soon discovered it was simply a matter of needing to restart. I also did a full cleanout of the build workspace so that potentially could have helped too, I really don't know. Jenkins
These are the remaining persistent failures on both 2008 and 2012.
If you want to contribute but don't feel confident enough to actually find and fix the underlying problem then running a
git bisectwould be a great help to narrow down the commits where these were introduced (I suppose one at a time, best not to assume they are all related). I don't know agoodto start from though so you'll have to go fishing.Build with
vcbuild x64 release nosign, then test withRelease\iojs.exe test\parallel\test-http-content-length.js(etc.).9 remaining items
Fishrock123 commentedon Mar 19, 2015
@mathiask88 thanks for pointing that out. Given that is a changelog commit I believe it was committed in error from dev testing. #1198
ping @rvagg
rvagg commentedon Mar 19, 2015
I restarted jenkins again on the 2008 machines (actually the 2012 ones as well) and we're back to parity between test runs on both: https://jenkins-iojs.nodesource.com/job/iojs+any-pr+multi/331/nodes=iojs-win2008r2/console
So now it's just test-regress-GH-io-1068 and test-net-reconnect-error left.
Fishrock123 commentedon Mar 19, 2015
As of dd37fb4 there might not be any more persistent windows timeouts: #1198
(Which leaves only the dreaded test-regress-GH-io-1068)
rvagg commentedon Mar 19, 2015
improving, but we now have test-child-process-stdout-flush-exit in the mix occasionally, not sure if this is a new failure but it's shown up a few times in recent runs.
mathiask88 commentedon Mar 19, 2015
Maybe it helps, but on my windows machine
test-regress-GH-io-1068works.Fishrock123 commentedon Mar 22, 2015
The windows build(s) build are green as of #1233 cheers all! 🍻
Of course, special thanks to @indutny for his hard work on lots of this. :)
indutny commentedon Mar 22, 2015
🍺 to everyone! Now to figure out some odd leak.
rvagg commentedon Mar 22, 2015
woo! thanks all
piscisaureus commentedon Mar 22, 2015
Thank you very much @indutny !
artnikpro commentedon Sep 13, 2015
Same thing on OS X Yosemite
nikolmarku1 commentedon Sep 30, 2015
Same thing on OS X Yosemite could we open this issue
indutny commentedon Sep 30, 2015
Perhaps you have a server listening on 8080 port?
Fishrock123 commentedon Oct 6, 2015
I'm going to lock this. This is historical only. Please report new issues if they arise in new issue thread. :)