Repository navigation
Should there be a Testing WG? #3872
Description
Would it be useful to have a Testing WG? Things a WG might do:
- OMG fix all those flaky tests.
- Create some guidelines around tests. (For example, when should a file containing multiple tests be split into multiple files?)
- Maybe work with the Build WG on the issue of buildbot failures, although I suspect the Build WG is all over that problem. (But just in case: Is there a way to test for buildbot failures? Automate notification to relevant parties so they can be fixed promptly? Is there a way to automate retry when a buildbot fails? Maybe a Test WG can be given limited privileges on the CI infrastructure such that the tedious/mechanistic elements of dealing with buildbot failures can be delegated to them?)
- Maybe see about porting as much of the Python test harness code to Node as is sensible.
Anyone interested?
Activity
Trott commentedon Nov 17, 2015
Oh, and if the Smoke Test WG never got off the ground, maybe there would be interest from a Testing WG to work on citgm.
jasnell commentedon Nov 17, 2015
Thus far @thealphanerd has been working on evolving citgm and has been making great progress but it would always be great to have additional people. I'm not sure we need yet another WG as much as we just need more people working on it. There have been a ton of test fixes landed here lately and it's great to see.
cjihrig commentedon Nov 17, 2015
I think putting smoke testing under a more general testing WG would make sense.
jasnell commentedon Nov 17, 2015
Ref: #2317
MylesBorins commentedon Nov 17, 2015
Maybe something akin to Tooling?
jasnell commentedon Nov 17, 2015
I do think a Tooling WG would be very interesting
MylesBorins commentedon Nov 17, 2015
Seems like a lot of technical challenges to various wg's could fall in here.
Although it may overlap with @nodejs/build a bit
cjihrig commentedon Nov 17, 2015
I agree. The line between build and tooling would be very blurry.
jasnell commentedon Nov 17, 2015
build tends to focus specifically on the build toolchain and CI. The kind of tooling I'm thinking would include tools like Rod's iojs-tools, citgm, eslint updates, that sort of thing. Thinking about it tho, that doesn't really put enough focus on the testing aspect. hmm
jbergstroem commentedon Nov 17, 2015
@jasnell people in build are pretty diverse in how time is spent. Sitting close to all vm's you tend to look at/attempt to fix failed tests, improve tooling (both for the build environment, jenkins and node in general), security, releases and so on. For me, a WG responsible for tests and conformance would make most sense. Stuff like this would suit perfectly in there.
jasnell commentedon Nov 17, 2015
Right on. OK, I'm on board :) let's do it!
On Nov 17, 2015 2:35 PM, "Johan Bergström" notifications@github.com wrote:
joaocgreis commentedon Nov 17, 2015
@Trott regarding your initial comment, we do have a tool to send us an e-mail when buildbots fail, needless to say we had a lot of activity last week. We are tracking the failures in nodejs/build#232 , please call it to our attention if you see any random disconnect. Thanks for the great work solving flaky tests!
orangemocha commentedon Nov 18, 2015
This sounds like a good idea to me. In the modern days' philosophy of "devs write the tests", I think there is something to be missed from having a dedicated group of people focused on how to improve the quality of the product through testing.
Certainly there is some overlap with the current Build WG responsibilities (especially on the test infra) but that doesn't need to be a problem. And if collaborators want to help improve the test infrastructure, they are welcome to do so regardless of which working groups they belong to 😄
I do think this might need to become the responsibility of all collaborators. If a testing WG helps getting traction on flaky tests, that's great. Please join the discussion at nodejs/build#248
Luckily, the buildbot disconnects seem to be fixed.
Adding a few things to the list of things this group might work on:
13 remaining items
Trott commentedon Jan 2, 2016
I've put together a draft of WG formation docs like a charter etc. Feedback welcome, of course. (Would weekly meetings be too often? Do we need to meet at all?)
Moreover, we should probably start Doing The Work before asking to be ratified, so maybe the next step is to put together a Doodle poll to figure out when people can meet (or to decide that meetings isn't how this WG is going to work).
mscdex commentedon Jan 2, 2016
I'm not sure how much time I'd be able to devote in/to a Testing WG, I just really like to see green CI statuses when there are no failures so that's why I had been trying to fix flaky tests recently :-)
santigimeno commentedon Jan 2, 2016
I would be indeed interested. Thanks for asking :)
Trott commentedon Jan 3, 2016
Someone with owner privs want to add santigimeno to nodejs/testing?
Trott commentedon Jan 3, 2016
I've set up a Doodle poll to try to figure out what might be a good time to have our first meeting via Google Hangout.
http://doodle.com/poll/cccu446299gae5d5http://doodle.com/poll/c24vvkga3zfrrhn8 /cc @nodejs/testing @santigimenoorangemocha commentedon Jan 4, 2016
Invited santigimeno to nodejs/testing.
orangemocha commentedon Jan 4, 2016
@Trott I filled the doodle, though it didn't show which time zone it is on. In the time proposals page, you need to enable time zone support, I think.
It might also be better to give more advance notice for the meeting dates so that everyone has the chance to respond and agree on a time.
Trott commentedon Jan 5, 2016
@orangemocha Alas, the "allow more advance notice" thing is hard to argue with. I'm going to cancel this Doodle poll and set up another one with dates slightly further out.
Trott commentedon Jan 5, 2016
OK, attempt number 2 at a poll for a meeting time. Meeting times are all at least one week out from now and I enabled time zone support so it will hopefully show you the time based on your time zone. I also tried to have a reasonable spread of times to avoid a "all these times are in the middle of the night in my time zone" issues.
Here's the link, please fill it out! Thanks! http://doodle.com/poll/c24vvkga3zfrrhn8
Trott commentedon Jan 7, 2016
Poll closed! Only time that was available for every single respondent is the time we're going with:
Friday, January 15, 10AM Pacific Time
More soon...
Hey, if someone with org privs wants to set up a testing repo, I could move this sort of organizational stuff in there and close this issue. :-)
orangemocha commentedon Jan 8, 2016
https://github.com/nodejs/testing
jasnell commentedon Jan 8, 2016
@Trott ... are you able to send out a regular calendar invite for the meeting?
Trott commentedon Jan 8, 2016
@jasnell When I set up a Google Hangout On Air, I believe it should send a calendar invite...
Trott commentedon Jan 8, 2016
@orangemocha Thanks! I'll shift everything over to that repo and close this issue.