Repository navigation
Support for running multiple runFlank tasks with configs #141
Description
I noticed that if I try to run multiple runFlank tasks for different configs, only the config for the last runFlank task is used
For example, if I have a build.gradle file that does this:
...
fladle {
smokeTest {
testTargets = ['annotation SmokeTest']
}
expandedSmokeTest {
testTargets = ['annotation ExpandedSmokeTest']
}
}
and then try to run
./gradlew :app:runFlankSmokeTests :app:runFlankExpandedSmokeTests
The output would be something like:
> Task :app:execFlankSmokeTest
version: v20.06.1
revision: ...
AndroidArgs
gcloud:
...
test-targets:
- annotation ExpandedSmokeTest
...
> Task :app:execFlankExpandedSmokeTest
version: v20.06.1
revision: ...
AndroidArgs
gcloud:
...
test-targets:
- annotation ExpandedSmokeTest
Digging into the gradle log, I think this might be caused by the yaml file being overwritten because I see these lines sequentially:
> Task :app:writeConfigPropsSmokeTest
> Task :app:writeConfigPropsExpandedSmokeTest
Activity
pawelpasterz commentedon Jun 25, 2020
I think this is due to how additional configs are created. For primitive values it's fine but here we have list. Then its reference is assigned to variable and then it is overwritten (but it's rather guess than assumption, @runningcode WDYT?)
fladle/buildSrc/src/main/java/com/osacky/flank/gradle/FlankGradleExtension.kt
Line 101 in 359f0fe
runningcode commentedon Jun 26, 2020
Yes, that's exactly it. The Yaml file is always written to the same directory. They should be written to a task specific directory.
To be honest, I've never thought about this use case, but I think it doesn't make sense because you will not get a unified test and/or error report. I would recommend creating a new config called
smokeTestAndExpandedSmokeTest.Can you tell me more about this use case and why
smokeTestAndExpandedSmokeTestdoesn't work here?If we wanted to fix this we would also need to fix that, all the outputs are also being written to the same directory. They will also need config specific outputs. This could break existing implementations. Flank clears certain output directories before writing to them so the outputs of one test would overwrite the outputs of the other.
SamThompson commentedon Jun 26, 2020
I did some testing and it looks like when annotations are used in a filter in
am instrument -e, it takes the intersection of the set of tests that have the specified annotations. I ended up adding theExpandedSmokeTestannotation to everywhere I haveSmokeTestand just went with running the expanded smoke test config.The usecase was mainly for convenience since it seemed like it was the easiest way to run both sets of tests. Just wanted to flag this since I was initially confused by the behavior.
bootstraponline commentedon Jun 27, 2020
I’m happy to make changes in flank to support this use case.
runningcode commentedon Jun 27, 2020
That sounds very nice! It's always good to understand the use case and see if this feature is worth maintaining and building.
There are two possible paths if i understand this correctly:
What do you think @bootstraponline ?
bootstraponline commentedon Jun 27, 2020
This seems like the best starting point, I'll discuss with the team.
jan-goral commentedon Jul 3, 2020
After call with team we agreed is impossible to fix issue only on flank side. Even if we adjust annotation filters to support union, this will not prevent from overriding results when running multiple executions so flade should handle output directory for that case. Anyway current test targets implementation in flank behave exactly like
adb shell am instrument ... - e ...filters. If we add support for union, there will be difference.bootstraponline commentedon Jul 3, 2020
That makes sense. I think looking into updating fladle to support multiple runs is the next step then.
runningcode commentedon Jul 11, 2020
I've created #147, #148 and #149 to outline what needs to happen from the fladle plugin side to make this possible. Feel free to pick up any of that work.
Will there also be some changes needed to flank itself in order to merge the test reports?
bootstraponline commentedon Jul 12, 2020
Yeah, it's not clear what the best way to do this is. I'll chat with the team.
pawelpasterz commentedon Jan 11, 2021
Hey @runningcode is it possible this issue was fixed with #194 ?
runningcode commentedon Jan 11, 2021
Correct, that should have fixed it. Going to close this out. Feel free to re-open if it still happens.