Too few answers to tell: 0.953 with the skill and 0.968 without, but the scores varied too much to call it.
Not enough draws to conclude at this effect floor. Draws needed: no draw count at these spreads (case decision: the two arms' spreads sum to 0.166, at or above the 0.05 floor).
decision, revision: too few answers to settle the result. Compare task scores
Task score bands: decision (underpowered): with skill 0.86 plus or minus 0.082158; without skill 0.93 plus or minus 0.083666; revision (underpowered): with skill 1 plus or minus 0; without skill 0.975 plus or minus 0.05.
A band is the average score plus or minus the recorded spread. These task bands, not the overall averages, determine the result.
Not enough draws to conclude at this effect floor. Draws needed: no draw count at these spreads (case decision: the two arms' spreads sum to 0.166, at or above the 0.05 floor).
Test details and sourcesHow this result was decided
The reader applies the task rule, with a separated-down task taking precedence. Overall means do not determine the verdict. Bands show the mean plus or minus the recorded spread; point bands can reflect score quantization.
decision: underpowered. With skill 0.86 plus or minus 0.082158; without skill 0.93 plus or minus 0.083666.
adr: not-separated. With skill 1 plus or minus 0; without skill 1 plus or minus 0.
revision: underpowered. With skill 1 plus or minus 0; without skill 0.975 plus or minus 0.05.
Three text-only tasks. Driftproof wrote the suite from the pinned skill description and README. These are not author-supplied tests. External tools, generated media and deployed services were not tested.
The eval account's configuration was read and found clean before and after each run, not during it.
The skill clearly hurt: 0.917 with it and 0.979 without, on 3 test tasks.
decision: lower scores with the skill. Compare task scores
Task score bands: decision (separated-down): with skill 0.75 plus or minus 0.061237; without skill 0.975 plus or minus 0.05.
A band is the average score plus or minus the recorded spread. These task bands, not the overall averages, determine the result.
0.917 with the skill and 0.979 without; on at least one of the 3 test tasks, the two ranges of scores (the average plus or minus one spread) did not overlap, and the gap was at least 0.05.
Test details and sourcesHow this result was decided
The reader applies the task rule, with a separated-down task taking precedence. Overall means do not determine the verdict. Bands show the mean plus or minus the recorded spread; point bands can reflect score quantization.
decision: separated-down. With skill 0.75 plus or minus 0.061237; without skill 0.975 plus or minus 0.05.
adr: not-separated. With skill 1 plus or minus 0; without skill 1 plus or minus 0.
revision: underpowered. With skill 1 plus or minus 0; without skill 0.9625 plus or minus 0.047871.
Three text-only tasks. Driftproof wrote the suite from the pinned skill description and README. These are not author-supplied tests. External tools, generated media and deployed services were not tested.
The eval account's configuration was read and found clean before and after each run, not during it.
