Separation Between DEMO Sample and Test

[bernhste]

Hi everyone

This question is related to #3262, as this proposal could help increase the test coverage without copying demo samples.

At the moment, the Demos and the Tests seem to be in a 1:1 relationship. For example a Demo is called "MASTG-DEMO-0022: Uses of Insecure Symmetric Encryption Algorithms in Cipher with semgrep".

The sample implements different weak cipher-init such as:

val cipher = Cipher.getInstance("RC4")
val cipher = Cipher.getInstance("DES")
val cipher = Cipher.getInstance("DESede")
[...]

Using semgrep it is possible to find insecure cipher in non-obfuscated source code or decompiled code. However, we often test obfuscated artifacts. There the API may be called like that: Cipher.getInstance(some.conveluted.function()). If we want to detect this, we would dynamically intercept the OS API in order to spot insecure API usage such as Cipher.getInstance("RC4")

To cover this test, we would have to write a new DEMO with the same sample but another tool (frida). So what about we split the SAMPLE from the TESTS?

At the moment the folder looks like that:

owasp-mastg/demos/MASTG-DEMO-XXXX/
├── MastgTest.kt
├── MastgTest_reversed.java
├── AndroidManifest.xml
├── AndroidManifest_reversed.xml
├── MASTG-DEMO-XXXX.md
├── output.txt
└── run.sh

What about we keep the MD file with in the root folder but move the TESTS to a separate subfolder named after the TOOL used? An example structure for "MASTG-DEMO-0022" would be:

owasp-mastg/demos/MASTG-DEMO-0022/
├── frida
├──── MASTG-DEMO-0022-frida.md
├──── output.txt
├──── script.js
├──── run.sh
├── sample
├──── MastgTest.java
├── semgrep
├──── MASTG-DEMO-0022-semgrep.md
├──── MastgTest_reversed.java
├──── output.txt
├──── run.sh
├── MASTG-DEMO-0022.md

The sample folder would contain the code which implements the weakness. The folders named after the tools, would then contain all the artifacts to run the test. The MD file in said folders would contain the capters Steps, Obervations and Evaluation with respect to the current test.

This would make sense because:

Tests ---use---> Techniques ---use---> Tools:

Theoretically we could also use a techniques folder with different sub-categories, but I think this is a bit over-engineered.

Using this structure it is possible to create a large "body of tests" for the different weaknesses using different techniques and tools (see also #3262). I'm very aware, that this is a large endeavor, but I think the demo structure should encourage people to commit many different tests for the OS APIs.

Please let me know what you think about this.

[bernhste]

Small update: Yesterday I made a small PoC for myself to test how we could use a demo for multiple tests with different tools. And I saw, that DEMO-0001 and DEMO-0002 share the same sample.

So for the PoC I merged them with separate tests for each tool in their subfolders. The UI is just as simple as possible, but I think it could work like that.

Here is the merged DEMO-0001:

And here would be one of the two tests:

Now we could implement a weakness or defense in depth mechanism once and provide multiple tests with different tools.