From c0d7c41198e42ae905dc5615995eafbfb15dfb1f Mon Sep 17 00:00:00 2001 From: the-themis-benchmarks Date: Thu, 3 Jun 2021 01:04:54 -0700 Subject: [PATCH] revise --- README.md | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index d5eea89..2c3d876 100644 --- a/README.md +++ b/README.md @@ -580,7 +580,7 @@ all the data files of our evaluation for inspection.* ** In the following, we take `Monkey` as a tool and `ActivityDiary-1.1.8-debug-#118.apk` as a target bug to illustrate how to replicate the whole evaluation, and how to validate the artifact if you do not have enough resources/time** -*To replicate the whole evaluation*: +[**To replicate the whole evaluation**]: **Step 1.** run `Monkey` on `ActivityDiary-1.1.8-debug-#118.apk` for 6 hours and repeat this process for 5 runs. **This step will take 30 hours to finish because of 5 runs of testing on one emulator**. We do not recommend to run @@ -600,6 +600,18 @@ Here, * `-o ../monkey-results/` specifies the output directory of testing results * `--monkey` specifies the testing tool +[**If you do not have enough resources/time, we recommend you to try running 1-2 tools on 1-2 bugs at your will to validate +the artifact**]: + +For example, in **Step 1**, we recommend you to shorten the testing time (e.g., use `--time 1h` for 1 hour or `--time 30m` for 30 minutes). +Thus, you can use the following command (**this step will take 2 hours to finish because of 2 runs of testing on one emulator**). + +``` +python3 themis.py --no-headless --avd Android7.1 --apk ../ActivityDiary/ActivityDiary-1.1.8-debug-#118.apk -n 1 --repeat 2 --time 1h -o ../monkey-results/ --monkey +``` + +You can follow the similar instructions in **Step 2** to check the bug finding results. + **Step 2.** When the testing terminates, you can inspect whether the target bug was found or not in each run, how long does it take to find the bug, and how many times the bug was found by using the command below. ``` @@ -711,18 +723,6 @@ row `ActivityDiary`'s `#118` for the value `0/5`), while `Monkey` found the targ In this way, you can validate the data in **Table 3** (`*` indicates the tool finds the bug in at least one run) and **Table 4** in the accepted paper (the breakdown of which bugs were successfully in how many runs). -*If you do not have enough resources/time, we recommend you to try running 1-2 tools on 1-2 bugs at your will to validate -the artifact.* - -For example, in **Step 1**, we recommend you to shorten the testing time (e.g., use `--time 1h` for 1 hour or `--time 30m` for 30 minutes). -Thus, you can use the following command (**this step will take 2 hours to finish because of 2 runs of testing on one emulator**). - -``` -python3 themis.py --no-headless --avd Android7.1 --apk ../ActivityDiary/ActivityDiary-1.1.8-debug-#118.apk -n 1 --repeat 2 --time 1h -o ../monkey-results/ --monkey -``` - -You can follow the similar instructions in **Step 2** to check the bug finding results. - ### Notes