Commit Graph

38 Commits

Author SHA1 Message Date
George Karpenkov ef9eb3e6cd [analyzer] [tests] Add an option for showing statistics after running tests.
Do not show statistics by default.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@338323 91177308-0d34-0410-b5e6-96231b3b80d8
2018-07-30 23:01:47 +00:00
George Karpenkov cfa1c8b413 [analyzer] [tests] Style fixes for testing harness.
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@338322 91177308-0d34-0410-b5e6-96231b3b80d8
2018-07-30 23:01:20 +00:00
Mikhail R. Gadelha 3d25876adf Fix a (possible) division by zero check in the CmpRuns script
I missed updating the check in r333375

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@333529 91177308-0d34-0410-b5e6-96231b3b80d8
2018-05-30 11:17:55 +00:00
Mikhail R. Gadelha cea74394ed Introduces --stats-only option to only show changes in statistics.
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@333375 91177308-0d34-0410-b5e6-96231b3b80d8
2018-05-28 15:40:39 +00:00
George Karpenkov a701b12944 [analyzer] [tests] Update CmpRuns to write to stdout correctly in multithreaded environment
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@325070 91177308-0d34-0410-b5e6-96231b3b80d8
2018-02-13 23:36:01 +00:00
George Karpenkov 9f84a49dbd [analyzer] [tests] Fixing an error after non-atomic cherry-pick
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@324762 91177308-0d34-0410-b5e6-96231b3b80d8
2018-02-09 18:48:31 +00:00
George Karpenkov 3cc1cf93fc [analyzer] [tests] [NFC] Remove a fragile tightly-coupled component emulating parser output
...when we can just use the real parser instead.

Differential Revision: https://reviews.llvm.org/D43098

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@324759 91177308-0d34-0410-b5e6-96231b3b80d8
2018-02-09 18:39:47 +00:00
George Karpenkov c58ab38fd0 [analyzer] [tests] Show function name in CmpRuns output
Combined with enabled flag for stable filenames, this greatly simplifies
finding the offending report.

Differential Revision: https://reviews.llvm.org/D42831

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@324362 91177308-0d34-0410-b5e6-96231b3b80d8
2018-02-06 17:22:09 +00:00
George Karpenkov 7a1006f6f3 [analyzer] [tests] Fix crash in SATestBuild.py
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@324114 91177308-0d34-0410-b5e6-96231b3b80d8
2018-02-02 18:27:14 +00:00
George Karpenkov de212366a3 [analyzer] [tests] [NFC] Remove dead code from CmpRuns
Indeed, "CHANGE" is not a thing yet, and we should probably not carry
around dead code which does not do anything apart from confusing the
reader.

Differential Revision: https://reviews.llvm.org/D42819

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@324027 91177308-0d34-0410-b5e6-96231b3b80d8
2018-02-01 22:40:01 +00:00
George Karpenkov ef79198e68 [analyzer] [tests] Add an option to show the histogram of path differences between the analyzer runs
Differential Revision: https://reviews.llvm.org/D42778

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@324021 91177308-0d34-0410-b5e6-96231b3b80d8
2018-02-01 22:25:18 +00:00
George Karpenkov ea016449cc [analyzer] [tests] Show the number of removed/added bug reports
Differential Revision: https://reviews.llvm.org/D42718

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@323941 91177308-0d34-0410-b5e6-96231b3b80d8
2018-02-01 02:38:42 +00:00
George Karpenkov 86395f102e [Analyzer] Make testing scripts flake8 compliant
Differential Review: https://reviews.llvm.org/D38213

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@314692 91177308-0d34-0410-b5e6-96231b3b80d8
2017-10-02 17:59:12 +00:00
George Karpenkov d578df7f78 [Analyzer] Refactor analyzer testing scripts.
- Exporting needed function for future reuse.
 - Idiomatic python: using with `file as f` instead of `try/finally`.
 - Fixing some indentation issues.
 - No need to reinvent python `multiprocessing.getCPUCount()`
 - Removing a function parameter which is always the same under all invocations.
 - Adding some docstrings.

Differential Revision: https://reviews.llvm.org/D38156

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@313949 91177308-0d34-0410-b5e6-96231b3b80d8
2017-09-22 01:41:16 +00:00
George Karpenkov b937f758ff [Analyzer] Remove dead code from CmpRuns.py.
Differential Revision: https://reviews.llvm.org/D38003

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@313923 91177308-0d34-0410-b5e6-96231b3b80d8
2017-09-21 21:47:13 +00:00
Gabor Horvath 61a690a04f [analyzer] Bug identification
This patch adds hashes to the plist and html output to be able to identfy bugs
for suppressing false positives or diff results against a baseline. This hash
aims to be resilient for code evolution and is usable to identify bugs in two
different snapshots of the same software. One missing piece however is a 
permanent unique identifier of the checker that produces the warning. Once that
issue is resolved, the hashes generated are going to change. Until that point
this feature is marked experimental, but it is suitable for early adoption.

Differential Revision: http://reviews.llvm.org/D10305 

Original patch by: Bence Babati!


git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@251011 91177308-0d34-0410-b5e6-96231b3b80d8
2015-10-22 11:53:04 +00:00
Ted Kremenek 99bb39afb4 [analyzer] Apply whitespace cleanups by Honggyu Kim.
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@246978 91177308-0d34-0410-b5e6-96231b3b80d8
2015-09-08 03:50:52 +00:00
Gabor Horvath 7d70c91050 [Static Analyzer] Make the paths relative from the project root when generating reference results in the test suite.
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@241706 91177308-0d34-0410-b5e6-96231b3b80d8
2015-07-08 18:39:31 +00:00
Gabor Horvath 939c4bfc23 [Static Analyzer] Minor improvements to SATest.
Differential Revision: http://reviews.llvm.org/D10812


git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@241073 91177308-0d34-0410-b5e6-96231b3b80d8
2015-06-30 15:31:17 +00:00
Jordan Rose e30024c96f [analyzer] CmpRuns.py: Accept single files as input.
This allows us to compare two direct invocations of the analyzer on a
single source file without having to wrap the output plists in their
own directories.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@177804 91177308-0d34-0410-b5e6-96231b3b80d8
2013-03-23 01:21:26 +00:00
Anna Zaks f0024960d5 [analyzer] CmpRuns: move the clang_version info into the run (out of
report)

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@168105 91177308-0d34-0410-b5e6-96231b3b80d8
2012-11-15 22:42:44 +00:00
Anna Zaks 25691f67b2 [analyzer] Ensure that CmpRuns recursively walks the output directory.
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@167981 91177308-0d34-0410-b5e6-96231b3b80d8
2012-11-14 21:32:16 +00:00
Anna Zaks 20d928164a [analyzer] CmpRuns: cleanup APIs, allow processing of plists with no
clang_version, fix the compare routine.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@166129 91177308-0d34-0410-b5e6-96231b3b80d8
2012-10-17 21:09:26 +00:00
Anna Zaks 68aa3a9389 [analyzer] Expose clang_version in CmpRuns
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@166048 91177308-0d34-0410-b5e6-96231b3b80d8
2012-10-16 19:36:39 +00:00
Anna Zaks 81765577db [analyzer] CmpRuns - cleaned up the API, added relative path getter
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@165995 91177308-0d34-0410-b5e6-96231b3b80d8
2012-10-15 22:48:21 +00:00
Anna Zaks 44cceb96ea [analyzer] Ted fixed the plist-html issue, so the assert can go away.
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@162598 91177308-0d34-0410-b5e6-96231b3b80d8
2012-08-24 21:07:49 +00:00
Anna Zaks b81a9adc24 [analyzer] Unbreak the static analyzer buildbot.
We no longer associate the correct HTML file with plist diagnostic.
Fall out from r162028.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@162071 91177308-0d34-0410-b5e6-96231b3b80d8
2012-08-17 01:09:13 +00:00
Anna Zaks 2a84b8bd10 [analyzer] CmpRuns should include file name in the issue identifier.
This prevents us from treating the issues from different files with the
same function names and same offsets as the same.

The issue identifier now includes the file name. Also added a way to
strip off the root directories form the source file names.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@161150 91177308-0d34-0410-b5e6-96231b3b80d8
2012-08-02 00:41:40 +00:00
Jordan Rose ac57d47546 Un-break standalone invocations of CmpRuns.py.
Fallout from CmpRuns.py API changes in r160314.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@160800 91177308-0d34-0410-b5e6-96231b3b80d8
2012-07-26 20:03:51 +00:00
Anna Zaks 7acc407240 [analyzer] Make CmpRuns external-user friendly.
CmpRuns can be used for static analyzer bug report comparison. However,
we want to make sure external users do not rely on the way bugs are
represented (plist files). Make sure that we have a user
friendly/documented API for CmpRuns script.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@160314 91177308-0d34-0410-b5e6-96231b3b80d8
2012-07-16 20:21:42 +00:00
Anna Zaks 19b17cb57a [analyzer] Use "issue hash" in CmpRuns; followup on r158180
(For the future: It would be more efficient to produce a hash key with
the embedded function info inside the compiler.)

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@158187 91177308-0d34-0410-b5e6-96231b3b80d8
2012-06-08 01:50:49 +00:00
Anna Zaks 2c3038edc2 [analyzer] Add the HTML file to the SATest diagnostic diff.
(Uses the functionality which has been in CmpRuns long before.)

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@148868 91177308-0d34-0410-b5e6-96231b3b80d8
2012-01-24 21:57:35 +00:00
Anna Zaks ab2d46eb04 [analyzer] Testing: make diagnostic diffs more informative (add
diagnostic message).

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@148867 91177308-0d34-0410-b5e6-96231b3b80d8
2012-01-24 21:57:32 +00:00
Anna Zaks a7a2564ff5 [analyzer] Testing: Display the number of reported differences in the main log.
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@144113 91177308-0d34-0410-b5e6-96231b3b80d8
2011-11-08 19:56:31 +00:00
Anna Zaks e1e73670b3 [analyzer] Relax the cmp criteria for analyzer reports.
Until we find out a way to easily find out what changed by looking at the logs.

git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@143789 91177308-0d34-0410-b5e6-96231b3b80d8
2011-11-05 05:20:56 +00:00
Anna Zaks f95a2012ab [analyzer] CmpRuns can now optionally delete empty reports.
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@139550 91177308-0d34-0410-b5e6-96231b3b80d8
2011-09-12 22:40:36 +00:00
Anna Zaks 544055fa46 [analyzer] CmpRuns.cmpScanBuildResults() should be easy to call from other modules.
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@139543 91177308-0d34-0410-b5e6-96231b3b80d8
2011-09-12 21:32:41 +00:00
Anna Zaks 22d70e0ca3 Rename CmpRuns into CmpRuns.py so that it could be used as a module.
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@139540 91177308-0d34-0410-b5e6-96231b3b80d8
2011-09-12 21:07:18 +00:00