This patch mechanically replaces None with std::nullopt where the
compiler would warn if None were deprecated. The intent is to reduce
the amount of manual work required in migrating from Optional to
std::optional.
This is part of an effort to migrate from llvm::Optional to
std::optional:
https://discourse.llvm.org/t/deprecating-llvm-optional-x-hasvalue-getvalue-getvalueor/63716
Before the fix the scanning would fail with
`-Werror,-Wnon-modular-include-in-module` despite the warning being
suppressed in the source code.
Existing approach with `-Wno-error` is not sufficient because it negates
only general `-Werror` but not specific `-Werror=...` and some warnings
can still emitted as errors. Make the approach stricter by using `-w`
flag and ignore all warnings, including those upgraded to errors. This
approach is still valid as it doesn't affect the dependencies.
rdar://101588531
Differential Revision: https://reviews.llvm.org/D138252
When a pcm has already been loaded from disk, reuse it from the
InMemoryModuleCache in readASTFileControlBlock. This avoids potentially
reading it again.
As noted in the FIXME, ideally we would also add the module to the cache
if it will be used again later, but that could modify its build state
and we do not have enough context currenlty to know if it's correct.
Differential Revision: https://reviews.llvm.org/D138160
This is a fix related to D135414. The original intention was to keep `BaseFS` as a member of the worker and conditionally overlay it with local in-memory FS. The `move` of ref-counted `BaseFS` was not intended, and it's a bug.
Disabling parallelism in the "by-module-name" test reliably reproduces this, and the test itself doesn't *need* parallelism. (I think `-j 4` was cargo culted from another test.) Reusing that test to check for correct behavior...
Reviewed By: DavidSpickett
Differential Revision: https://reviews.llvm.org/D136124
The `-ivfsoverlay` flag was only being respected when the scanner was instructed to minimize the inputs. This patch respects that flag even in canonical preprocessing mode.
Depends on D135414.
Reviewed By: Bigcheese
Differential Revision: https://reviews.llvm.org/D135416
The dependency scanning worker may create an in-memory VFS and inject that into its `FileManager`. (Implemented in D109485.) This VFS currently persists between scans, which is not correct: in-memory files created in one scan could affect subsequent scans. This patch changes things so that the in-memory VFS is local to `DependencyScanningWorker::computeDependencies()`.
To test this, one would first scan for dependencies of a named module and then run second scan (on the same worker) for something unrelated, which would pick up the in-memory file created in the first scan. This set up is impossible to achieve with `clang-scan-deps`. We could set this up in `ToolingTests`, but the scanner needs to access the physical filesystem to store `.pcm` files. AFAIK we don't have a good way to propagate something like `%t` into unit tests to make that feasible. (This could be achieved with something like the virtualized `OutputManager`.)
Reviewed By: Bigcheese
Differential Revision: https://reviews.llvm.org/D135414
This patch provides `FileManager` with the CWD on construction in the worker, rather than later in the action.
Depends on D134976.
Reviewed By: benlangmuir
Differential Revision: https://reviews.llvm.org/D134977
This patch removes the ability of a dependency scanning worker to share a `FileManager` instance between individual scans. It's not sound and doesn't provide performance benefits (due to the underlying caching VFS).
Reviewed By: benlangmuir
Differential Revision: https://reviews.llvm.org/D134976
This patch adds new member function to `DependencyScanningWorker` that allows clients to pass custom `DiagnosticConsumer`, and returns `bool`.
This provides more flexibility compared to the existing version that automatically stringifies diagnostics and returns them in `llvm::Error`.
Reviewed By: benlangmuir
Differential Revision: https://reviews.llvm.org/D134838
The `ScanInstance` is a local variable in `DependencyScanningAction::runInvocation()` that is referenced by `ModuleDepCollector`. Since D132405, `ModuleDepCollector` can escape the function and can outlive its `ScanInstance`. This patch fixes that.
Reviewed By: benlangmuir
Differential Revision: https://reviews.llvm.org/D133988
Attempt to fix the test failures observed in CI:
* Add Option dependency, which caused BUILD_SHARED_LIBS builds to fail
* Adapt tests that accidentally depended on the host platform: platforms
that don't use an integrated assembler (e.g. AIX) get a different set
of commands from the driver. Most dependency scanner tests can use
-fsyntax-only or -E instead of -c to avoid this, and in the rare case
we want to check -c specifically, set an explicit target so the
behaviour is independent of the host.
Original commit message follows.
---
Instead of trying to "fix" the original driver invocation by appending
arguments to it, split it into multiple commands, and for each -cc1
command use a CompilerInvocation to give precise control over the
invocation.
This change should make it easier to (in the future) canonicalize the
command-line (e.g. to improve hits in something like ccache), apply
optimizations, or start supporting multi-arch builds, which would
require different modules for each arch.
In the long run it may make sense to treat the TU commands as a
dependency graph, each with their own dependencies on modules or earlier
TU commands, but for now they are simply a list that is executed in
order, and the dependencies are simply duplicated. Since we currently
only support single-arch builds, there is no parallelism available in
the execution.
Differential Revision: https://reviews.llvm.org/D132405
Instead of trying to "fix" the original driver invocation by appending
arguments to it, split it into multiple commands, and for each -cc1
command use a CompilerInvocation to give precise control over the
invocation.
This change should make it easier to (in the future) canonicalize the
command-line (e.g. to improve hits in something like ccache), apply
optimizations, or start supporting multi-arch builds, which would
require different modules for each arch.
In the long run it may make sense to treat the TU commands as a
dependency graph, each with their own dependencies on modules or earlier
TU commands, but for now they are simply a list that is executed in
order, and the dependencies are simply duplicated. Since we currently
only support single-arch builds, there is no parallelism available in
the execution.
Differential Revision: https://reviews.llvm.org/D132405
This patch introduces new option `-eager-load-pcm` to `clang-scan-deps`, which controls whether the resulting command-lines will load PCM files eagerly (at the start of compilation) or lazily (when handling import directive). This patch also switches the default from eager to lazy.
To reduce the potential for churn in LIT tests in the future, this patch also removes redundant checks of command-line arguments and introduces new test `modules-dep-args.c` as a substitute.
Reviewed By: benlangmuir
Differential Revision: https://reviews.llvm.org/D132066
Sharing the FileManager across implicit module builds currently leaks
paths looked up in an importer into the built module itself. This can
cause non-deterministic results across scans. It is especially bad for
modules since the path can be saved into the pcm file itself, leading to
stateful behaviour if the cache is shared.
This should not impact the number of real filesystem accesses in the
scanner, since it is already caching in the
DependencyScanningWorkerFilesystem.
Note: this change does not affect whether or not the FileManager is
shared across TUs in the scanner, which is a separate issue.
Differential Revision: https://reviews.llvm.org/D131412
It's an accident that we started return asbolute paths from
FileEntry::getName for all relative paths. Prepare for getName to get
(closer to) return the requested path. Note: conceptually it might make
sense for the dependency scanner to allow relative paths and have the
DependencyConsumer decide if it wants to make them absolute, but we
currently document that it's absolute and I didn't want to change
behaviour here.
Differential Revision: https://reviews.llvm.org/D130934
The command-line arguments for module builds are cc1 commands, so they
do not implicitly set -disable-free like a driver invocation, and
Tooling will disable it for the scanning instance itself. Set
-disable-free explicitly so that separate invocations for building
modules will not pay for freeing memory unnecessarily.
Differential Revision: https://reviews.llvm.org/D127229
This is a commit with the following changes:
* Remove `ExcludedPreprocessorDirectiveSkipMapping` and related functionality
Removes `ExcludedPreprocessorDirectiveSkipMapping`; its intended benefit for fast skipping of excluded directived blocks
will be superseded by a follow-up patch in the series that will use dependency scanning lexing for the same purpose.
* Refactor dependency scanning to produce pre-lexed preprocessor directive tokens, instead of minimized sources
Replaces the "source minimization" mechanism with a mechanism that produces lexed dependency directives tokens.
* Make the special lexing for dependency scanning a first-class feature of the `Preprocessor` and `Lexer`
This is bringing the following benefits:
* Full access to the preprocessor state during dependency scanning. E.g. a component can see what includes were taken and where they were located in the actual sources.
* Improved performance for dependency scanning. Measurements with a release+thin-LTO build shows ~ -11% reduction in wall time.
* Opportunity to use dependency scanning lexing to speed-up skipping of excluded conditional blocks during normal preprocessing (as follow-up, not part of this patch).
For normal preprocessing measurements show differences are below the noise level.
Since, after this change, we don't minimize sources and pass them in place of the real sources, `DependencyScanningFilesystem` is not technically necessary, but it has valuable performance benefits for caching file `stat`s along with the results of scanning the sources. So the setup of using the `DependencyScanningFilesystem` during a dependency scan remains.
Differential Revision: https://reviews.llvm.org/D125486
Differential Revision: https://reviews.llvm.org/D125487
Differential Revision: https://reviews.llvm.org/D125488
This is first of a series of patches for making the special lexing for dependency scanning a first-class feature of the `Preprocessor` and `Lexer`.
This patch only includes NFC renaming changes to make reviewing of the functionality changing parts easier.
Differential Revision: https://reviews.llvm.org/D125484
This is to improve maintenance a bit and remove need to maintain the additional option and related code-paths.
Differential Revision: https://reviews.llvm.org/D124558
The code for traversing precompiled dependencies is somewhat complicated and contains a dangling iterator bug.
This patch simplifies the code and fixes the bug.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D121533
While scanning dependencies of a TU that depends on a PCH, the scanner basically performs mixed implicit/explicit modular compilation. (Explicit modules come from the PCH.) This seems to trip up the global module index.
This patch disables global module index in the dependency scanner.
Reviewed By: Bigcheese
Differential Revision: https://reviews.llvm.org/D118890
The filesystem used during dependency scanning does two things: it caches file entries and minimizes source file contents. We use the term "ignored file" in a couple of places, but it's not clear what exactly that means. This commit clears up the semantics, explicitly spelling out this relates to minimization.
The dependency scanner works with multiple instances of `Compiler{Instance,Invocation}`. From names of the variables/members, their purpose is not obvious.
This patch gives a distinct name to the `CompilerInstance` that's used to run the implicit build during dependency scan.
Depends on D111724.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D111725
To reduce the number of explicit builds of a single module, we can try to squash multiple occurrences of the module with different command-lines (and context hashes) by removing benign command-line options. The greatest contributors to benign differences between command-lines are the header search paths.
In this patch, the lookup cache in `HeaderSearch` is used to identify paths that were actually used when implicitly building the module during scanning. This information is serialized into the unhashed control block of the implicitly-built PCM. The dependency scanner then loads this and may use it to prune the header search paths before computing the context hash of the module and generating the command-line.
We could also prune the header search paths when serializing `HeaderSearchOptions` into the PCM. That way, we could do it only once instead of every load of the PCM file by dependency scanner. However, that would result in a PCM file whose contents don't produce the same context hash as the original build, which is probably highly surprising.
There is an alternative approach to storing extra information into the PCM: wire up preprocessor callbacks to capture the used header search paths on-the-fly during preprocessing of modularized headers (similar to what we currently do for the main source file and textual headers). Right now, that's not compatible with the fact that we do an actual implicit build producing PCM files during dependency scanning. The second run of dependency scanner loads the PCM from the first run, skipping the preprocessing altogether, which would result in different results between runs. We can revisit this approach when we stop building implicitly during dependency scanning.
Depends on D102923.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D102488
During dependency scanning, we generally want to suppress -Werror. Apply the same logic to the DiagnosticOptions instance used for command-line parsing.
This fixes a test failure on the PS4 bot, where the system header directory could not be found, which was reported due to -Werror being on the command line and not being sanitized.
In this patch the dependency scanner starts using proper `DiagnosticOptions` parsed from the actual TU command-line in order to mimic what the actual compiler would do. The actual functionality will be enabled and tested in follow-up patches. (This split is necessary to avoid temporary regression.)
Depends on D108976.
Reviewed By: dexonsmith, arphaman
Differential Revision: https://reviews.llvm.org/D108982
The dependency scanner currently uses `ClangTool` to invoke the dependency scanning action.
However, `ClangTool` seems to be the wrong level of abstraction. It's intended to be run over a collection of compile commands, which we actively avoid via `SingleCommandCompilationDatabase`. It automatically injects `-fsyntax-only` and other flags, which we avoid by calling `clearArgumentsAdjusters()`. It deduces the resource directory based on the current executable path, which we'd like to change to deducing from `argv[0]`.
Internally, `ClangTool` uses `ToolInvocation` which seems to be more in line with what the dependency scanner tries to achieve. This patch switches to directly using `ToolInvocation` instead. NFC.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D108979
This patch changes how the dependency scanner creates the fake input file when scanning dependencies of a single module (introduced in D109485). The scanner now has its own `InMemoryFilesystem` which sits under the minimizing FS (when that's requested). This makes it possible to drop the duplicate work in `DependencyScanningActions::runInvocation` that sets up the main file ID. Besides that, this patch makes it possible to land D108979, where we drop `ClangTool` entirely.
Depends on D109485.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D109498
module lookup by name alone
This removes the need to create a fake source file that imports a
module.
rdar://64538073
Differential Revision: https://reviews.llvm.org/D109485
This patch avoid minimizing input files that contributed to a PCH or its modules. This prevents the implicit modular build to fail on unexpected file size. Depends on D106146.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D104536
This patch normalizes filenames in `DependencyScanningWorkerFilesystem` so that lookup of ignored files works correctly on Windows (where `/` and `\` are equivalent).
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D106064
The dependency scanning worker uses `std::move` to "reset" `DependencyOutputOptions` in the `CompilerInstance` that performs the implicit build. It's probably preferable to replace the object with value-initialized instance, rather than depending on the behavior of a moved-from object.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D104106
There's no need to pass `DependencyOutputOptions` to each call of `handleFileDependency`, since the options don't ever change.
This patch adds new `handleDependencyOutputOpts` method to the `DependencyConsumer` interface and the dependency scanner uses it to report the options only once.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D104104
One of the goals of the dependency scanner is to generate command-lines that can be used to explicitly build modular dependencies of a translation unit. The only modifications to these command-lines should be for the purposes of explicit modular build.
However, the current version of dependency scanner leaks its implementation details into the command-lines.
The first problem is that the `clang-scan-deps` tool adjusts the original textual command-line (adding `-Eonly -M -MT <target> -sys-header-deps -Wno-error -o /dev/null `, removing `--serialize-diagnostics`) in order to set up the `DependencyScanning` library. This has been addressed in D103461, D104012, D104030, D104031, D104033. With these patches, the `DependencyScanning` library receives the unmodified `CompilerInvocation`, sets it up and uses it for the implicit modular build.
Finally, to prevent leaking the implementation details to the resulting command-lines, this patch generates them from the **original** unmodified `CompilerInvocation` rather than from the one that drives the implicit build.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D104036
This patch moves enabling system header deps from `clang-scan-deps` into the `DependencyScanning` library. This will make it easier to preserve semantics of the original TU command-line for modular dependencies (see D104036).
Reviewed By: arphaman
Differential Revision: https://reviews.llvm.org/D104033
This moves another piece of logic specific to `clang-scan-deps` into the `DependencyScanning` library. This makes it easier to check how the original command-line looked like in the library and will enable the library to stop inventing `-Wno-error` for modular dependencies (see D104036).
Reviewed By: arphaman
Differential Revision: https://reviews.llvm.org/D104031
The `clang-scan-deps` tool has some logic that parses and modifies the original Clang command-line. The goal is to setup `DependencyOutputOptions` by injecting `-M -MT <target>` and prevent the creation of output files.
This patch moves the logic into the `DependencyScanning` library, and uses the parsed `CompilerInvocation` instead of the raw command-line. The code simpler and can be used from the C++ API as well.
The `-o /dev/null` arguments are not necessary, since the `DependencyScanning` library only runs a preprocessing action, so there's no way it'll produce an actual object file.
Related: The `-M` argument implies `-w`, which would appear on the command-line of modular dependencies even though it was not on the original TU command line (see D104036).
Some related tests were updated.
Reviewed By: arphaman
Differential Revision: https://reviews.llvm.org/D104030
To prevent the creation of diagnostics file, `clang-scan-deps` strips the corresponding command-line argument. This behavior is useful even when using the C++ `DependencyScanner` library.
This patch transforms stripping of command-line in `clang-scan-deps` into stripping of `CompilerInvocation` in `DependencyScanning`.
AFAIK, the `clang-cl` driver doesn't even accept `--serialize-diagnostics`, so I've removed the test. (It would fail with an unknown command-line argument otherwise.)
Note: Since we're generating command-lines for modular dependencies from `CompilerInvocation`, the `--serialize-diagnostics` will be dropped. This was already happening in `clang-scan-deps` before this patch, but it will now happen also when using `DependencyScanning` library directly. This is resolved in D104036.
Reviewed By: dexonsmith, arphaman
Differential Revision: https://reviews.llvm.org/D104012
When a translation unit uses a PCH and imports the same modules as the PCH, we'd prefer to resolve to those modules instead of inventing new modules and reporting them as modular dependencies. Since the PCH modules have already been built nudge the compiler to reuse them when deciding whether to build a new module and don't report them as regular modular dependencies.
Depends on D103524 & D103802.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D103526
The `PreprocessOnlyAction` doesn't support loading the AST file of a precompiled header. This is problematic for dependency scanning, since the `#include` manufactured for the PCH is treated as textual. This means the PCH contents get scanned with each TU, which is redundant. Moreover, dependencies of the PCH end up being considered dependency of the TU.
To handle AST file of PCH properly, this patch creates new `FrontendAction` that behaves the same way `PreprocessorOnlyAction` does, but treats the manufactured PCH `#include` as a normal compilation would (by not claiming it only uses a preprocessor and creating the default AST consumer).
The AST file is now reported as a file dependency of the TU.
Depends on D103519.
Reviewed By: Bigcheese
Differential Revision: https://reviews.llvm.org/D103524
When a project uses PCH with explicit modules, the build will look like this:
1. scan PCH dependencies
2. explicitly build PCH
3. scan TU dependencies
4. explicitly build TU
Step 2 produces an object file for the PCH, which the dependency scanner needs to read in step 3. This patch adds support for this.
The `clang-scan-deps` invocation in the attached test would fail without this change.
Depends on D103516.
Reviewed By: Bigcheese
Differential Revision: https://reviews.llvm.org/D103519
Allow a `std::unique_ptr` to be moved into the an `IntrusiveRefCntPtr`,
and remove a couple of now-unnecessary `release()` calls.
Differential Revision: https://reviews.llvm.org/D92888
This is how it should've been and brings it more in line with
std::string_view. There should be no functional change here.
This is mostly mechanical from a custom clang-tidy check, with a lot of
manual fixups. It uncovers a lot of minor inefficiencies.
This doesn't actually modify StringRef yet, I'll do that in a follow-up.
Kazu Hirata