This patch teaches clang to parse statements on the global scope to allow:
```
./bin/clang-repl
clang-repl> int i = 12;
clang-repl> ++i;
clang-repl> extern "C" int printf(const char*,...);
clang-repl> printf("%d\n", i);
13
clang-repl> %quit
```
Generally, disambiguating between statements and declarations is a non-trivial
task for a C++ parser. The challenge is to allow both standard C++ to be
translated as if this patch does not exist and in the cases where the user typed
a statement to be executed as if it were in a function body.
Clang's Parser does pretty well in disambiguating between declarations and
expressions. We have added DisambiguatingWithExpression flag which allows us to
preserve the existing and optimized behavior where needed and implement the
extra rules for disambiguating. Only few cases require additional attention:
* Constructors/destructors -- Parser::isConstructorDeclarator was used in to
disambiguate between ctor-looking declarations and statements on the global
scope(eg. `Ns::f()`).
* The template keyword -- the template keyword can appear in both declarations
and statements. This patch considers the template keyword to be a declaration
starter which breaks a few cases in incremental mode which will be tackled
later.
* The inline (and similar) keyword -- looking at the first token in many cases
allows us to classify what is a declaration.
* Other language keywords and specifiers -- ObjC/ObjC++/OpenCL/OpenMP rely on
pragmas or special tokens which will be handled in subsequent patches.
The patch conceptually models a "top-level" statement into a TopLevelStmtDecl.
The TopLevelStmtDecl is lowered into a void function with no arguments.
We attach this function to the global initializer list to execute the statement
blocks in the correct order.
Differential revision: https://reviews.llvm.org/D127284
Added support for incremental mode 8 and 28 ie. `frontend::EmitBC:` and `frontend::PrintPreprocessedInput:`
Added supporting clang tests to test in clang-repl mode
Reviewed By: v.g.vassilev
Differential Revision: https://reviews.llvm.org/D125946
This is required to support RISC-V where the '+d' target feature
indicates the presence of the D instruction set extension, which
changes to the Hard-float 'd' ABI.
Differential Revision: https://reviews.llvm.org/D128853
Without this, it is possible that Interpreter.cpp is being built before
clang/Driver/Options.inc is generated.
Observed only infrequently, serial builds and ext4 manifest the problem
much more often than parallel builds and btrfs.
https://reviews.llvm.org/rG06487b010d48c36c7714ee083ed38dff65711812
is a very similar case.
Patch by t184256!
Differential revision: https://reviews.llvm.org/D115827
In interactive C++ it is convenient to roll back to a previous state of the
compiler. For example:
clang-repl> int x = 42;
clang-repl> %undo
clang-repl> float x = 24 // not an error
To support this, the patch extends the functionality used to recover from
errors and adds functionality to recover the low-level execution infrastructure.
The current implementation is based on watermarks. It exploits the fact that
at each incremental input the underlying compiler infrastructure is in a valid
state. We can only go N incremental inputs back to a previous valid state. We do
not need and do not do any further dependency tracking.
This patch was co-developed with V. Vassilev, relies on the past work of Purva
Chaudhari in clang-repl and is inspired by the past work on the same feature
in the Cling interpreter.
Co-authored-by: Purva-Chaudhari <purva.chaudhari02@gmail.com>
Co-authored-by: Vassil Vassilev <v.g.vassilev@gmail.com>
Signed-off-by: Jun Zhang <jun@junz.org>
This patch implements soft reset and adds tests for soft reset success of the
diagnostics engine. This allows us to recover from errors in clang-repl without
resetting the pragma handlers' state.
Differential revision: https://reviews.llvm.org/D126183
Removes memory leak of ASTContext and TargetMachine. When DisableFree is turned on, it intentionally leaks these instances as they can be trivially deallocated. This patch turns this off and delete Parser instance early so that they will not reference dangling pargma headers.
Asan shouldn't detect these as leaks normally, since burypointer is called for them. But, every invocation of incremental parser createa an additional leak of TargetMachine. If there are many invocations within a single test case, we easily reach number of leaks exceeding kGraveYardMaxSize (which is 12) and leaks start to get reported by asan buildbots.
Reviewed By: v.g.vassilev
Differential Revision: https://reviews.llvm.org/D127991
Before this patch, there was re-declaration error if error was encountered in
the same line. The recovery support acted only if this type of error was
encountered in the first line of the program and not in subsequent lines.
For example:
```
clang-repl> int i=9;
clang-repl> int j=9; err;
input_line_3:1:5: error: redefinition of 'j'
int j = 9;
```
Differential revision: https://reviews.llvm.org/D123674
This patch will allow better incremental adoption of these changes in downstream
cling and other users which want to experiment by customizing the execution
engine.
In C++20 modules imports must be together and at the start of the module.
Rather than growing more ad-hoc flags to test state, this keeps track of the
phase of of a valid module TU (first decl, global module frag, module,
private module frag). If the phasing is broken (with some diagnostic) the
pattern does not conform to a valid C++20 module, and we set the state
accordingly.
We can thus issue diagnostics when imports appear in the wrong places and
decouple the C++20 modules state from other module variants (modules-ts and
clang modules). Additionally, we attempt to diagnose wrong imports before
trying to find the module where possible (the latter will generally emit an
unhelpful diagnostic about the module not being available).
Although this generally simplifies the handling of C++20 module import
diagnostics, the motivation was that, in particular, it allows detecting
invalid imports like:
import module A;
int some_decl();
import module B;
where being in a module purview is insufficient to identify them.
Differential Revision: https://reviews.llvm.org/D118893
In C++20 modules imports must be together and at the start of the module.
Rather than growing more ad-hoc flags to test state, this keeps track of the
phase of of a valid module TU (first decl, global module frag, module,
private module frag). If the phasing is broken (with some diagnostic) the
pattern does not conform to a valid C++20 module, and we set the state
accordingly.
We can thus issue diagnostics when imports appear in the wrong places and
decouple the C++20 modules state from other module variants (modules-ts and
clang modules). Additionally, we attempt to diagnose wrong imports before
trying to find the module where possible (the latter will generally emit an
unhelpful diagnostic about the module not being available).
Although this generally simplifies the handling of C++20 module import
diagnostics, the motivation was that, in particular, it allows detecting
invalid imports like:
import module A;
int some_decl();
import module B;
where being in a module purview is insufficient to identify them.
Differential Revision: https://reviews.llvm.org/D118893
Original commit message: "
Original commit message: "
Original commit message: "
Original commit message:"
The current infrastructure in lib/Interpreter has a tool, clang-repl, very
similar to clang-interpreter which also allows incremental compilation.
This patch moves clang-interpreter as a test case and drops it as conditionally
built example as we already have clang-repl in place.
"
This patch also ignores ppc due to missing weak symbol for __gxx_personality_v0
which may be a feature request for the jit infrastructure. Also, adds a missing
build system dependency to the orc jit.
"
Additionally, this patch defines a custom exception type and thus avoids the
requirement to include header <exception>, making it easier to deploy across
systems without standard location of the c++ headers.
"
This patch also works around PR49692 and finds a way to use llvm::consumeError
in rtti mode.
"
This patch also checks if stl is built with rtti.
Differential revision: https://reviews.llvm.org/D107049
Original commit message: "
Original commit message: "
Original commit message:"
The current infrastructure in lib/Interpreter has a tool, clang-repl, very
similar to clang-interpreter which also allows incremental compilation.
This patch moves clang-interpreter as a test case and drops it as conditionally
built example as we already have clang-repl in place.
Differential revision: https://reviews.llvm.org/D107049
"
This patch also ignores ppc due to missing weak symbol for __gxx_personality_v0
which may be a feature request for the jit infrastructure. Also, adds a missing
build system dependency to the orc jit.
"
Additionally, this patch defines a custom exception type and thus avoids the
requirement to include header <exception>, making it easier to deploy across
systems without standard location of the c++ headers.
"
This patch also works around PR49692 and finds a way to use llvm::consumeError
in rtti mode.
Differential revision: https://reviews.llvm.org/D107049
mingw-g++ does not correctly support the full `std::errc` namespace as
worded in the standard[1]. As such, we cannot reliably use all names
therein. This patch changes the use of
`std::errc::state_not_recoverable`, to use portable error codes from the
`llvm::errc` equivalent.
[1] https://gcc.gnu.org/bugzilla/show_bug.cgi?id=71444
Reviewed by v.g.vassilev
Differential Revision: https://reviews.llvm.org/D111315
Original commit message: "
Original commit message:"
The current infrastructure in lib/Interpreter has a tool, clang-repl, very
similar to clang-interpreter which also allows incremental compilation.
This patch moves clang-interpreter as a test case and drops it as conditionally
built example as we already have clang-repl in place.
Differential revision: https://reviews.llvm.org/D107049
"
This patch also ignores ppc due to missing weak symbol for __gxx_personality_v0
which may be a feature request for the jit infrastructure. Also, adds a missing
build system dependency to the orc jit.
"
Additionally, this patch defines a custom exception type and thus avoids the
requirement to include header <exception>, making it easier to deploy across
systems without standard location of the c++ headers.
Differential revision: https://reviews.llvm.org/D107049
The way we parse `DiagnosticOptions` is a bit involved.
`DiagnosticOptions` are parsed as part of the cc1-parsing function `CompilerInvocation::CreateFromArgs` which takes `DiagnosticsEngine` as an argument to be able to report errors in command-line arguments. But to create `DiagnosticsEngine`, `DiagnosticOptions` are needed. This is solved by exposing the `ParseDiagnosticArgs` to clients and making its `DiagnosticsEngine` argument optional, essentially breaking the dependency cycle.
The `ParseDiagnosticArgs` function takes `llvm::opt::ArgList &`, which each client needs to create from the command-line (typically represented as `std::vector<const char *>`). Creating this data structure in this context is somewhat particular. This code pattern is copy-pasted in some places across the upstream code base and also in downstream repos. To make things a bit more uniform, this patch extracts the code into a new reusable function: `CreateAndPopulateDiagOpts`.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D108918
Original commit message:"
The current infrastructure in lib/Interpreter has a tool, clang-repl, very
similar to clang-interpreter which also allows incremental compilation.
This patch moves clang-interpreter as a test case and drops it as conditionally
built example as we already have clang-repl in place.
Differential revision: https://reviews.llvm.org/D107049
"
This patch also ignores ppc due to missing weak symbol for __gxx_personality_v0
which may be a feature request for the jit infrastructure. Also, adds a missing
build system dependency to the orc jit.
The current infrastructure in lib/Interpreter has a tool, clang-repl, very
similar to clang-interpreter which also allows incremental compilation.
This patch moves clang-interpreter as a test case and drops it as conditionally
built example as we already have clang-repl in place.
Differential revision: https://reviews.llvm.org/D107049
Original commit message:
[clang-repl] Implement partial translation units and error recovery.
https://reviews.llvm.org/D96033 contained a discussion regarding efficient
modeling of error recovery. @rjmccall has outlined the key ideas:
Conceptually, we can split the translation unit into a sequence of partial
translation units (PTUs). Every declaration will be associated with a unique PTU
that owns it.
The first key insight here is that the owning PTU isn't always the "active"
(most recent) PTU, and it isn't always the PTU that the declaration
"comes from". A new declaration (that isn't a redeclaration or specialization of
anything) does belong to the active PTU. A template specialization, however,
belongs to the most recent PTU of all the declarations in its signature - mostly
that means that it can be pulled into a more recent PTU by its template
arguments.
The second key insight is that processing a PTU might extend an earlier PTU.
Rolling back the later PTU shouldn't throw that extension away. For example, if
the second PTU defines a template, and the third PTU requires that template to
be instantiated at float, that template specialization is still part of the
second PTU. Similarly, if the fifth PTU uses an inline function belonging to the
fourth, that definition still belongs to the fourth. When we go to emit code in
a new PTU, we map each declaration we have to emit back to its owning PTU and
emit it in a new module for just the extensions to that PTU. We keep track of
all the modules we've emitted for a PTU so that we can unload them all if we
decide to roll it back.
Most declarations/definitions will only refer to entities from the same or
earlier PTUs. However, it is possible (primarily by defining a
previously-declared entity, but also through templates or ADL) for an entity
that belongs to one PTU to refer to something from a later PTU. We will have to
keep track of this and prevent unwinding to later PTU when we recognize it.
Fortunately, this should be very rare; and crucially, we don't have to do the
bookkeeping for this if we've only got one PTU, e.g. in normal compilation.
Otherwise, PTUs after the first just need to record enough metadata to be able
to revert any changes they've made to declarations belonging to earlier PTUs,
e.g. to redeclaration chains or template specialization lists.
It should even eventually be possible for PTUs to provide their own slab
allocators which can be thrown away as part of rolling back the PTU. We can
maintain a notion of the active allocator and allocate things like Stmt/Expr
nodes in it, temporarily changing it to the appropriate PTU whenever we go to do
something like instantiate a function template. More care will be required when
allocating declarations and types, though.
We would want the PTU to be efficiently recoverable from a Decl; I'm not sure
how best to do that. An easy option that would cover most declarations would be
to make multiple TranslationUnitDecls and parent the declarations appropriately,
but I don't think that's good enough for things like member function templates,
since an instantiation of that would still be parented by its original class.
Maybe we can work this into the DC chain somehow, like how lexical DCs are.
We add a different kind of translation unit `TU_Incremental` which is a
complete translation unit that we might nonetheless incrementally extend later.
Because it is complete (and we might want to generate code for it), we do
perform template instantiation, but because it might be extended later, we don't
warn if it declares or uses undefined internal-linkage symbols.
This patch teaches clang-repl how to recover from errors by disconnecting the
most recent PTU and update the primary PTU lookup tables. For instance:
```./clang-repl
clang-repl> int i = 12; error;
In file included from <<< inputs >>>:1:
input_line_0:1:13: error: C++ requires a type specifier for all declarations
int i = 12; error;
^
error: Parsing failed.
clang-repl> int i = 13; extern "C" int printf(const char*,...);
clang-repl> auto r1 = printf("i=%d\n", i);
i=13
clang-repl> quit
```
Differential revision: https://reviews.llvm.org/D104918
This reverts commit 6775fc6ffa.
It also reverts "[lldb] Fix compilation by adjusting to the new ASTContext signature."
This reverts commit 03a3f86071.
We see some failures on the lldb infrastructure, these changes might play a role
in it. Let's revert it now and see if the bots will become green.
Ref: https://reviews.llvm.org/D104918
https://reviews.llvm.org/D96033 contained a discussion regarding efficient
modeling of error recovery. @rjmccall has outlined the key ideas:
Conceptually, we can split the translation unit into a sequence of partial
translation units (PTUs). Every declaration will be associated with a unique PTU
that owns it.
The first key insight here is that the owning PTU isn't always the "active"
(most recent) PTU, and it isn't always the PTU that the declaration
"comes from". A new declaration (that isn't a redeclaration or specialization of
anything) does belong to the active PTU. A template specialization, however,
belongs to the most recent PTU of all the declarations in its signature - mostly
that means that it can be pulled into a more recent PTU by its template
arguments.
The second key insight is that processing a PTU might extend an earlier PTU.
Rolling back the later PTU shouldn't throw that extension away. For example, if
the second PTU defines a template, and the third PTU requires that template to
be instantiated at float, that template specialization is still part of the
second PTU. Similarly, if the fifth PTU uses an inline function belonging to the
fourth, that definition still belongs to the fourth. When we go to emit code in
a new PTU, we map each declaration we have to emit back to its owning PTU and
emit it in a new module for just the extensions to that PTU. We keep track of
all the modules we've emitted for a PTU so that we can unload them all if we
decide to roll it back.
Most declarations/definitions will only refer to entities from the same or
earlier PTUs. However, it is possible (primarily by defining a
previously-declared entity, but also through templates or ADL) for an entity
that belongs to one PTU to refer to something from a later PTU. We will have to
keep track of this and prevent unwinding to later PTU when we recognize it.
Fortunately, this should be very rare; and crucially, we don't have to do the
bookkeeping for this if we've only got one PTU, e.g. in normal compilation.
Otherwise, PTUs after the first just need to record enough metadata to be able
to revert any changes they've made to declarations belonging to earlier PTUs,
e.g. to redeclaration chains or template specialization lists.
It should even eventually be possible for PTUs to provide their own slab
allocators which can be thrown away as part of rolling back the PTU. We can
maintain a notion of the active allocator and allocate things like Stmt/Expr
nodes in it, temporarily changing it to the appropriate PTU whenever we go to do
something like instantiate a function template. More care will be required when
allocating declarations and types, though.
We would want the PTU to be efficiently recoverable from a Decl; I'm not sure
how best to do that. An easy option that would cover most declarations would be
to make multiple TranslationUnitDecls and parent the declarations appropriately,
but I don't think that's good enough for things like member function templates,
since an instantiation of that would still be parented by its original class.
Maybe we can work this into the DC chain somehow, like how lexical DCs are.
We add a different kind of translation unit `TU_Incremental` which is a
complete translation unit that we might nonetheless incrementally extend later.
Because it is complete (and we might want to generate code for it), we do
perform template instantiation, but because it might be extended later, we don't
warn if it declares or uses undefined internal-linkage symbols.
This patch teaches clang-repl how to recover from errors by disconnecting the
most recent PTU and update the primary PTU lookup tables. For instance:
```./clang-repl
clang-repl> int i = 12; error;
In file included from <<< inputs >>>:1:
input_line_0:1:13: error: C++ requires a type specifier for all declarations
int i = 12; error;
^
error: Parsing failed.
clang-repl> int i = 13; extern "C" int printf(const char*,...);
clang-repl> auto r1 = printf("i=%d\n", i);
i=13
clang-repl> quit
```
Differential revision: https://reviews.llvm.org/D104918
Vassil Vassilev