Reapply r269100 and r269270, reverted due to
https://llvm.org/bugs/show_bug.cgi?id=27725. Isolate the testcase that
corresponds to the new feature side of this commit and skip it on
windows hosts until we find why it does not work on these platforms.
Original commit message:
The way we currently build the internal VFS overlay representation leads
to inefficient path search and might yield wrong answers when asked for
recursive or regular directory iteration.
Currently, when reading an YAML file, each YAML root entry is placed
inside a new root in the filesystem overlay. In the crash reproducer, a
simple "@import Foundation" currently maps to 43 roots, and when looking
up paths, we traverse a directory tree for each of these different
roots, until we find a match (or don't). This has two consequences:
- It's slow.
- Directory iteration gives incomplete results since it only return
results within one root - since contents of the same directory can be
declared inside different roots, the result isn't accurate.
This is in part fault of the way we currently write out the YAML file
when emitting the crash reproducer - we could generate only one root and
that would make it fast and correct again. However, we should not rely
on how the client writes the YAML, but provide a good internal
representation regardless.
Build a proper virtual directory tree out of the YAML representation,
allowing faster search and proper iteration. Besides the crash
reproducer, this potentially benefits other VFS clients.
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@269327 91177308-0d34-0410-b5e6-96231b3b80d8
The way we currently build the internal VFS overlay representation leads
to inefficient path search and might yield wrong answers when asked for
recursive or regular directory iteration.
Currently, when reading an YAML file, each YAML root entry is placed
inside a new root in the filesystem overlay. In the crash reproducer, a
simple "@import Foundation" currently maps to 43 roots, and when looking
up paths, we traverse a directory tree for each of these different
roots, until we find a match (or don't). This has two consequences:
- It's slow.
- Directory iteration gives incomplete results since it only return
results within one root - since contents of the same directory can be
declared inside different roots, the result isn't accurate.
This is in part fault of the way we currently write out the YAML file
when emitting the crash reproducer - we could generate only one root and
that would make it fast and correct again. However, we should not rely
on how the client writes the YAML, but provide a good internal
representation regardless.
This patch builds a proper virtual directory tree out of the YAML
representation, allowing faster search and proper iteration. Besides the
crash reproducer, this potentially benefits other VFS clients.
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@269270 91177308-0d34-0410-b5e6-96231b3b80d8
Bruno made a couple valiant attempts but the bot is still red.
This reverts r269100 (primary commit), r269108 (fix attempt), r269133
(fix attempt).
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@269160 91177308-0d34-0410-b5e6-96231b3b80d8
The way we currently build the internal VFS overlay representation leads
to inefficient path search and might yield wrong answers when asked for
recursive or regular directory iteration.
Currently, when reading an YAML file, each YAML root entry is placed
inside a new root in the filesystem overlay. In the crash reproducer, a
simple "@import Foundation" currently maps to 43 roots, and when looking
up paths, we traverse a directory tree for each of these different
roots, until we find a match (or don't). This has two consequences:
- It's slow.
- Directory iteration gives incomplete results since it only return
results within one root - since contents of the same directory can be
declared inside different roots, the result isn't accurate.
This is in part fault of the way we currently write out the YAML file
when emitting the crash reproducer - we could generate only one root and
that would make it fast and correct again. However, we should not rely
on how the client writes the YAML, but provide a good internal
representation regardless.
This patch builds a proper virtual directory tree out of the YAML
representation, allowing faster search and proper iteration. Besides the
crash reproducer, this potentially benefits other VFS clients.
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@269100 91177308-0d34-0410-b5e6-96231b3b80d8
Hide the real paths when rebuilding from VFS by setting up the crash
reproducer to use 'use-external-names' = false. This way we avoid
module redifinition errors and consistently use the same paths against
all modules.
With this change on Darwin we are able to simulate a crash for a simple
application using "Foundation/Foundation.h" (which relies on a bunch of
different frameworks and headers) and successfully rebuild all the
modules by relying solely at the VFS overlay.
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@266234 91177308-0d34-0410-b5e6-96231b3b80d8
The VFS YAML files contain empty directory entries to describe that it's
returning from a subdirectory before describing new files in the parent.
In the future, we should properly sort and write YAML files avoiding
such empty dirs and mitigate the extra recurson cost. However, since
this is used by previous existing YAMLs, make the traversal work in
their presence.
rdar://problem/24499339
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@264970 91177308-0d34-0410-b5e6-96231b3b80d8
The FileSystem::makeAbsolute function has been calculating the current
working directory unconditionally, even when it is not needed. This calls
down to llvm::sys::fs::current_path, which is relatively expensive
because it stats two directories, regardless of whether those paths are
already in the stat cache. The net effect is that when using the
VFS, every stat during header search turns into three stats. With this
change, we get back to a single stat for absolute directory paths.
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@264519 91177308-0d34-0410-b5e6-96231b3b80d8
This reapplies r261552 and r263748. Fixed testcase to reapply.
The VFS overlay mapping between virtual paths and real paths is done through
the 'external-contents' entries in YAML files, which contains hardcoded paths
to the real files.
When a module compilation crashes, headers are dumped into <name>.cache/vfs
directory and are mapped via the <name>.cache/vfs/vfs.yaml. The script
generated for reproduction uses -ivfsoverlay pointing to file to gather the
mapping between virtual paths and files inside <name>.cache/vfs. Currently, we
are only capable of reproducing such crashes in the same machine as they
happen, because of the hardcoded paths in 'external-contents'.
To be able to reproduce a crash in another machine, this patch introduces a new
option in the VFS yaml file called 'overlay-relative'. When it's equal to
'true' it means that the provided path to the YAML file through the
-ivfsoverlay option should also be used to prefix the final path for every
'external-contents'.
Example, given the invocation snippet "... -ivfsoverlay
<name>.cache/vfs/vfs.yaml" and the following entry in the yaml file:
"overlay-relative": "true",
"roots": [
...
"type": "directory",
"name": "/usr/include",
"contents": [
{
"type": "file",
"name": "stdio.h",
"external-contents": "/usr/include/stdio.h"
},
...
Here, a file manager request for virtual "/usr/include/stdio.h", that will map
into real path "/<absolute_path_to>/<name>.cache/vfs/usr/include/stdio.h.
This is a useful feature for debugging module crashes in machines other than
the one where the error happened.
Differential Revision: http://reviews.llvm.org/D17457
rdar://problem/24499339
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@263893 91177308-0d34-0410-b5e6-96231b3b80d8
This reapplies r261552.
The VFS overlay mapping between virtual paths and real paths is done through
the 'external-contents' entries in YAML files, which contains hardcoded paths
to the real files.
When a module compilation crashes, headers are dumped into <name>.cache/vfs
directory and are mapped via the <name>.cache/vfs/vfs.yaml. The script
generated for reproduction uses -ivfsoverlay pointing to file to gather the
mapping between virtual paths and files inside <name>.cache/vfs. Currently, we
are only capable of reproducing such crashes in the same machine as they
happen, because of the hardcoded paths in 'external-contents'.
To be able to reproduce a crash in another machine, this patch introduces a new
option in the VFS yaml file called 'overlay-relative'. When it's equal to
'true' it means that the provided path to the YAML file through the
-ivfsoverlay option should also be used to prefix the final path for every
'external-contents'.
Example, given the invocation snippet "... -ivfsoverlay
<name>.cache/vfs/vfs.yaml" and the following entry in the yaml file:
"overlay-relative": "true",
"roots": [
...
"type": "directory",
"name": "/usr/include",
"contents": [
{
"type": "file",
"name": "stdio.h",
"external-contents": "/usr/include/stdio.h"
},
...
Here, a file manager request for virtual "/usr/include/stdio.h", that will map
into real path "/<absolute_path_to>/<name>.cache/vfs/usr/include/stdio.h.
This is a useful feature for debugging module crashes in machines other than
the one where the error happened.
Differential Revision: http://reviews.llvm.org/D17457
rdar://problem/24499339
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@263748 91177308-0d34-0410-b5e6-96231b3b80d8
This was applied twice r261551 and 263617 and later reverted because:
(1) Windows bot failing on unittests. Change the current behavior to do
not handle path traversals on windows.
(2) Windows bot failed to include llvm/Config/config.h in order to use
HAVE_REALPATH. Use LLVM_ON_UNIX instead, as done in lib/Basic/FileManager.cpp.
Handle ".", ".." and "./" with trailing slashes while collecting files
to be dumped into the vfs overlay directory.
Include the support for symlinks into components. Given the path:
/install-dir/bin/../lib/clang/3.8.0/include/altivec.h, if "bin"
component is a symlink, it's not safe to use `path::remove_dots` here,
and `realpath` is used to get the right answer. Since `realpath`
is expensive, we only do it at collecting time (which only happens
during the crash reproducer) and cache the base directory for fast lookups.
Overall, this makes the input to the VFS YAML file to be canonicalized
to never contain traversal components.
Differential Revision: http://reviews.llvm.org/D17104
rdar://problem/24499339
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@263686 91177308-0d34-0410-b5e6-96231b3b80d8
This is originally r261551, reverted because of windows bots failing on
unittests. Change the current behavior to do not handle path traversals
on windows.
Handle ".", ".." and "./" with trailing slashes while collecting files
to be dumped into the vfs overlay directory.
Include the support for symlinks into components. Given the path:
/install-dir/bin/../lib/clang/3.8.0/include/altivec.h, if "bin"
component is a symlink, it's not safe to use `path::remove_dots` here,
and `realpath` is used to get the right answer. Since `realpath`
is expensive, we only do it at collecting time (which only happens
during the crash reproducer) and cache the base directory for fast lookups.
Overall, this makes the input to the VFS YAML file to be canonicalized
to never contain traversal components.
Differential Revision: http://reviews.llvm.org/D17104
rdar://problem/24499339
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@263617 91177308-0d34-0410-b5e6-96231b3b80d8
The SafelyCloseFileDescriptor machinery does the right thing in the face
of signals while close will do something platform specific which results
in the FD potentially getting leaked.
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@262687 91177308-0d34-0410-b5e6-96231b3b80d8
The VFS overlay mapping between virtual paths and real paths is done through
the 'external-contents' entries in YAML files, which contains hardcoded paths
to the real files.
When a module compilation crashes, headers are dumped into <name>.cache/vfs
directory and are mapped via the <name>.cache/vfs/vfs.yaml. The script
generated for reproduction uses -ivfsoverlay pointing to file to gather the
mapping between virtual paths and files inside <name>.cache/vfs. Currently, we
are only capable of reproducing such crashes in the same machine as they
happen, because of the hardcoded paths in 'external-contents'.
To be able to reproduce a crash in another machine, this patch introduces a new
option in the VFS yaml file called 'overlay-relative'. When it's equal to
'true' it means that the provided path to the YAML file through the
-ivfsoverlay option should also be used to prefix the final path for every
'external-contents'.
Example, given the invocation snippet "... -ivfsoverlay
<name>.cache/vfs/vfs.yaml" and the following entry in the yaml file:
"overlay-relative": "true",
"roots": [
...
"type": "directory",
"name": "/usr/include",
"contents": [
{
"type": "file",
"name": "stdio.h",
"external-contents": "/usr/include/stdio.h"
},
...
Here, a file manager request for virtual "/usr/include/stdio.h", that will map
into real path "/<absolute_path_to>/<name>.cache/vfs/usr/include/stdio.h.
This is a useful feature for debugging module crashes in machines other than
the one where the error happened.
Differential Revision: http://reviews.llvm.org/D17457
rdar://problem/24499339
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@261552 91177308-0d34-0410-b5e6-96231b3b80d8
Handle ".", ".." and "./" with trailing slashes while collecting files
to be dumped into the vfs overlay directory.
Include the support for symlinks into components. Given the path:
/install-dir/bin/../lib/clang/3.8.0/include/altivec.h, if "bin"
component is a symlink, it's not safe to use `path::remove_dots` here,
and `realpath` is used to get the right answer. Since `realpath`
is expensive, we only do it at collecting time (which only happens
during the crash reproducer) and cache the base directory for fast lookups.
Overall, this makes the input to the VFS YAML file to be canonicalized
to never contain traversal components.
Differential Revision: http://reviews.llvm.org/D17104
rdar://problem/24499339
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@261551 91177308-0d34-0410-b5e6-96231b3b80d8
FixedCompilationDatabase sets the working dir to "." by default. For
chdir(".") this is a noop but this lead to InMemoryFileSystem to create
bogus paths. Fixes PR25327.
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@257260 91177308-0d34-0410-b5e6-96231b3b80d8
Make RedirectedFileSystem::openFilForRead(path)->status() the same as
RedirectedFileSystem::status(path). Previously we would just get the
status of the underlying real file, which would not have the IsVFSMapped
bit set.
This fixes rebuilding a module that has an include that is relative to
the includer where we will lookup the real path of that file before we
lookup the VFS location.
rdar://problem/23640339
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@255312 91177308-0d34-0410-b5e6-96231b3b80d8
This is a more principled version of what I did earlier. Path
normalization is generally a good thing, but may break users in strange
environments, e. g. using lots of symlinks. Let the user choose and
default it to on.
This also changes adding a duplicated file into returning an error if
the file contents are different instead of an assertion failure.
Differential Revision: http://reviews.llvm.org/D13658
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@250060 91177308-0d34-0410-b5e6-96231b3b80d8
Actually the only special path we have to handle is ./foo, the rest is
tricky to get right so do the same thing as the existing YAML vfs here.
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@250036 91177308-0d34-0410-b5e6-96231b3b80d8
This can fail badly if we're overlaying a real file system and there are
symlinks there. Just keep the path as-is for now.
This essentially reverts r249830.
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@250021 91177308-0d34-0410-b5e6-96231b3b80d8
There are still some loose ends here but it's sufficient so we can detect
GCC headers that are inside of a VFS.
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@249556 91177308-0d34-0410-b5e6-96231b3b80d8
- Rename it to RedirectingFileSystem. This is what it does, YAML is just a
serialization format for it.
- Consistently use unique_ptr for memory management.
No functional change intended.
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@249532 91177308-0d34-0410-b5e6-96231b3b80d8
Apart from being cleaner this also means that clang-format no longer has
access to the host file system. This isn't necessary because clang-format
never reads includes :)
Includes minor tweaks and bugfixes found in the VFS implementation while
running clang-format tests.
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@249385 91177308-0d34-0410-b5e6-96231b3b80d8
For RealFileSystem this is getcwd()/chdir(), the synthetic file systems can
make up one for themselves. OverlayFileSystem now synchronizes the working
directories when a new FS is added to the overlay or the overlay working
directory is set. This allows purely artificial file systems that have zero
ties to the underlying disks.
Differential Revision: http://reviews.llvm.org/D13430
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@249316 91177308-0d34-0410-b5e6-96231b3b80d8
This is a simple file system tree of memory buffers that can be filled by a
client. In conjunction with an OverlayFS it can be used to make virtual
files accessible right next to physical files. This can be used as a
replacement for the virtual file handling in FileManager and which I intend
to remove eventually.
git-svn-id: https://llvm.org/svn/llvm-project/cfe/trunk@249315 91177308-0d34-0410-b5e6-96231b3b80d8
Taewook Oh