When creating an archive, llvm-ar looks at the host to determine the
archive format to use, on Apple platforms this means it uses the
K_DARWIN format. K_DARWIN is _virtually_ equivalent to K_BSD, expect for
some very slight differences around padding, timestamps in deterministic
mode, and 64 bit formats. When updating an archive using llvm-ar, or
llvm-objcopy, Archive would try to determine the kind, but it was not
possible to get K_DARWIN in the initialization of the archive, because
they're virtually inciting usable from K_BSD, especially since the
slight differences only apply in very specific cases. This leads to
linker failures when the alignment workaround is not applied to an
archive copied with llvm-objcopy. This change teaches Archive to infer
the K_DARWIN type in the cases where it's possible and the first object
in the archive is a macho object. This avoids using the host triple to
determine this to not affect cross compiling.
Ideally we would eliminate the separate K_DARWIN type entirely since
it's not a truly separate archive type, but then we'd have to force the
macho workarounds on the BSD format generally. This might be acceptable
but then it would be unclear how to handle this case without forcing the
K_DARWIN64 format on all BSD users:
```
if (LastOffset >= Sym64Threshold) {
if (Kind == object::Archive::K_DARWIN)
Kind = object::Archive::K_DARWIN64;
else
Kind = object::Archive::K_GNU64;
}
```
The logic used to determine if the object is macho is derived from the
logic llvm-ar uses.
Previous context:
- 111cd669e9
- 23a76be5ad
Differential Revision: https://reviews.llvm.org/D124895
SUMMARY
1. Enable supporting the write operation of big archive.
2. the first commit come from https://reviews.llvm.org/D104367
3. refactor the first commit and implement writing symbol table.
4. fixed the bugs and add more test cases in the second commit.
Reviewers: James Henderson
Differential Revision: https://reviews.llvm.org/D123949
Most notably,
llvm/Object/Binary.h no longer includes llvm/Support/MemoryBuffer.h
llvm/Object/MachOUniversal*.h no longer include llvm/Object/Archive.h
llvm/Object/TapiUniversal.h no longer includes llvm/Object/TapiFile.h
llvm-project preprocessed size:
before: 1068185081
after: 1068324320
Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.llvm.org/D119457
Summary:
The patch is based on the EGuesnet's implement of the "Support of Big archive (read)
the first commit of the patch is come from https://reviews.llvm.org/D100651.
the rest of commits of the patch
1 Addressed the comments on the https://reviews.llvm.org/D100651
2 according to https://www.ibm.com/docs/en/aix/7.2?topic=formats-ar-file-format-big
using the "fl_fstmoff" for the first object file number, using "char ar_nxtmem[20]" to get next object file ,
using the "char fl_lstmoff[20]" for the last of the object file will fix the following problems:
2.1 can not correct reading a archive files which has padding data between too object file
2.2 can not correct reading a archive files from which some object file has be deleted
3 introduce a new derived class BigArchive for big ar file.
Reviewers: James Henderson
Differential Revision: https://reviews.llvm.org/D111889
Summary:
The patch is based on the EGuesnet's implement of the "Support of Big archive (read)
the first commit of the patch is come from https://reviews.llvm.org/D100651.
the rest of commits of the patch
1 Addressed the comments on the https://reviews.llvm.org/D100651
2 according to https://www.ibm.com/docs/en/aix/7.2?topic=formats-ar-file-format-big
using the "fl_fstmoff" for the first object file number, using "char ar_nxtmem[20]" to get next object file ,
using the "char fl_lstmoff[20]" for the last of the object file will fix the following problems:
2.1 can not correct reading a archive files which has padding data between too object file
2.2 can not correct reading a archive files from which some object file has be deleted
3 introduce a new derived class BigArchive for big ar file.
Reviewers: James Henderson
Differential Revision: https://reviews.llvm.org/D111889
Summary:
The patch is based on the EGuesnet's implement of the "Support of Big archive (read)
the first commit of the patch is come from https://reviews.llvm.org/D100651.
the rest of commits of the patch
1 Addressed the comments on the https://reviews.llvm.org/D100651
2 according to https://www.ibm.com/docs/en/aix/7.2?topic=formats-ar-file-format-big
using the "fl_fstmoff" for the first object file number, using "char ar_nxtmem[20]" to get next object file ,
using the "char fl_lstmoff[20]" for the last of the object file will fix the following problems:
2.1 can not correct reading a archive files which has padding data between too object file
2.2 can not correct reading a archive files from which some object file has be deleted
3 introduce a new derived class BigArchive for big ar file.
Reviewers: James Henderson
Differential Revision: https://reviews.llvm.org/D111889
Most of `MemoryBuffer` interfaces expose a `RequiresNullTerminator` parameter that's being used to:
* determine how to open a file (`mmap` vs `open`),
* assert newly initialized buffer indeed has an implicit null terminator.
This patch adds the paramater to the `SmallVectorMemoryBuffer` constructors, meaning:
* null terminator can now be added to `SmallVector`s that didn't have one before,
* `SmallVectors` that had a null terminator before keep it even after the move.
In line with existing code, the new parameter is defaulted to `true`. This patch makes sure all calls to the `SmallVectorMemoryBuffer` constructor set it to `false` to preserve the current semantics.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D115331
The code to detect the requirement for 64-bit offsets in the archive
symbol table was not correctly accounting for the archive file signature
and the size of all the contents of the symbol table itself, e.g. the
symbol table's header and string table. Also was not considering the
variation in symbol table formats. This could result in the creation of
large archives with a corrupt symbol table.
Change the testing environment variable SYM64_THRESHOLD to be an
absolute value rather than a power of 2 in order to enable precise
testing of this detection code.
Differential Revision: https://reviews.llvm.org/D89891
When adding an archive member with a problem, e.g. a new bitcode with an
old archiver, containing an unsupported attribute, or an ELF file with a
malformed symbol table, the archiver would throw away the error and
simply add the member to the archive without any symbol entries. This
meant that the resultant archive could be silently unusable when not
using --whole-archive, and result in unexpected undefined symbols.
This change fixes this issue by addressing two FIXMEs and only throwing
away not-an-object errors. However, this meant that some LLD tests which
didn't need symbol tables and were using invalid members deliberately to
test the linker's malformed input handling no longer worked, so this
patch also stops the archiver from looking for symbols in an object if
it doesn't require a symbol table, and updates the tests accordingly.
Differential Revision: https://reviews.llvm.org/D88288
Reviewed by: grimar, rupprecht, MaskRay
Refactoring function `writeArchive` in ArchiveWriter. Added a new
function `writeArchiveBuffer` that returns the archive in a memory
buffer instead of writing it out to the disk. This refactor is necessary
so as to allow `llvm-libtool-darwin` to write universal files containing
archives.
Reviewed by jhenderson, MaskRay, smeenai
Differential Revision: https://reviews.llvm.org/D84858
This is how it should've been and brings it more in line with
std::string_view. There should be no functional change here.
This is mostly mechanical from a custom clang-tidy check, with a lot of
manual fixups. It uncovers a lot of minor inefficiencies.
This doesn't actually modify StringRef yet, I'll do that in a follow-up.
Summary:
This is patch is part of a series to introduce an Alignment type.
See this thread for context: http://lists.llvm.org/pipermail/llvm-dev/2019-July/133851.html
See this patch for the introduction of the type: https://reviews.llvm.org/D64790
Reviewers: courbet, JDevlieghere, alexshap, rupprecht, jhenderson
Subscribers: sdardis, nemanjai, hiraditya, kbarton, jakehehrlich, jrtc27, MaskRay, atanasyan, jsji, seiya, cfe-commits, llvm-commits
Tags: #clang, #llvm
Differential Revision: https://reviews.llvm.org/D67499
llvm-svn: 371742
llvm-ar outputs a strange error message when handling archives with
members larger than 4GB due to not checking file size when passing the
value as an unsigned 32 bit integer. This overflow issue caused
malformed archives to be created.:
https://bugs.llvm.org/show_bug.cgi?id=38058
This change allows for members above 4GB and will error in a case that
is over the formats size limit, a 10 digit decimal integer.
Differential Revision: https://reviews.llvm.org/D65093
llvm-svn: 366813
Summary:
On Windows, Posix integer file descriptors are a compatibility layer
over native file handles provided by the C runtime. There is a hard
limit on the maximum number of file descriptors that a process can open,
and the limit is 8192. LLD typically doesn't run into this limit because
it opens input files, maps them into memory, and then immediately closes
the file descriptor. This prevents it from running out of FDs.
For various reasons, I'd like to open handles to every input file and
keep them open during linking. That requires migrating MemoryBuffer over
to taking open native file handles instead of integer FDs.
Reviewers: aganea, Bigcheese
Reviewed By: aganea
Subscribers: smeenai, silvas, mehdi_amini, hiraditya, steven_wu, dexonsmith, dang, llvm-commits, zturner
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D63453
llvm-svn: 365588
Includes a fix for an introduced build failure due to a post c++11 use of std::mismatch.
This fixes some thin archive relative path issues, paths are shortened where possible and paths are output correctly when using the display table command.
Differential Revision: https://reviews.llvm.org/D59491
llvm-svn: 362484
This reverts commit r362407. It broke compilation of
llvm/lib/Object/ArchiveWriter.cpp:
error: type 'llvm::sys::path::const_iterator' does not provide a call
operator
llvm-svn: 362413
This fixes some thin archive relative path issues, paths are shortened where possible and paths are output correctly when using the display table command.
Differential Revision: https://reviews.llvm.org/D59491
llvm-svn: 362407
Summary:
When adding one thin archive to another, we currently chop off the relative path to the flattened members. For instance, when adding `foo/child.a` (which contains `x.txt`) to `parent.a`, when flattening it we should add it as `foo/x.txt` (which exists) instead of `x.txt` (which does not exist).
As a note, this also undoes the `IsNew` parameter of handling relative paths in r288280. The unit test there still passes.
This was reported as part of testing the kernel build with llvm-ar: https://patchwork.kernel.org/patch/10767545/ (see the second point).
Reviewers: mstorsjo, pcc, ruiu, davide, david2050, inglorion
Reviewed By: ruiu
Subscribers: void, jdoerfert, tpimh, mgorny, hans, nickdesaulniers, hiraditya, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D57842
llvm-svn: 353995
This broke the Chromium build on Windows, see https://crbug.com/930058
> Summary:
> When adding one thin archive to another, we currently chop off the relative path to the flattened members. For instance, when adding `foo/child.a` (which contains `x.txt`) to `parent.a`, whe
> lattening it we should add it as `foo/x.txt` (which exists) instead of `x.txt` (which does not exist).
>
> As a note, this also undoes the `IsNew` parameter of handling relative paths in r288280. The unit test there still passes.
>
> This was reported as part of testing the kernel build with llvm-ar: https://patchwork.kernel.org/patch/10767545/ (see the second point).
>
> Reviewers: mstorsjo, pcc, ruiu, davide, david2050
>
> Subscribers: hiraditya, llvm-commits
>
> Tags: #llvm
>
> Differential Revision: https://reviews.llvm.org/D57842
This reverts commit bf990ab5aa.
llvm-svn: 353507
Summary:
When adding one thin archive to another, we currently chop off the relative path to the flattened members. For instance, when adding `foo/child.a` (which contains `x.txt`) to `parent.a`, when flattening it we should add it as `foo/x.txt` (which exists) instead of `x.txt` (which does not exist).
As a note, this also undoes the `IsNew` parameter of handling relative paths in r288280. The unit test there still passes.
This was reported as part of testing the kernel build with llvm-ar: https://patchwork.kernel.org/patch/10767545/ (see the second point).
Reviewers: mstorsjo, pcc, ruiu, davide, david2050
Subscribers: hiraditya, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D57842
llvm-svn: 353424
to reflect the new license.
We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.
Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.
llvm-svn: 351636
Summary:
Import libraries as created by llvm-dlltool always use the same archive
member name for every object file (namely, the DLL library name). Ensure
that long names are not repeatedly stored in the string table.
Reviewed By: ruiu
Differential Revision: https://reviews.llvm.org/D55860
llvm-svn: 349637
* Support writing the DARWIN64 symbol table format.
* In darwin archives, emit a symbol table whenever requested, even
when there are no members, as the apple linker will abort if given
an archive without a symbol table.
Added tests for same, and also simplified and moved the GNU 64-bit
symbol table test into archive-symtab.test.
llvm-svn: 344183
This change ensures that the (membername,timestamp) tuple uniquely
identifies an entry in an archive for format=darwin, in deterministic
mode (which is the default).
That, then, enables lldb and dsymutil to locate the appropriate object
within the archive.
Differential Revision: https://reviews.llvm.org/D47659
llvm-svn: 343805
In this diff we adjust the code of getSymbols to avoid creating LLVMContext when it's not necessary.
Without this patch when the function getSymbols was called on a MachO object with a __bitcode section
it was parsing the embedded bitcode and then ignoring the result.
Test plan: make check-all
Differential revision: https://reviews.llvm.org/D51759
llvm-svn: 341998
Provide some free functions to reduce verbosity of endian-writing
a single value, and replace the endianness template parameter with
a field.
Part of PR37466.
Differential Revision: https://reviews.llvm.org/D47032
llvm-svn: 332757
LLVM_ON_WIN32 is set exactly with MSVC and MinGW (but not Cygwin) in
HandleLLVMOptions.cmake, which is where _WIN32 defined too. Just use the
default macro instead of a reinvented one.
See thread "Replacing LLVM_ON_WIN32 with just _WIN32" on llvm-dev and cfe-dev.
No intended behavior change.
This moves over all uses of the macro, but doesn't remove the definition
of it in (llvm-)config.h yet.
llvm-svn: 331127
This allows syntax like:
$ llvm-ar -c -r -u file.a file.o
This is in addition to the other formats that are already supported:
$ llvm-ar cru file.a file.o
$ llvm-ar -cru file.a file.o
Patch by Tom Anderson!
Differential Revision: https://reviews.llvm.org/D44452
llvm-svn: 328716
Even with the sparse file optimizations the SYM64 test can still be painfully
slow. This unnecessarily slows down devs. It's critical that we test that the
switch to the SYM64 format occurs at 4GB but there isn't any better of a way to
fake the size of the file than sparse files. This change introduces a flag that
allows the cutoff to be arbitrarily set to whatever power of two is desired.
The flag is hidden as it really isn't meant to be used outside this one test.
This is unfortunate but appears necessary, at least until the average hard
drive is much faster.
The changes to the test require some explanation. Prior to this change we knew
that the SYM64 format was being used because the file was simply too large to
have validly handled this case if the SYM64 format were not used. To ensure
that the SYM64 format is still being used I am grepping the file for "SYM64".
Without changing the filename however this would be pointless because "SYM64"
would occur in the file either way. So the filename of the test is also changed
in order to avoid this issue.
Differential Revision: https://reviews.llvm.org/D40632
llvm-svn: 319507
Tests were failing because some bots were running out of address
space and memory. Additionally the test was very slow. These issues
were solved by changing the test to take advantage of sparse filse and
restricting the test to run only on 64-bit systems.
This should fix https://bugs.llvm.org//show_bug.cgi?id=34189
This change makes it so that if writing a K_GNU style archive, you need
to output a > 32-bit offset it should output in K_GNU64 style instead.
Differential Revision: https://reviews.llvm.org/D36812
llvm-svn: 317352
An archive looks like
<header>
<symbol table>
<tail>
The symbol table refers to offsets in the tail. A complication is that
we would like to support symbol tables that use 64 bit offsets if it
turns out that any of the offsets is too big.
This patch changes the archive writer to first compute the tail. We
cannot just compute one big StringRef since that would require reading
every member upfront, but we can represent it as a series of
StringRefs.
Having done that it is much easier to compute the symbol table and all
offsets are computed before it is written. With this if there is an
accounting problem it will show up with a regular symbol table, not
just when a 64 bit one is needed.
llvm-svn: 314844
Keith Smiley