Ye olde PCRE (8.45) was end-of-lifed several years ago. For our bundled
regular expression implementation, we want to include the new,
still-maintained PCRE2 framework. Include PCRE2 v10.47.
This was caused by the tests being compiled with -D_FILE_OFFSET_BITS=64
which causes incompatibilities if libgit2 itself is not.
As this has been resolved the environment variable is no longer
necessary and can be removed.
On 32bit systems the git_fs_path_lstat from libgit2 by default uses 32bit stat
structs while the tests are being compiled with struct stat64 via
_FILE_OFFSET_BITS=64.
This discrepancy causes the "flaky" stat failures in tests.
The solution is to use the same _FILE_OFFSET_BITS as the library by
setting _FILE_OFFSET_BITS globally
Co-authored-by: Edward Thomson <ethomson@edwardthomson.com>
Make a distinction between generated headers and "translated" headers.
This is important to support build-time dependencies when headers are
updated.
Generated headers are those which contain build-time feature
specifications, like `git2_features.h` that are internal to the build
and `experimental.h` that contain API information.
Translated headers are the headers that are in `include/git2`, but may
be translated to have a unique prefix like `incklude/git2-experimental`.
This distinction is important so that the CMakeFiles.txt depend on the
in-tree include files (`src/include`) and the generated header files
_but not_ the translated header files. Otherwise there are two `pack.h`
and it's unclear whether the in-tree build is targeting the one in
`src/include` or the one in the build tree.
Without this, updating an in-tree header file like `pack.h` would not
cause a rebuild of its dependencies.
The implementation here seems to be sort of a copy
from the reference impl in RFC 6234 [2].
When multiple threads hash concurrently,
they race on this shared static variable.
It then corrupts the length-overflow detection,
and produces incorrect SHA-256 digests.
Here we replace it with a `static` function with a local variable.
The bug only affects the `GIT_SHA256_BUILTIN` backend.
The SHA-1 code path uses `sha1dc` which does not have this issue.
Reproducer:
```c
#include <stddef.h>
#include <stdio.h>
#include <string.h>
#include <pthread.h>
#include <git2.h>
#define NUM_THREADS 8
#define ITERATIONS 100000
static volatile int found_bug = 0;
void *hash_thread(void *arg) {
int id = *(int *)arg;
const char *data = "hello world\n";
size_t len = strlen(data);
git_object_id_options opts = GIT_OBJECT_ID_OPTIONS_INIT;
opts.object_type = GIT_OBJECT_BLOB;
opts.oid_type = GIT_OID_SHA256;
git_oid reference, result;
git_object_id_from_buffer(&reference, data, len, &opts);
for (int i = 0; i < ITERATIONS && !found_bug; i++) {
git_object_id_from_buffer(&result, data, len, &opts);
if (!git_oid_equal(&reference, &result)) {
found_bug = 1;
printf("BUG: thread %d, iteration %d\n", id, i);
break;
}
}
return NULL;
}
int main(void) {
git_libgit2_init();
pthread_t threads[NUM_THREADS];
int ids[NUM_THREADS];
for (int i = 0; i < NUM_THREADS; i++) {
ids[i] = i;
pthread_create(&threads[i], NULL, hash_thread, &ids[i]);
}
for (int i = 0; i < NUM_THREADS; i++)
pthread_join(threads[i], NULL);
if (!found_bug)
printf("No bug triggered\n");
git_libgit2_shutdown();
return found_bug ? 1 : 0;
}
```
Build and run (from libgit2 repo root):
```sh
mkdir build && cd build
cmake .. -DEXPERIMENTAL_SHA256=ON -DUSE_SHA256=Builtin \
-DUSE_HTTPS=OFF -DUSE_SSH=OFF -DUSE_NTLMCLIENT=OFF \
-DBUILD_SHARED_LIBS=OFF -DCMAKE_BUILD_TYPE=Debug
make libgit2package
cd ..
cc -O0 -pthread -DGIT_EXPERIMENTAL_SHA256=1 \
-I include -o repro repro.c \
build/libgit2-experimental.a -lz -lpcre2-8
./repro
```
See <https://github.com/rust-lang/git2-rs/issues/1255> for more.
[1]: https://github.com/libgit2/libgit2/blob/1affb8b19/src/util/hash/rfc6234/sha224-256.c#L86-L91
[2]: https://www.rfc-editor.org/rfc/rfc6234#section-8.2.2
PR #7202 (`1ab42f3`) accidentally dropped the `PUBLIC` in the includes;
this meant that local build was accidentally looking at the in-build
include files instead of the in-source include files, and updates to
source include files would not trigger a rebuild.
Git supports relative worktrees since Git v2.48 - cf6f63ea6b/Documentation/RelNotes/2.48.0.adoc (L57)
This was already handled programatically in libgit2, but was
not recognized as an extension, meaning downstream consumers
like Nix had issues with relative worktree-enabled repos.
Fixes#7210
Now that we have two types of object IDs, with different sizes, we
expect shorter object ID types (in other words, SHA1 object ids) to be
zero-padded at their suffix. This allows us to use faster comparison and
copy routines over the entirety of the structure, instead of trying to
examine the type and then do a comparison of the appropriately sized
structure.
For pure manipulation of object IDs, this produces parity with the
SHA1-only object ID code.
SHA1:
oid::cmp_sha1: 8.065 ms ± 703.9 μs / range: 7.875 ms … 14.88 ms (201 runs)
oid::cmp_sha256: skipped
oid::cpy_sha1: 5.340 ms ± 47.26 μs / range: 5.272 ms … 5.617 ms (548 runs)
oid::cpy_sha256: skipped
oid::zero_sha1: 5.327 ms ± 49.27 μs / range: 5.271 ms … 5.612 ms (553 runs)
oid::zero_sha256: skipped
SHA256 (before this change; testing the `type`):
oid::cmp_sha1: 10.82 ms ± 1.029 ms / range: 10.57 ms … 20.63 ms (145 runs)
oid::cmp_sha256: 10.63 ms ± 103.9 μs / range: 10.50 ms … 11.48 ms (279 runs)
oid::cpy_sha1: 26.13 ms ± 63.91 μs / range: 26.07 ms … 26.45 ms (113 runs)
oid::cpy_sha256: 20.92 ms ± 58.32 μs / range: 20.86 ms … 21.25 ms (141 runs)
oid::zero_sha1: 13.19 ms ± 129.1 μs / range: 13.11 ms … 13.72 ms (224 runs)
oid::zero_sha256: 13.12 ms ± 30.06 μs / range: 13.10 ms … 13.30 ms (225 runs)
SHA256 (with this change):
oid::cmp_sha1: 7.985 ms ± 562.3 μs / range: 7.874 ms … 14.32 ms (209 runs)
oid::cmp_sha256: 6.609 ms ± 30.77 μs / range: 6.584 ms … 6.870 ms (443 runs)
oid::cpy_sha1: 5.282 ms ± 21.90 μs / range: 5.266 ms … 5.524 ms (543 runs)
oid::cpy_sha256: 5.279 ms ± 17.57 μs / range: 5.263 ms … 5.415 ms (554 runs)
oid::zero_sha1: 5.288 ms ± 22.92 μs / range: 5.268 ms … 5.508 ms (544 runs)
oid::zero_sha256: 5.286 ms ± 21.29 μs / range: 5.271 ms … 5.527 ms (542 runs)
