This was caused by the tests being compiled with -D_FILE_OFFSET_BITS=64
which causes incompatibilities if libgit2 itself is not.
As this has been resolved the environment variable is no longer
necessary and can be removed.
On 32bit systems the git_fs_path_lstat from libgit2 by default uses 32bit stat
structs while the tests are being compiled with struct stat64 via
_FILE_OFFSET_BITS=64.
This discrepancy causes the "flaky" stat failures in tests.
The solution is to use the same _FILE_OFFSET_BITS as the library by
setting _FILE_OFFSET_BITS globally
Co-authored-by: Edward Thomson <ethomson@edwardthomson.com>
Make a distinction between generated headers and "translated" headers.
This is important to support build-time dependencies when headers are
updated.
Generated headers are those which contain build-time feature
specifications, like `git2_features.h` that are internal to the build
and `experimental.h` that contain API information.
Translated headers are the headers that are in `include/git2`, but may
be translated to have a unique prefix like `incklude/git2-experimental`.
This distinction is important so that the CMakeFiles.txt depend on the
in-tree include files (`src/include`) and the generated header files
_but not_ the translated header files. Otherwise there are two `pack.h`
and it's unclear whether the in-tree build is targeting the one in
`src/include` or the one in the build tree.
Without this, updating an in-tree header file like `pack.h` would not
cause a rebuild of its dependencies.
The implementation here seems to be sort of a copy
from the reference impl in RFC 6234 [2].
When multiple threads hash concurrently,
they race on this shared static variable.
It then corrupts the length-overflow detection,
and produces incorrect SHA-256 digests.
Here we replace it with a `static` function with a local variable.
The bug only affects the `GIT_SHA256_BUILTIN` backend.
The SHA-1 code path uses `sha1dc` which does not have this issue.
Reproducer:
```c
#include <stddef.h>
#include <stdio.h>
#include <string.h>
#include <pthread.h>
#include <git2.h>
#define NUM_THREADS 8
#define ITERATIONS 100000
static volatile int found_bug = 0;
void *hash_thread(void *arg) {
int id = *(int *)arg;
const char *data = "hello world\n";
size_t len = strlen(data);
git_object_id_options opts = GIT_OBJECT_ID_OPTIONS_INIT;
opts.object_type = GIT_OBJECT_BLOB;
opts.oid_type = GIT_OID_SHA256;
git_oid reference, result;
git_object_id_from_buffer(&reference, data, len, &opts);
for (int i = 0; i < ITERATIONS && !found_bug; i++) {
git_object_id_from_buffer(&result, data, len, &opts);
if (!git_oid_equal(&reference, &result)) {
found_bug = 1;
printf("BUG: thread %d, iteration %d\n", id, i);
break;
}
}
return NULL;
}
int main(void) {
git_libgit2_init();
pthread_t threads[NUM_THREADS];
int ids[NUM_THREADS];
for (int i = 0; i < NUM_THREADS; i++) {
ids[i] = i;
pthread_create(&threads[i], NULL, hash_thread, &ids[i]);
}
for (int i = 0; i < NUM_THREADS; i++)
pthread_join(threads[i], NULL);
if (!found_bug)
printf("No bug triggered\n");
git_libgit2_shutdown();
return found_bug ? 1 : 0;
}
```
Build and run (from libgit2 repo root):
```sh
mkdir build && cd build
cmake .. -DEXPERIMENTAL_SHA256=ON -DUSE_SHA256=Builtin \
-DUSE_HTTPS=OFF -DUSE_SSH=OFF -DUSE_NTLMCLIENT=OFF \
-DBUILD_SHARED_LIBS=OFF -DCMAKE_BUILD_TYPE=Debug
make libgit2package
cd ..
cc -O0 -pthread -DGIT_EXPERIMENTAL_SHA256=1 \
-I include -o repro repro.c \
build/libgit2-experimental.a -lz -lpcre2-8
./repro
```
See <https://github.com/rust-lang/git2-rs/issues/1255> for more.
[1]: https://github.com/libgit2/libgit2/blob/1affb8b19/src/util/hash/rfc6234/sha224-256.c#L86-L91
[2]: https://www.rfc-editor.org/rfc/rfc6234#section-8.2.2
PR #7202 (`1ab42f3`) accidentally dropped the `PUBLIC` in the includes;
this meant that local build was accidentally looking at the in-build
include files instead of the in-source include files, and updates to
source include files would not trigger a rebuild.
Git supports relative worktrees since Git v2.48 - cf6f63ea6b/Documentation/RelNotes/2.48.0.adoc (L57)
This was already handled programatically in libgit2, but was
not recognized as an extension, meaning downstream consumers
like Nix had issues with relative worktree-enabled repos.
Fixes#7210
Now that we have two types of object IDs, with different sizes, we
expect shorter object ID types (in other words, SHA1 object ids) to be
zero-padded at their suffix. This allows us to use faster comparison and
copy routines over the entirety of the structure, instead of trying to
examine the type and then do a comparison of the appropriately sized
structure.
For pure manipulation of object IDs, this produces parity with the
SHA1-only object ID code.
SHA1:
oid::cmp_sha1: 8.065 ms ± 703.9 μs / range: 7.875 ms … 14.88 ms (201 runs)
oid::cmp_sha256: skipped
oid::cpy_sha1: 5.340 ms ± 47.26 μs / range: 5.272 ms … 5.617 ms (548 runs)
oid::cpy_sha256: skipped
oid::zero_sha1: 5.327 ms ± 49.27 μs / range: 5.271 ms … 5.612 ms (553 runs)
oid::zero_sha256: skipped
SHA256 (before this change; testing the `type`):
oid::cmp_sha1: 10.82 ms ± 1.029 ms / range: 10.57 ms … 20.63 ms (145 runs)
oid::cmp_sha256: 10.63 ms ± 103.9 μs / range: 10.50 ms … 11.48 ms (279 runs)
oid::cpy_sha1: 26.13 ms ± 63.91 μs / range: 26.07 ms … 26.45 ms (113 runs)
oid::cpy_sha256: 20.92 ms ± 58.32 μs / range: 20.86 ms … 21.25 ms (141 runs)
oid::zero_sha1: 13.19 ms ± 129.1 μs / range: 13.11 ms … 13.72 ms (224 runs)
oid::zero_sha256: 13.12 ms ± 30.06 μs / range: 13.10 ms … 13.30 ms (225 runs)
SHA256 (with this change):
oid::cmp_sha1: 7.985 ms ± 562.3 μs / range: 7.874 ms … 14.32 ms (209 runs)
oid::cmp_sha256: 6.609 ms ± 30.77 μs / range: 6.584 ms … 6.870 ms (443 runs)
oid::cpy_sha1: 5.282 ms ± 21.90 μs / range: 5.266 ms … 5.524 ms (543 runs)
oid::cpy_sha256: 5.279 ms ± 17.57 μs / range: 5.263 ms … 5.415 ms (554 runs)
oid::zero_sha1: 5.288 ms ± 22.92 μs / range: 5.268 ms … 5.508 ms (544 runs)
oid::zero_sha256: 5.286 ms ± 21.29 μs / range: 5.271 ms … 5.527 ms (542 runs)
When we were done reading headers, we checked if we needed to read a
body, or if we were done. The body check was done by looking at the
transfer encoding type and the content type. If we were chunked, then we
know we have a body (it may be a zero byte body, but we would need to
read the chunk length to know this). But looking at the content _type_
was erroneous; we should have been looking at the content _length_.
The effect of this is that when a server sends a zero byte response
with a content _type_, we try to go read the body, which does not exist.
We will hang waiting for the body that the server will never send.
Correct this typo. Now we will try to read the body if there was a
content _length_ specified, or if the transfer encoding is chunked.
poxygit now supports a "specification" within the URI that can provide
additional details about the mock/debugging connection. The `:none`
suffix on the redirect request indicates that the proxy should send a 0
byte response body.
Update our CI to use poxygit v0.8.1, which has additional mocking and
debugging capabilities.
As part of this, the paths to the `speed` test routes changed - now they
are `speed:<n>` where `<n>` is the speed to emulate in bps.
