Independent compatibility study

Result

GrantTrace was evaluated against source and a mocked runtime without GitHub credentials from the independently maintained All Contributors Bot, an MIT-licensed Probot GitHub App. The study pins upstream commit 00f6362ffcc927a2d05fec27f42c3d09e4b03adb and verifies SHA-256 hashes for every inspected upstream file. The bounded verifier also checks that every declared route source URL names that repository and commit, a hashed file and line range, and text containing the declared Octokit source call.

The result is mixed in a useful way:

• GrantTrace resolves the pinned source's standalone comment-reply route and preserves GitHub's issues: write or pull_requests: write choice.
• The pinned App's real Probot handler completed its upstream-mocked happy path and emitted ten Octokit requests across seven unique normalized routes. Nock rejected unmatched Node HTTP(S) requests on the observed Octokit path, and GrantTrace's real recorder captured all ten requests.
• Three unique runtime routes are covered and four are catalog gaps. GrantTrace blocks the full analysis instead of presenting a partial permission result as sufficient.
• The resolvable PR-path subset does not add issues: write, because pull_requests: write is already needed to open the pull request and also satisfies the comment route. That is a concrete permission assumption to review, not evidence that the App can drop the grant globally.

This is a source-pinned compatibility evaluation with mocked runtime evidence, not a live GitHub integration, an All Contributors endorsement, or an adoption claim.

Why this App

The candidate was selected before interpreting the permission result:

• it is a real public GitHub App outside the GrantTrace project;
• the pinned source has an explicit GitHub App manifest, a clear Octokit call surface, and an MIT license;
• its issue_comment.created handler leads to both a small independently reviewable reply route and a multi-route pull request workflow; and
• it exercises covered routes and genuine catalog gaps, making the study more informative than a fixture chosen only because every route already passes.

Evidence layers

The case study deliberately separates four different claims.

1. Pinned upstream source inspection

case-studies/all-contributors-app/case-study.json records the repository, commit, license, declared permissions, source-linked route inventory, and whole-file hashes for the route-defining and transitive source-flow files. The bounded network verifier downloads those files from the immutable commit and checks their hashes:

node scripts/verify-external-case-study.mjs

This establishes which source was inspected. It does not establish that the code path ran, and no fixture line is a runtime observation of the App. Route mapping is mechanically bound to the hashed source range and declared call; the mapping from that call to a canonical REST route remains human-reviewed.

2. Fixture analysis without GitHub credentials

Two strict NDJSON fixtures feed the canonical, source-derived route set through GrantTrace's real parser, evidence resolver, and permission solver:

pnpm granttrace analyze \
  case-studies/all-contributors-app/reply-only.observations.ndjson

# Expected exit: 7 (evidence blocked by catalog gaps)
pnpm granttrace analyze \
  case-studies/all-contributors-app/new-branch-pr.observations.ndjson

The fixtures intentionally use status: null, carry no runtime permission header, and rely on catalog-only resolution. They do not invent a GitHub response, token, repository, runtime observation, or successful live run. Offline compatibility assertions lock the expected frontier, covered route set, blocking findings, fixture-to-manifest counts, and source manifest:

pnpm vitest run test/integration/external-case-study.test.ts

Each fixture has one record per distinct route or catalog gap in its scenario. It tests GrantTrace's deduplicated contract behavior; it does not claim the App would make each call exactly once.

3. Mocked runtime execution without GitHub credentials

The runtime pilot checks out the exact commit, verifies the pinned package-lock.json, installs it with npm ci --ignore-scripts, and loads the upstream App and fixtures with upstream Probot 12.2.8. It adapts the pinned upstream happy-path test by composing GrantTrace's Octokit request hook with ProbotOctokit. The App then receives the real issue_comment.created payload and executes its real application and Octokit request paths.

nock.disableNetConnect() rejects unmatched Node HTTP(S) requests for clients it intercepts, including the observed Octokit path. A child process receives no GitHub credential environment variables; Probot gets only the upstream test's literal dummy token test. In this study, credential-free means no GitHub credentials were passed. The harness requires every expected mock to be consumed and compares the recorder output and normalized request trace to the checked-in assets:

pnpm install --frozen-lockfile --ignore-scripts
pnpm case-study:runtime

runtime.observations.ndjson is the real GrantTrace recorder artifact. runtime-emitted-requests.json is a companion Octokit-hook artifact that keeps only method, normalized route template, and mocked response status after a request completes. This companion is necessary because GrantTrace intentionally drops the template for catalog gaps. Neither artifact retains concrete route values, request data, credentials, or response bodies.

Nock is not an OS network or filesystem sandbox. The pinned third-party code and its dependencies can still read accessible files, spawn processes, or use network channels nock does not intercept. Reproduce this pilot only in a disposable container or VM when those host capabilities are not acceptable.

The mocked responses do not contain GitHub's x-accepted-github-permissions header. Covered routes therefore resolve from GrantTrace's pinned catalog, while uncovered routes remain blocked. This is runtime route evidence, not a simulation of live GitHub permission headers.

4. Live and maintainer evidence not obtained

No GitHub App registration, credentials, live GitHub request, or live repository mutation was used. No maintainer was contacted: feedback is unavailable and was not solicited. The study therefore says nothing about unobserved branches, production configuration, live GitHub response behavior, or whether the maintainers would adopt the workflow.

No new negative controls were added for this artifact. Permission-removal controls require a real adopter scenario and disposable live fixture so that authorization rejection and mutation cleanup can be observed safely. Neither source-derived fixtures nor responses mocked by the upstream test can provide that evidence, so treating those controls as complete here would overstate the study.

Route inventory

The runtime “new branch and pull request” path read configured files, looked up a contributor, created a branch, updated two files, opened a pull request, and sent the reply.

The runtime emitted ten calls across those seven unique templates. The four gaps collapse to three safe findings in the contract (GET, POST, and PUT) because GrantTrace does not retain the template in an unresolved observation. The companion runtime inventory retains only canonical parameterized templates so catalog work remains reviewable without storing concrete owners, repositories, usernames, refs, paths, or payload data.

Permission result

The pinned App manifest declares contents: write, issues: write, metadata: read, and pull_requests: write.

The source-derived standalone reply fixture produces this complete frontier:

issues: write
OR
pull_requests: write

For the catalog-resolvable subset of both the source fixture and the executed runtime workflow, the solver retains one nondominated assignment:

contents: read
pull_requests: write

In that subset, the comment route did not add issues: write: the existing pull-request grant satisfies it. This demonstrates why a permission frontier matters. However, analysis of the full executed scenario is blocked. In particular, two actually emitted PUT /contents/{path} calls are runtime evidence of the catalog gap around the App's contents writes, and the webhook handler can receive comments on issues as well as pull requests. It would be unsafe to turn the subset result into an App-level permission reduction.

Setup time

These are wall-clock measurements from the captured researcher run on Darwin 27.0.0 arm64 with Node 26.0.0, npm 11.12.1, and pnpm 9.15.4. They are not a benchmark or maintainer-onboarding estimate:

The exact measured command shapes and elapsed seconds are retained in runtime-case-study.json; machine-specific paths are replaced with stable <granttrace-checkout>, <temporary-root>, and <pinned-upstream-checkout> placeholders. The one-command total includes orchestration, evidence comparison, and cleanup, so it is larger than the listed child phase. The local pnpm was 9.15.4 rather than the package manifest's declared 10.28.2; it accepted the frozen lockfile unchanged, but that version mismatch is part of the captured environment rather than an exact package-manager reproduction. The upstream frozen install reported 62 audit findings (9 low, 31 moderate, 20 high, and 2 critical); no dependency update or audit fix was attempted because that would violate the pinned dependency execution. Time spent understanding and adapting the upstream source was not measured prospectively. Maintainer setup time remains unknown because feedback was not solicited.

What should happen next

The four source-backed gaps give catalog expansion a concrete workflow-based order: contents writes and Git reference creation enable the core mutation path; the global user lookup needs a separate product decision because GrantTrace currently limits its claim to repository-scoped permissions.

Maintainer feedback, if requested later with explicit authorization, should focus on observable questions:

• Which existing upstream test best represents a comment on a pull request?
• Which additional upstream scenarios should be executed before anyone reasons about application-level permission reduction?
• Does the permission frontier make the manifest review clearer?
• Which blocked route would prevent trial adoption first?

Until those answers exist, this page remains a source-pinned, mocked-runtime compatibility study—not a user testimonial.