Upstream Mirrors
A repository can mirror an upstream package repository.
Paquetier copies what changed upstream into the repository’s own storage and serves it under /cran/{slug} like any other repository, so R installs from your storage and never reaches the upstream.
The whole sync runs inside Paquetier: no rsync job, cron container or extra tool is needed next to it.
Mirrored packages are imported, not uploaded, so they count against no package or repository limit.
A Python repository mirrors a simple index such as PyPI instead; see Mirroring PyPI.
Supported Upstreams
| Upstream | Examples |
|---|---|
| Repository URL | CRAN (https://cloud.r-project.org), a CRAN mirror, R-universe (https://<user>.r-universe.dev), another Paquetier (https://<host>/cran/<slug>) |
| Storage connection | A prefix in a bucket that holds a CRAN layout, read the way a scan reads it |
A repository URL is the one R would use in options(repos = ...).
Mirroring from a storage connection takes the connection’s owner or an admin.
Setting Up a Mirror
- Open the repository and go to Settings.
- Under Upstream mirror, fill in the settings below.
- Click Set up mirror, then Sync now to start the first sync instead of waiting for the schedule.
| Setting | Description |
|---|---|
| Upstream | A repository URL (HTTP or HTTPS) or a storage connection |
| Repository URL | The upstream’s root, such as https://cloud.r-project.org |
| Storage connection, Prefix | The connection and the directory that holds src/contrib; an empty prefix is the bucket root |
| Index paths | One per line: src/contrib for sources, bin/<platform>/contrib/<R minor> for binaries, such as bin/windows/contrib/4.5 or bin/macosx/big-sur-arm64/contrib/4.5 |
| Packages to mirror | Selected packages and what they need to install, or everything in the upstream index |
| Packages | The packages to start from, separated by commas or new lines, at most 2000 |
| Copy older releases on the first sync | Also copies each package’s upstream Archive/ once |
| Sync on a schedule | Checks the upstream regularly and syncs when it changed |
| Check every (minutes) | How often the upstream is checked, at least 5 minutes, 60 by default |
| Parallel downloads | Files copied at once, 8 by default, at most 64 |
| Requests per second | A cap on requests to a repository URL; 0 is unlimited |
Configuring or removing a mirror takes a repository owner or an admin. Anyone with write access can see its settings and start a sync.
Each index path keeps its layout under the repository, so R finds it under /cran/{slug}/src/contrib and /cran/{slug}/bin/... without any extra configuration.
Linux binary layouts such as Posit Package Manager’s __linux__/<distro>/latest/src/contrib are not supported yet.
Choosing What to Mirror
With Selected packages and what they need to install, Paquetier starts from the packages you list and follows their Depends, Imports and LinkingTo in the upstream index until nothing new turns up.
Suggests is left out, since following it pulls in a large part of CRAN.
A dependency the upstream does not have and that is not part of R is listed under missing_dependencies in the sync’s result.
With Everything in the upstream index, the whole index is copied. All of CRAN’s sources are about 22,000 packages and several GB, and its binaries are much larger, so start with a list unless the repository really is meant to be a full mirror.
How a Sync Works
For each index path, in order:
- Check: the upstream’s
PACKAGES.gz(orPACKAGES) is checked against what the last sync saw. An unchanged upstream costs one request per index and copies nothing. - Compare: every package in scope is compared by name, version and MD5 checksum with what the repository already holds. Only new or changed files are downloaded.
- Copy: each file streams from the upstream straight into the repository’s storage and is checked against the upstream’s MD5 checksum. A file that does not match is discarded. Failed downloads are retried with backoff.
- Archive: a release that drops out of the upstream index moves to
Archive/<package>/, as CRAN moves it, instead of being deleted. An upstream index with nothing in scope archives nothing, so a broken upstream cannot empty a mirror. - Publish: once every file is in place, the repository’s
PACKAGESindexes are rewritten, so R never sees a package whose file is still on its way.
A sync shows up in the repository’s activity with live progress, like a scan. Only one sync runs per repository at a time. If the instance running it stops, another instance picks it up, and the new run skips every file that was already copied.
When the sync finishes, Paquetier reads authors, URLs and bug report links from the copied packages, since CRAN’s PACKAGES does not carry them.
Schedule and Failures
With Sync on a schedule, Paquetier checks the upstream at the chosen interval and starts a sync only when an index changed, the mirror’s settings changed or the last sync failed. A mirror in step with its upstream leaves nothing in the activity feed.
If the upstream index cannot be read, the sync fails and the repository keeps serving what it had. If some files fail to download, the rest are still published, the mirror’s Last error says how many failed, and the next check starts a sync that fetches only what is still missing.
What a Mirror Owns
A mirror owns the files under its index paths. It never overwrites a package that was uploaded: an upload with the same file name is kept and the sync reports it as a failure. Removing a mirror stops syncing and leaves everything it copied in the repository.
API
| Method | Path | Description |
|---|---|---|
GET |
/api/v1/repos/{slug}/mirror |
The mirror’s settings and the state of its last sync |
PUT |
/api/v1/repos/{slug}/mirror |
Create or change the mirror |
DELETE |
/api/v1/repos/{slug}/mirror |
Remove the mirror |
POST |
/api/v1/repos/{slug}/mirror/sync |
Start a sync; returns a job_id whose progress streams at /api/v1/jobs/{id}/stream |
The request fields are upstream_type (http or storage), upstream_url, storage_connection_id, upstream_prefix, paths, scope (list or all), packages, backfill_archive, enabled, sync_interval_seconds, concurrency and rate_limit.
Private Networks
By default a repository URL may not resolve to a private, loopback, link-local or carrier-grade NAT address, since the server fetches whatever URL a repository owner enters.
To mirror a Paquetier or CRAN mirror on your own network, an administrator sets PAQUETIER_MIRROR_ALLOW_PRIVATE_UPSTREAMS (see Configuration).
