Move the progress package up into the client as a temporary shared location for
common clients like CLI and compose.
The progress package is used by the daemon to write progress updates to
some sink, typically a streamformatter. This package is of little use to
API clients as this package does not provide any facilities to consume
the progress updates.
Co-authored-by: Cory Snider <csnider@mirantis.com>
Signed-off-by: Austin Vazquez <austin.vazquez@docker.com>
The stringid package is used in many places; while it's trivial
to implement a similar utility, let's just provide it as a utility
package in the client, removing the daemon-specific logic.
For integration tests, I opted to use the implementation in the
client, as those should not ideally not make assumptions about
the daemon implementation.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Pulling some images that share the same content blob but have different chain
IDs caused a panic:
```
panic: runtime error: slice bounds out of range [1:0]
goroutine 318661 [running]:
github.com/docker/docker/daemon/containerd.(*pullProgress).UpdateProgress(0x400fd02d70, {0xaaaada2fda38, 0x400fd02e10}, 0x4019d38810, {0xaaaada2d1640, 0x4018c94600}, {0x0?, 0x0?, 0xaaaadb7c7200?})
/root/build-deb/engine/daemon/containerd/progress.go:232 +0xd84
github.com/docker/docker/daemon/containerd.(*jobs).showProgress.func1()
/root/build-deb/engine/daemon/containerd/progress.go:55 +0x144
created by github.com/docker/docker/daemon/containerd.(*jobs).showProgress in goroutine 318659
/root/build-deb/engine/daemon/containerd/progress.go:48 +0x128
```
The panic was caused by attempting to remove the same committed
layer multiple times from the `p.layers` slice.
This occurred because, in such images, multiple snapshots matched the
same layer by digest rather than by the full layer chain and layer removal
was done by index, leading to repeated deletions at the same index.
This commit:
- Selects a specific snapshot to ensure only one removal per layer.
- Changes snapshot matching to compare the full layer chain instead of
just the layer digest.
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
Before this patch, pull progress wouldn't show the `Extracting` layer
status which made the pull look like it got stuck when extracting a big
layer.
Use the `containerd.io/snapshot/cri.layer-digest` snapshot labels to
find a corresponding snapshot and check whether it's `active` or
`committed` to set the layer status accordingly.
Despite the `cri.` component in the label name, it's not CRI specific -
it only depends on the `snapshotters.AppendInfoHandlerWrapper`.
We _could_ also use the `Usage` snapshot method to query the exact
progress of the unpack, but it would be too expensive as the
implementation time complexity will be proportional to the snapshot size.
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
This internal package was added in f6e44bc0e8
to preserve compatibility with go1.20 and older. At the time, our vendor.mod
still had go1.18 as minimum version requirement (see [1]), which got updated to go1.20
in 16063c7456, and go1.21 in f90b03ee5d
The version of BuildKit we use already started using context.WithoutCancel,
without a fallback, so we no longer can provide compatibility with older
versions of Go, which makes our compatiblity package redundant.
This patch removes the package, and updates our code to use stdlib's context
instead.
[1]: f6e44bc0e8/vendor.mod (L7)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
They might still change to "Mounted from" or "Already exists" when
containerd updates the status in tracker.
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
Add a OCI platform fields as parameters to the `POST /images/{id}/push`
that allow to specify a specific-platform manifest to be pushed instead
of the whole image index.
When no platform was requested and pushing whole index failed, fallback
to pushing a platform-specific manifest with a best candidate (if it's
possible to choose one).
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
Update to containerd 1.7.18, which now migrated to the errdefs module. The
existing errdefs package is now an alias for the module, and should no longer
be used directly.
This patch:
- updates the containerd dependency: https://github.com/containerd/containerd/compare/v1.7.17...v1.7.18
- replaces uses of the old package in favor of the new module
- adds a linter check to prevent accidental re-introduction of the old package
- adds a linter check to prevent using the "log" package, which was also
migrated to a separate module.
There are still some uses of the old package in (indirect) dependencies,
which should go away over time.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The github.com/containerd/containerd/log package was moved to a separate
module, which will also be used by upcoming (patch) releases of containerd.
This patch moves our own uses of the package to use the new module.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Instead of passing a completely fresh context without any values, just
discard the cancellation.
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
- check if we have to download layers and print the approriate message
- show the digest of the pulled manifest(list)
- skip pulling if we already have the right manifest
Signed-off-by: Djordje Lukic <djordje.lukic@docker.com>
Docker with containerd integration emits "Exists" progress action when a
layer of the currently pulled image already exists. This is different
from the non-c8d Docker which emits "Already exists".
This makes both implementations consistent by emitting backwards
compatible "Already exists" action.
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
- make jobs.Add accept a list of jobs, so that we don't have to
repeatedly lock/unlock the mutex
- rename some variables that collided with imports or types
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This implements `docker push` under containerd image store. When
pushing manifest lists that reference a content which is not present in
the local content store, it will attempt to perform the cross-repo mount
the content if possible.
Considering this scenario:
```bash
$ docker pull docker.io/library/busybox
```
This will download manifest list and only host platform-specific
manifest and blobs.
Note, tagging to a different repository (but still the same registry) and pushing:
```bash
$ docker tag docker.io/library/busybox docker.io/private-repo/mybusybox
$ docker push docker.io/private-repo/mybusybox
```
will result in error, because the neither we nor the target repository
doesn't have the manifests that the busybox manifest list references
(because manifests can't be cross-repo mounted).
If for some reason the manifests and configs for all other platforms
would be present in the content store, but only layer blobs were
missing, then the push would work, because the blobs can be cross-repo
mounted (only if we push to the same registry).
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
After the context is cancelled, update the progress for the last time.
This makes sure that the progress also includes finishing updates.
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
