InTheForest/moby
1
0
Fork 0
mirror of https://github.com/moby/moby.git synced 2026-01-11 18:51:37 +00:00
Code Issues Packages Projects Releases Wiki Activity

daemon: clean up dead containers on start

Browse Source 
Stopping the Engine while a container with autoremove set is running may
leave behind dead containers on disk. These containers aren't reclaimed
on next start, appear as "dead" in `docker ps -a` and can't be
inspected or removed by the user.

This bug has existed since a long time but became user visible with
9f5f4f5a42. Prior to that commit,
containers with no rwlayer weren't added to the in-memory viewdb, so
they weren't visible in `docker ps -a`. However, some dangling files
would still live on disk (e.g. folder in /var/lib/docker/containers,
mount points, etc).

The underlying issue is that when the daemon stops, it tries to stop all
running containers and then closes the containerd client. This leaves a
small window of time where the Engine might receive 'task stop' events
from containerd, and trigger autoremove. If the containerd client is
closed in parallel, the Engine is unable to complete the removal,
leaving the container in 'dead' state. In such case, the Engine logs the
following error:

    cannot remove container "bcbc98b4f5c2b072eb3c4ca673fa1c222d2a8af00bf58eae0f37085b9724ea46": Canceled: grpc: the client connection is closing: context canceled

Solving the underlying issue would require complex changes to the
shutdown sequence. Moreover, the same issue could also happen if the
daemon crashes while it deletes a container. Thus, add a cleanup step
on daemon startup to remove these dead containers.

Signed-off-by: Albin Kerouanton <albin.kerouanton@docker.com>
This commit is contained in:
Albin Kerouanton
2025-12-11 17:42:21 +01:00
committed by Austin Vazquez
parent a11c3b9a89
commit ec9315cd4f
2 changed files with 36 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files

24
integration/container/remove_test.go
View File

@@ -8,6 +8,7 @@ import (
containertypes "github.com/moby/moby/api/types/container"
"github.com/moby/moby/client"
"github.com/moby/moby/v2/integration/internal/container"
"github.com/moby/moby/v2/internal/testutil/daemon"
"gotest.tools/v3/assert"
is "gotest.tools/v3/assert/cmp"
"gotest.tools/v3/fs"
@@ -107,3 +108,26 @@ func TestRemoveInvalidContainer(t *testing.T) {
assert.Check(t, is.ErrorType(err, cerrdefs.IsNotFound))
assert.Check(t, is.ErrorContains(err, "No such container"))
}
func TestRemoveDeadContainersOnDaemonRestart(t *testing.T) {
skip.If(t, testEnv.IsRemoteDaemon)
skip.If(t, testEnv.DaemonInfo.OSType == "windows", "FIXME: Windows CI does not support multiple daemons yet")
ctx := setupTest(t)
d := daemon.New(t)
d.StartWithBusybox(ctx, t)
defer d.Stop(t)
apiClient := d.NewClientT(t)
container.Run(ctx, t, apiClient, container.WithCmd("top"), container.WithAutoRemove)
list, err := apiClient.ContainerList(ctx, client.ContainerListOptions{All: true})
assert.NilError(t, err)
assert.Check(t, is.Len(list.Items, 1))
d.Restart(t)
list, err = apiClient.ContainerList(ctx, client.ContainerListOptions{All: true})
assert.NilError(t, err)
assert.Check(t, is.Len(list.Items, 0))
}