The remote API allows full privilege escalation and is equivalent to
having root access on the host. Because of this, the API should never
be accessible through an insecure connection (TCP without TLS, or TCP
without TLS verification).
Although a warning is already logged on startup if the daemon uses an
insecure configuration, this warning is not very visible (unless someone
decides to read the logs).
This patch attempts to make insecure configuration more visible by sending
back warnings through the API (which will be printed when using `docker info`).
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
When requesting information about the daemon's configuration through the `/info`
endpoint, missing features (or non-recommended settings) may have to be presented
to the user.
Detecting these situations, and printing warnings currently is handled by the
cli, which results in some complications:
- duplicated effort: each client has to re-implement detection and warnings.
- it's not possible to generate warnings for reasons outside of the information
returned in the `/info` response.
- cli-side detection has to be updated for new conditions. This means that an
older cli connecting to a new daemon may not print all warnings (due to
it not detecting the new conditions)
- some warnings (in particular, warnings about storage-drivers) depend on
driver-status (`DriverStatus`) information. The format of the information
returned in this field is not part of the API specification and can change
over time, resulting in cli-side detection no longer being functional.
This patch adds a new `Warnings` field to the `/info` response. This field is
to return warnings to be presented by the user.
Existing warnings that are currently handled by the CLI are copied to the daemon
as part of this patch; This change is backward-compatible with existing
clients; old client can continue to use the client-side warnings, whereas new
clients can skip client-side detection, and print warnings that are returned by
the daemon.
Example response with this patch applied;
```bash
curl --unix-socket /var/run/docker.sock http://localhost/info | jq .Warnings
```
```json
[
"WARNING: bridge-nf-call-iptables is disabled",
"WARNING: bridge-nf-call-ip6tables is disabled"
]
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Blocks the execution of tasks during the Prepare phase until there
exists an IP address for every overlay network in use by the task. This
prevents a task from starting before the NetworkAttachment containing
the IP address has been sent down to the node.
Includes a basic test for the correct use case.
Signed-off-by: Drew Erny <drew.erny@docker.com>
This feature allows user to specify list of subnets for global
default address pool. User can configure subnet list using
'swarm init' command. Daemon passes the information to swarmkit.
We validate the information in swarmkit, then store it in cluster
object. when IPAM init is called, we pass subnet list to IPAM driver.
Signed-off-by: selansen <elango.siva@docker.com>
* Expose license status in Info
This wires up a new field in the Info payload that exposes the license.
For moby this is hardcoded to always report a community edition.
Downstream enterprise dockerd will have additional licensing logic wired
into this function to report details about the current license status.
Signed-off-by: Daniel Hiltgen <daniel.hiltgen@docker.com>
* Code review comments
Signed-off-by: Daniel Hiltgen <daniel.hiltgen@docker.com>
* Add windows autogen support
Signed-off-by: Daniel Hiltgen <daniel.hiltgen@docker.com>
This driver uses protobuf to store log messages and has better defaults
for log file handling (e.g. compression and file rotation enabled by
default).
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Signed-off-by: John Howard <jhoward@microsoft.com>
Fixes#36764
@johnstep PTAL. @jterry75 FYI.
There are two commits in this PR. The first ensure that errors are actually returned to the caller - it was being thrown away.
The second commit changes the LCOW driver to map, on a per service VM basis, "long" container paths such as `/tmp/c8fa0ae1b348f505df2707060f6a49e63280d71b83b7936935c827e2e9bde16d` to much shorter paths, based on a per-service VM counter, so something more like /tmp/d3. This means that the root cause of the failure where the mount call to create the overlay was failing due to command line length becomes something much shorter such as below.
`mount -t overlay overlay -olowerdir=/tmp/d3:/tmp/d4:/tmp/d5:/tmp/d6:/tmp/d7:/tmp/d8:/tmp/d9:/tmp/d10:/tmp/d11:/tmp/d12:/tmp/d13:/tmp/d14:/tmp/d15:/tmp/d16:/tmp/d17:/tmp/d18:/tmp/d19:/tmp/d20:/tmp/d21:/tmp/d22:/tmp/d23:/tmp/d24:/tmp/d25:/tmp/d26:/tmp/d27:/tmp/d28:/tmp/d29:/tmp/d30:/tmp/d31:/tmp/d32:/tmp/d33:/tmp/d34:/tmp/d35:/tmp/d36:/tmp/d37:/tmp/d38:/tmp/d39:/tmp/d40:/tmp/d41:/tmp/d42:/tmp/d43:/tmp/d44:/tmp/d45:/tmp/d46:/tmp/d47:/tmp/d48:/tmp/d49:/tmp/d50:/tmp/d51:/tmp/d52:/tmp/d53:/tmp/d54:/tmp/d55:/tmp/d56:/tmp/d57:/tmp/d58:/tmp/d59:/tmp/d60:/tmp/d61:/tmp/d62,upperdir=/tmp/d2/upper,workdir=/tmp/d2/work /tmp/c8fa0ae1b348f505df2707060f6a49e63280d71b83b7936935c827e2e9bde16d-mount`
For those worrying about overflow (which I'm sure @thaJeztah will mention...): It's safe to use a counter here as SVMs are disposable in the default configuration. The exception is when running the daemon in unsafe LCOW "global" mode (ie `--storage-opt lcow.globalmode=1`) where the SVMs aren't disposed of, but a single one is reused. However, to overflow the command line length, it would require several hundred-thousand trillion (conservative, I should sit down and work it out accurately if I get -really- bored) of SCSI hot-add operations, and even to hit that would be hard as just running containers normally uses the VPMEM path for the containers UVM, not to the global SVM on SCSI. It gets incremented by one per build step (commit more accurately) as a general rule. Hence it would be necessary to have to be doing automated builds without restarting the daemon for literally years on end in unsafe mode. 😇
Note that in reality, the previous limit of ~47 layers before hitting the command line length limit is close to what is possible in the platform, at least as of RS5/Windows Server 2019 where, in the HCS v1 schema, a single SCSI controller is used, and that can only support 64 disks per controller per the Hyper-V VDEV. And remember we have one slot taken up for the SVMs scratch, and another for the containers scratch when committing a layer. So the best you can architecturally get on the platform is around the following (it's also different by 1 depending on whether in unsafe or default mode)
```
PS E:\docker\build\36764\short> docker build --no-cache .
Sending build context to Docker daemon 2.048kB
Step 1/4 : FROM alpine as first
---> 11cd0b38bc3c
Step 2/4 : RUN echo test > /test
---> Running in 8ddfe20e5bfb
Removing intermediate container 8ddfe20e5bfb
---> b0103a00b1c9
Step 3/4 : FROM alpine
---> 11cd0b38bc3c
Step 4/4 : COPY --from=first /test /test
---> 54bfae391eba
Successfully built 54bfae391eba
PS E:\docker\build\36764\short> cd ..
PS E:\docker\build\36764> docker build --no-cache .
Sending build context to Docker daemon 4.689MB
Step 1/61 : FROM alpine as first
---> 11cd0b38bc3c
Step 2/61 : RUN echo test > /test
---> Running in 02597ff870db
Removing intermediate container 02597ff870db
---> 3096de6fc454
Step 3/61 : RUN echo test > /test
---> Running in 9a8110f4ff19
Removing intermediate container 9a8110f4ff19
---> 7691808cf28e
Step 4/61 : RUN echo test > /test
---> Running in 9afb8f51510b
Removing intermediate container 9afb8f51510b
---> e42a0df2bb1c
Step 5/61 : RUN echo test > /test
---> Running in fe977ed6804e
Removing intermediate container fe977ed6804e
---> 55850c9b0479
Step 6/61 : RUN echo test > /test
---> Running in be65cbfad172
Removing intermediate container be65cbfad172
---> 0cf8acba70f0
Step 7/61 : RUN echo test > /test
---> Running in fd5b0907b6a9
Removing intermediate container fd5b0907b6a9
---> 257a4493d85d
Step 8/61 : RUN echo test > /test
---> Running in f7ca0ffd9076
Removing intermediate container f7ca0ffd9076
---> 3baa6f4fa2d5
Step 9/61 : RUN echo test > /test
---> Running in 5146814d4727
Removing intermediate container 5146814d4727
---> 485b9d5cf228
Step 10/61 : RUN echo test > /test
---> Running in a090eec1b743
Removing intermediate container a090eec1b743
---> a7eb10155b51
Step 11/61 : RUN echo test > /test
---> Running in 942660b288df
Removing intermediate container 942660b288df
---> 9d286a1e2133
Step 12/61 : RUN echo test > /test
---> Running in c3d369aa91df
Removing intermediate container c3d369aa91df
---> f78be4788992
Step 13/61 : RUN echo test > /test
---> Running in a03c3ac6888f
Removing intermediate container a03c3ac6888f
---> 6504363f61ab
Step 14/61 : RUN echo test > /test
---> Running in 0c3c2fca3f90
Removing intermediate container 0c3c2fca3f90
---> fe3448b8bb29
Step 15/61 : RUN echo test > /test
---> Running in 828d51c76d3b
Removing intermediate container 828d51c76d3b
---> 870684e3aea0
Step 16/61 : RUN echo test > /test
---> Running in 59a2f7c5f3ad
Removing intermediate container 59a2f7c5f3ad
---> cf84556ca5c0
Step 17/61 : RUN echo test > /test
---> Running in bfb4e088eeb3
Removing intermediate container bfb4e088eeb3
---> 9c8f9f652cef
Step 18/61 : RUN echo test > /test
---> Running in f1b88bb5a2d7
Removing intermediate container f1b88bb5a2d7
---> a6233ad21648
Step 19/61 : RUN echo test > /test
---> Running in 45f70577d709
Removing intermediate container 45f70577d709
---> 1b5cc52d370d
Step 20/61 : RUN echo test > /test
---> Running in 2ce231d5043d
Removing intermediate container 2ce231d5043d
---> 4a0e17cbebaa
Step 21/61 : RUN echo test > /test
---> Running in 52e4b0928f1f
Removing intermediate container 52e4b0928f1f
---> 99b50e989bcb
Step 22/61 : RUN echo test > /test
---> Running in f7ba3da7460d
Removing intermediate container f7ba3da7460d
---> bfa3cad88285
Step 23/61 : RUN echo test > /test
---> Running in 60180bf60f88
Removing intermediate container 60180bf60f88
---> fe7271988bcb
Step 24/61 : RUN echo test > /test
---> Running in 20324d396531
Removing intermediate container 20324d396531
---> e930bc039128
Step 25/61 : RUN echo test > /test
---> Running in b3ac70fd4404
Removing intermediate container b3ac70fd4404
---> 39d0a11ea6d8
Step 26/61 : RUN echo test > /test
---> Running in 0193267d3787
Removing intermediate container 0193267d3787
---> 8062d7aab0a5
Step 27/61 : RUN echo test > /test
---> Running in f41f45fb7985
Removing intermediate container f41f45fb7985
---> 1f5f18f2315b
Step 28/61 : RUN echo test > /test
---> Running in 90dd09c63d6e
Removing intermediate container 90dd09c63d6e
---> 02f0a1141f11
Step 29/61 : RUN echo test > /test
---> Running in c557e5386e0a
Removing intermediate container c557e5386e0a
---> dbcd6fb1f6f4
Step 30/61 : RUN echo test > /test
---> Running in 65369385d855
Removing intermediate container 65369385d855
---> e6e9058a0650
Step 31/61 : RUN echo test > /test
---> Running in d861fcc388fd
Removing intermediate container d861fcc388fd
---> 6e4c2c0f741f
Step 32/61 : RUN echo test > /test
---> Running in 1483962b7e1c
Removing intermediate container 1483962b7e1c
---> cf8f142aa055
Step 33/61 : RUN echo test > /test
---> Running in 5868934816c1
Removing intermediate container 5868934816c1
---> d5ff87cdc204
Step 34/61 : RUN echo test > /test
---> Running in e057f3201f3a
Removing intermediate container e057f3201f3a
---> b4031b7ab4ac
Step 35/61 : RUN echo test > /test
---> Running in 22b769b9079c
Removing intermediate container 22b769b9079c
---> 019d898510b6
Step 36/61 : RUN echo test > /test
---> Running in f1d364ef4ff8
Removing intermediate container f1d364ef4ff8
---> 9525cafdf04d
Step 37/61 : RUN echo test > /test
---> Running in 5bf505b8bdcc
Removing intermediate container 5bf505b8bdcc
---> cd5002b33bfd
Step 38/61 : RUN echo test > /test
---> Running in be24a921945c
Removing intermediate container be24a921945c
---> 8675db44d1b7
Step 39/61 : RUN echo test > /test
---> Running in 352dc6beef3d
Removing intermediate container 352dc6beef3d
---> 0ab0ece43c71
Step 40/61 : RUN echo test > /test
---> Running in eebde33e5d9b
Removing intermediate container eebde33e5d9b
---> 46ca4b0dfc03
Step 41/61 : RUN echo test > /test
---> Running in f920313a1e85
Removing intermediate container f920313a1e85
---> 7f3888414d58
Step 42/61 : RUN echo test > /test
---> Running in 10e2f4dc1ac7
Removing intermediate container 10e2f4dc1ac7
---> 14db9e15f2dc
Step 43/61 : RUN echo test > /test
---> Running in c849d6e89aa5
Removing intermediate container c849d6e89aa5
---> fdb770494dd6
Step 44/61 : RUN echo test > /test
---> Running in 419d1a8353db
Removing intermediate container 419d1a8353db
---> d12e9cf078be
Step 45/61 : RUN echo test > /test
---> Running in 0f1805263e4c
Removing intermediate container 0f1805263e4c
---> cd005e7b08a4
Step 46/61 : RUN echo test > /test
---> Running in 5bde05b46441
Removing intermediate container 5bde05b46441
---> 05aa426a3d4a
Step 47/61 : RUN echo test > /test
---> Running in 01ebc84bd1bc
Removing intermediate container 01ebc84bd1bc
---> 35d371fa4342
Step 48/61 : RUN echo test > /test
---> Running in 49f6c2f51dd4
Removing intermediate container 49f6c2f51dd4
---> 1090b5dfa130
Step 49/61 : RUN echo test > /test
---> Running in f8a9089cd725
Removing intermediate container f8a9089cd725
---> b2d0eec0716d
Step 50/61 : RUN echo test > /test
---> Running in a1697a0b2db0
Removing intermediate container a1697a0b2db0
---> 10d96ac8f497
Step 51/61 : RUN echo test > /test
---> Running in 33a2332c06eb
Removing intermediate container 33a2332c06eb
---> ba5bf5609c1c
Step 52/61 : RUN echo test > /test
---> Running in e8920392be0d
Removing intermediate container e8920392be0d
---> 5b3a95685c7e
Step 53/61 : RUN echo test > /test
---> Running in 4b9298587c65
Removing intermediate container 4b9298587c65
---> d4961a349141
Step 54/61 : RUN echo test > /test
---> Running in 8a0c960c2ba1
Removing intermediate container 8a0c960c2ba1
---> b413197fcfa2
Step 55/61 : RUN echo test > /test
---> Running in 536ee3b9596b
Removing intermediate container 536ee3b9596b
---> fc16b69b224a
Step 56/61 : RUN echo test > /test
---> Running in 8b817b8d7b59
Removing intermediate container 8b817b8d7b59
---> 2f0896400ff9
Step 57/61 : RUN echo test > /test
---> Running in ab0ed79ec3d4
Removing intermediate container ab0ed79ec3d4
---> b4fb420e736c
Step 58/61 : RUN echo test > /test
---> Running in 8548d7eead1f
Removing intermediate container 8548d7eead1f
---> 745103fd5a38
Step 59/61 : RUN echo test > /test
---> Running in 1980559ad5d6
Removing intermediate container 1980559ad5d6
---> 08c1c74a5618
Step 60/61 : FROM alpine
---> 11cd0b38bc3c
Step 61/61 : COPY --from=first /test /test
---> 67f053c66c27
Successfully built 67f053c66c27
PS E:\docker\build\36764>
```
Note also that subsequent error messages once you go beyond current platform limitations kind of suck (such as insufficient resources with a bunch of spew which is incomprehensible to most) and we could do better to detect this earlier in the daemon. That'll be for a (reasonably low-priority) follow-up though as and when I have time. Theoretically we *may*, if the platform doesn't require additional changes for RS5, be able to have bigger platform limits using the v2 schema with up to 127 VPMem devices, and the possibility to have multiple SCSI controllers per SVM/UVM. However, currently LCOW is using HCS v1 schema calls, and there's no plans to rewrite the graphdriver/libcontainerd components outside of the moving LCOW fully over to the containerd runtime/snapshotter using HCS v2 schema, which is still some time off fruition.
PS OK, while waiting for a full run to complete, I did get bored. Turns out it won't overflow line length as max(uint64) is 18446744073709551616 which would still be short enough at 127 layers, double the current platform limit. And I could always change it to hex or base36 to make it even shorter, or remove the 'd' from /tmp/dN. IOW, pretty sure no-one is going to hit the limit even if we could get the platform to 256 which is the current Hyper-V SCSI limit per VM (4x64), although PMEM at 127 would be the next immediate limit.
Modify hack/make.ps1 to use the version value used in
"FROM golang" statement.
While at it:
1. Make search expression a bit more strict (use ^ to match at BOL only).
2. Simplify by removing Get-Contents as Select-String can read files.
After this, ENV GO_VERSION can be removed from Dockerfile.
Unfortunately it can't be done in one commit as Windows CI
fails (presumably because Dockerfile is being modified in
place).
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
1. After running d.Cmd(), in case an error is returned, it makes sense
to print command output, as its stderr may contain a clue about what
went wrong. This is by no means complete, just as far as I could go.
2. In case the comment in c.Assert is a constant string, it's better
to provide it as a comment which will be printed.
3. An arbitrary string should not be passed on to a function expecting
%-style formatting. Use %s to fix this.
4. Print the output string before transformation, not after.
5. Unify the output format (drop "out:" prefix").
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
It is wrong to pass an arbitrary string to a function expecting
%-style formatting. One solution would be to replace any % with %%,
but it's easier to just do what this patch does.
Generated with:
for f in $(git grep -l 'check.Commentf(out)'); do \
sed -i -e 's/check\.Commentf(out)/check.Commentf("%s", out)/g' $f; \
done
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
This implements chown support on Windows. Built-in accounts as well
as accounts included in the SAM database of the container are supported.
NOTE: IDPair is now named Identity and IDMappings is now named
IdentityMapping.
The following are valid examples:
ADD --chown=Guest . <some directory>
COPY --chown=Administrator . <some directory>
COPY --chown=Guests . <some directory>
COPY --chown=ContainerUser . <some directory>
On Windows an owner is only granted the permission to read the security
descriptor and read/write the discretionary access control list. This
fix also grants read/write and execute permissions to the owner.
Signed-off-by: Salahuddin Khan <salah@docker.com>
This makes it so consumers of `LogFile` should pass in how to get an
io.Reader to the requested number of lines to tail.
This is also much more efficient when tailing a large number of lines.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This fixes a bug when the build results in an error and instead of sending
the logs of the container to the client, it signals a cancelation.
The context should not be wired to the status request, as only the
build requests need to be canceled.
Signed-off-by: Tibor Vass <tibor@docker.com>
Adds a supervisor package for starting and monitoring containerd.
Separates grpc connection allowing access from daemon.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Relevant changes;
- swarmkit #2593 agent: return error when failing to apply network key
- swarmkit #2645 Replace deprecated grpc functions
- swarmkit #2720 Test if error is nil before to log it
- swarmkit #2712 [orchestrator] Fix task sorting
- swarmkit #2677 [manager/orchestrator/reaper] Fix the condition used for skipping over running tasks
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Else it is harder to understand where we've failed in StartWithLogFile
We have an errror:
[d42ce729d0b06] waiting for daemon to start
[d42ce729d0b06] waiting for daemon to start
[d42ce729d0b06] waiting for daemon to start
[d42ce729d0b06] waiting for daemon to start
[d42ce729d0b06] waiting for daemon to start
[d42ce729d0b06] waiting for daemon to start
[d42ce729d0b06] waiting for daemon to start
[d42ce729d0b06] waiting for daemon to start
[d42ce729d0b06] waiting for daemon to start
[d42ce729d0b06] waiting for daemon to start
[d42ce729d0b06] waiting for daemon to start
[d42ce729d0b06] waiting for daemon to start
docker_cli_userns_test.go:27:
s.d.StartWithBusybox(c, "--userns-remap", "default")
/go/src/github.com/docker/docker/internal/test/daemon/daemon.go:203:
t.Fatalf("Error starting daemon with arguments: %v", args)
... Error: Error starting daemon with arguments: [--userns-remap default]
[d42ce729d0b06] exiting daemon
Likely it is "[..] Daemon exited during startup" case, but these error
message is lost
Also we lose the error returned by waiting daemon so also add it
Signed-off-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>
WithDialContext() allows specifying custom dialer for hijacking and supposed to
replace WithDialer().
WithDialer() is also updated to use WithDialContext().
client.Dialer() returns the dialer configured with WithDialContext().
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
Handle the case of systemd-resolved, and if in place
use a different resolv.conf source.
Set appropriately the option on libnetwork.
Move unix specific code to container_operation_unix
Signed-off-by: Flavio Crisciani <flavio.crisciani@docker.com>
Previously, dockerd would always ask containerd to pass --leave-running
to runc/runsc, ignoring the exit boolean value. Hence, even `docker
checkpoint create --leave-running=false ...` would not stop the
container.
Signed-off-by: Brielle Broder <bbroder@google.com>
Disable cri plugin by default in containerd and
allows an option to enable the plugin. This only
has an effect on containerd when supervised by
dockerd. When containerd is managed outside of
dockerd, the configuration is not effected.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Go 1.11 includes a fix to os/user to be working in a static binary
(fixing https://github.com/golang/go/issues/23265). The fix requires
`osusergo` build tag to be set for static binaries, which is what
this commit adds (also for containerd).
[v2: sort build tags alphabetically]
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Since go-1.11beta1 archive/tar, tar headers with Typeflag == TypeRegA
(numeric 0) (which is the default unless explicitly initialized) are
modified to have Typeflag set to either tar.TypeReg (character value
'0', not numeric 0) or tar.TypeDir (character value '5') [1].
This results in different Typeflag value in the resulting header,
leading to a different Checksum, and causing the following test
case errors:
> 12:09:14 --- FAIL: TestTarSums (0.05s)
> 12:09:14 tarsum_test.go:393: expecting
> [tarsum+sha256:8bf12d7e67c51ee2e8306cba569398b1b9f419969521a12ffb9d8875e8836738],
> but got
> [tarsum+sha256:75258b2c5dcd9adfe24ce71eeca5fc5019c7e669912f15703ede92b1a60cb11f]
> ... (etc.)
All the other code explicitly sets the Typeflag field, but this test
case is not, causing the incompatibility with Go 1.11. Therefore,
the fix is to set TypeReg explicitly, and change the expected checksums
in test cases).
Alternatively, we can vendor archive/tar again (for the 100th time),
but given that the issue is limited to the particular test case it
does not make sense.
This fixes the test for all Go versions.
[1] https://go-review.googlesource.com/c/go/+/85656
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
changes included:
- swarmkit #2706 address unassigned task leak when service is removed
- swarmkit #2676 Fix racy batching on the dispatcher
- swarmkit #2693 Fix linting issues revealed by Go 1.11
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This test occassionally fails on s390x and Power;
03:16:04 --- FAIL: TestExternalGraphDriver/pull (1.08s)
03:16:04 external_test.go:402: assertion failed: error is not nil: Error: No such image: busybox:latest
Most likely these failures are caused due to Docker Hub updating
the busybox:latest image, but not all architectures yet being
available.
Instead of using `:latest`, pull an image by digest, so that
the test doesn't depend on Docker Hub having all architectures
available for `:latest`.
I selected the same digest as is currently used as "frozen image"
in the Dockerfile.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This introduces a PRODUCT environment variable that is used to set a constant
at dockerversion.ProductName.
That is then used to set BuildKit's ExportedProduct variable in order to show
useful error messages to users when a certain version of the product doesn't
support a BuildKit feature.
Signed-off-by: Tibor Vass <tibor@docker.com>
* Regex name filters were display undesired behavior due to
names containing the trailing slash when being compared
* Adjusted filterByNameIDMatches and includeContainerInList to
strip the slash prefix before doing name comparisons
* Added test case and helper functions for the test to list_test
* Force failed tests during development to ensure there were
no false positives
Signed-off-by: Chris White <me@cwprogram.com>
Instead of waiting for the DNS to fail, try to access
a specific external IP and verify that 100% of the pakcets
are being lost.
Signed-off-by: Flavio Crisciani <flavio.crisciani@docker.com>
Should fix
```
api/types/volume/volume_create.go
Line 10: warning: comment on exported type VolumeCreateBody should be of the form "VolumeCreateBody ..." (with optional leading article) (golint)
api/types/volume/volume_list.go
Line 12: warning: comment on exported type VolumeListOKBody should be of the form "VolumeListOKBody ..." (with optional leading article) (golint)
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
When go-1.11beta1 is used for building, the following error is
reported:
> 14:56:20 daemon\graphdriver\lcow\lcow.go:236: Debugf format %s reads
> arg #2, but call has 1 arg
While fixing this, let's also fix a few other things in this
very function (startServiceVMIfNotRunning):
1. Do not use fmt.Printf when not required.
2. Use `title` whenever possible.
3. Don't add `id` to messages as `title` already has it.
4. Remove duplicated colons.
5. Try to unify style of messages.
6. s/startservicevmifnotrunning/startServiceVMIfNotRunning/
...
In general, logging/debugging here is a mess and requires much more
love than I can give it at the moment. Areas for improvement:
1. Add a global var logger = logrus.WithField("storage-driver", "lcow")
and use it everywhere else in the code.
2. Use logger.WithField("id", id) whenever possible (same for "context"
and other similar fields).
3. Revise all the errors returned to be uniform.
4. Make use of errors.Wrap[f] whenever possible.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
There are two build errors when using go-1.11beta1:
> daemon/logger/loggerutils/logfile.go:367: Warningf format %q arg f.Name is a func value, not called
> daemon/logger/loggerutils/logfile.go:564: Debug call has possible formatting directive %v
In the first place, the file name is actually not required as error
message already includes it.
While at it, fix a couple of other places for more correct messages, and
make sure to not add a file name if an error already has it.
Fixes: f69f09f44c
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Fix the following go-1.11beta1 build error:
> daemon/graphdriver/aufs/aufs.go:376: Wrapf format %s reads arg #1, but call has 0 args
While at it, change '%s' to %q.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Go 1.11beta1 (rightfully) complains:
> 15:38:37 daemon/cluster/controllers/plugin/controller.go:183:
> Entry.Debugf format %#T has unrecognized flag #
This debug print was added by commit 72c3bcf2a5.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
In particular, these two:
> daemon/daemon_unix.go:1129: Wrapf format %v reads arg #1, but call has 0 args
> daemon/kill.go:111: Warn call has possible formatting directive %s
and a few more.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The absence of the file /proc/sys/net/ipv6/conf/all/disable_ipv6
doesn't appear to affect functionality, at least at this time.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
dockerd allows the `--log-level` to be specified, but this log-level
was not forwarded to the containerd process.
This patch sets containerd's log-level to the same as dockerd if a
custom level is provided.
Now that `--log-level` is also passed to containerd, the default "info"
is removed, so that containerd's default (or the level configured in containerd.toml)
is still used if no log-level is set.
Before this change:
containerd would always be started without a log-level set (only the level that's configured in `containerd.toml`);
```
root 1014 2.5 2.1 496484 43468 pts/0 Sl+ 12:23 0:00 dockerd
root 1023 1.2 1.1 681768 23832 ? Ssl 12:23 0:00 \_ docker-containerd --config /var/run/docker/containerd/containerd.toml
```
After this change:
when running `dockerd` without options (same as current);
```
root 1014 2.5 2.1 496484 43468 pts/0 Sl+ 12:23 0:00 dockerd
root 1023 1.2 1.1 681768 23832 ? Ssl 12:23 0:00 \_ docker-containerd --config /var/run/docker/containerd/containerd.toml
```
when running `dockerd --debug`:
```
root 600 0.8 2.1 512876 43180 pts/0 Sl+ 12:20 0:00 dockerd --debug
root 608 0.6 1.1 624428 23672 ? Ssl 12:20 0:00 \_ docker-containerd --config /var/run/docker/containerd/containerd.toml --log-level debug
```
when running `dockerd --log-level=panic`
```
root 747 0.6 2.1 496548 43996 pts/0 Sl+ 12:21 0:00 dockerd --log-level=panic
root 755 0.7 1.1 550696 24100 ? Ssl 12:21 0:00 \_ docker-containerd --config /var/run/docker/containerd/containerd.toml --log-level panic
```
combining `--debug` and `--log-level` (`--debug` takes precedence):
```
root 880 2.7 2.1 634692 43336 pts/0 Sl+ 12:23 0:00 dockerd --debug --log-level=panic
root 888 1.0 1.1 616232 23652 ? Ssl 12:23 0:00 \_ docker-containerd --config /var/run/docker/containerd/containerd.toml --log-level debug
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
It said `TESTFLAGS='-test.run ^TestValidateIPAddress$' make test-unit`
runs `TestBuild` test, but actually runs `TestValidateIPAddress` test.
Signed-off-by: Donghwa Kim <shanytt@gmail.com>
Before this commit Healthcheck run if HEALTHCHECK
instruction appears before RUN instruction.
By passing `withoutHealthcheck` to `copyRunConfig`,
always RUN instruction run without Healthcheck.
Fix: https://github.com/moby/moby/issues/37362
Signed-off-by: Yuichiro Kaneko <spiketeika@gmail.com>
Commit c0bc14e8 wrapped the return value of nw.Delete() with some extra
information. However, this breaks the code in
containerAdaptor.removeNetworks() which ignores certain specific
libnetwork error return codes. Said codes actually don't represent
errors, but just regular conditions to be expected in normal operation.
The removeNetworks() call checked for these errors by type assertions
which the errors.Wrap(err...) breaks.
This has a cascading effect, because controller.Remove() invokes
containerAdaptor.removeNetworks() and if the latter returns an error,
then Remove() fails to remove the container itself. This is not
necessarily catastrophic since the container reaper apparently will
purge the container later, but it is clearly not the behavior we want.
Signed-off-by: Chris Telfer <ctelfer@docker.com>
Bump libnetwork to 3ac297bc7fd0afec9051bbb47024c9bc1d75bf5b in order to
get fix 0c3d9f00 which addresses a flaw that the scalable load balancing
code revealed. Attempting to print sandbox IDs where the sandbox name
was too short results in a goroutine panic. This can occur with
sandboxes with names of 1 or 2 characters in the previous code. But due
to naming updates in the scalable load balancing code, it could now
occur for networks whose name was 3 characters and at least one of the
integration tests employed such networks (named 'foo', 'bar' and 'baz').
This update also brings in several changes as well:
* 6c7c6017 - Fix error handling about bridgeSetup
* 5ed38221 - Optimize networkDB queue
* cfa9afdb - ndots: produce error on negative numbers
* 5586e226 - improve error message for invalid ndots number
* 449672e5 - Allows to set generic knobs on the Sandbox
* 6b4c4af7 - do not ignore user-provided "ndots:0" option
* 843a0e42 - Adjust corner case for reconnect logic
Signed-off-by: Chris Telfer <ctelfer@docker.com>
The deafult OCI linux spec in oci/defaults{_linux}.go in Docker/Moby
from 1.11 to current upstream master does not block /proc/acpi pathnames
allowing attackers to modify host's hardware like enabling/disabling
bluetooth or turning up/down keyboard brightness. SELinux prevents all
of this if enabled.
Signed-off-by: Antonio Murdaca <runcom@redhat.com>
This partially reverts https://github.com/moby/moby/pull/37350
Although specs.Platform is desirable in the API, there is more work
to be done on helper functions, namely containerd's platforms.Parse
that assumes the default platform of the Go runtime.
That prevents a client to use the recommended Parse function to
retrieve a specs.Platform object.
With this change, no parsing is expected from the client.
Signed-off-by: Tibor Vass <tibor@docker.com>
This patch is required for the updated version of libnetwork and entails
two minor changes.
First, it uses the new libnetwork.NetworkDeleteOptionRemoveLB option to
the network.Delete() method to automatically remove the load balancing
endpoint for ingress networks. This allows removal of the
deleteLoadBalancerSandbox() function whose functionality is now within
libnetwork.
The second change is to allocate a load balancer endpoint IP address for
all overlay networks rather than just "ingress" and windows overlay
networks. Swarmkit is already performing this allocation, but moby was
not making use of these IP addresses for Linux overlay networks (except
ingress). The current version of libnetwork makes use of these IP
addresses by creating a load balancing sandbox and endpoint similar to
ingress's for all overlay network and putting all load balancing state
for a given node in that sandbox only. This reduces the amount of linux
kernel state required per node.
In the prior scheme, libnetwork would program each container's network
namespace with every piece of load balancing state for every other
container that shared *any* network with the first container. This
meant that the amount of kernel state on a given node scaled with the
square of the number of services in the cluster and with the square of
the number of containers per service. With the new scheme, kernel state
at each node scales linearly with the number of services and the number
of containers per service. This also reduces the number of system calls
required to add or remove tasks and containers. Previously the number
of system calls required grew linearly with the number of other
tasks that shared a network with the container. Now the number of
system calls grows linearly only with the number of networks that the
task/container is attached to. This results in a significant
performance improvement when adding and removing services to a cluster
that already heavily loaded.
The primary disadvantage to this scheme is that it requires the
allocation of an additional IP address per node per subnet for every
node in the cluster that has a task on the given subnet. However, as
mentioned, swarmkit is already allocating these IP addresses for every
node and they are going unused. Future swarmkit modifications should be
examined to only allocate said IP addresses when nodes actually require
them.
Signed-off-by: Chris Telfer <ctelfer@docker.com>
Bump libnetwork to b0186632522c68f4e1222c4f6d7dbe518882024f. This
includes the following changes:
* Dockerize protocol buffer generation and update (78d9390a..e12dd44c)
* Use new plugin interfaces provided by plugin pkg (be94e134)
* Improve linux load-balancing scalability (5111c24e..366b9110)
Signed-off-by: Chris Telfer <ctelfer@docker.com>
By 0296797f0f, `progressReader`
and `remoteURL` were removed from arguments. So developers who
use `Detect` not need to care about when `ProgressReaderFunc`
is used.
Signed-off-by: Yuichiro Kaneko <spiketeika@gmail.com>
When using the mounts API, bind mounts are not supposed to be
automatically created.
Before this patch there is a race condition between valiating that a
bind path exists and then actually setting up the bind mount where the
bind path may exist during validation but was removed during mountpooint
setup.
This adds a field to the mountpoint struct to ensure that binds created
over the mounts API are not accidentally created.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
The TestDockerNetworkIPAMMultipleNetworks test allocates several
networks simultaneously with overlapping IP addresses. Libnetwork now
forbids this. Adjust the test case to use distinct IP ranges for the
networks it creates.
Signed-off-by: Chris Telfer <ctelfer@docker.com>
Bump libnetwork to 430c00a6a6b3dfdd774f21e1abd4ad6b0216c629. This
includes the following moby-affecting changes:
* Update vendoring for go-sockaddr (8df9f31a)
* Fix inconsistent subnet allocation by preventing allocation of
overlapping subnets (8579c5d2)
* Handle IPv6 literals correctly in port bindings (474fcaf4)
* Update vendoring for miekg/dns (8f307ac8)
* Avoid subnet reallocation until required (9756ff7ed)
* Bump libnetwork build to use go version 1.10.2 (603d2c1a)
* Unwrap error type returned by PluginGetter (aacec8e1)
* Update vendored components to match moby (d768021dd)
* Add retry field to cluster-peers probe (dbbd06a7)
* Fix net driver response loss on createEndpoint (1ab6e506)
(fixes https://github.com/docker/for-linux/issues/348)
Signed-off-by: Chris Telfer <ctelfer@docker.com>
When a manifest list contains both Linux and Windows images, always
prefer Windows when the platform OS is unspecified. Also, filter out any
Windows images with a higher build than the host, since they cannot run.
Signed-off-by: John Stephens <johnstep@docker.com>
OCI types are backwards compatible with Docker manifest
types, however the media types must be registered.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
This stuff doesn't belong here and is causing imports of libnetwork into
the router, which is not what we want.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Signed-off-by: John Howard <jhoward@microsoft.com>
Addresses https://github.com/moby/moby/pull/35089#issuecomment-367802698.
This change enables the daemon to automatically select an image under LCOW
that can be used if the API doesn't specify an explicit platform.
For example:
FROM supertest2014/nyan
ADD Dockerfile /
And docker build . will download the linux image (not a multi-manifest image)
And similarly docker pull ubuntu will match linux/amd64
This test is testing if any "no space left on device" errors
that occur during `docker pull` will not be masked by other
errors. To test for this, a new loopback-device was created,
and used as `--data-dir` ("/var/lib/docker").
However, `/var/lib/docker` is used for storing various
other things, including a `cache.db` database, used by
BuildKit, which is created during startup of the daemon.
Creation of that file failed (due to `--data-dir` path
being on a mount with limited size), which caused daemon
start to fail before the test was able to run.
This patch changes the size-limited mount to be used for
the storage-driver directory only, so that the test is
not affected by other parts of the code attempting to
write files in it.
To have a predictable path; the daemon used in this test
is configured to use the `vfs` storage-driver.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
github.com/gotestyourself/gotestyourself moved to gotest.tools with
version 2.0.0. Moving to that one, bumping it to v2.1.0.
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
Since go1.8, the stdlib TLS net.Conn implementation implements the
`CloseWrite()` interface.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Scenario:
Daemon is ungracefully shutdown and leaves plugins running (no
live-restore).
Daemon comes back up.
The next time a container tries to use that plugin it will cause a
daemon panic because the plugin client is not set.
This fixes that by ensuring that the plugin does get shutdown.
Note, I do not think there would be any harm in just re-attaching to the
running plugin instead of shutting it down, however historically we shut
down plugins and containers when live-restore is not enabled.
[kir@: consolidate code to deleteTaskAndContainer, a few minor nits]
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
… discovered recently that our very own Sebastiaan was not even listed
in the maintainers, so fixing this mistake !
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
This adds MaskedPaths and ReadOnlyPaths options to HostConfig for containers so
that a user can override the default values.
When the value sent through the API is nil the default is used.
Otherwise the default is overridden.
Adds integration tests for MaskedPaths and ReadonlyPaths.
Signed-off-by: Jess Frazelle <acidburn@microsoft.com>
This PR is trying to refactor the `probeAndCreate` and cleanup
related codes based on the refactoring.
Signed-off-by: Dennis Chen <dennis.chen@arm.com>
`make DOCKERD_ARGS=--init binary run` should start the daemon with
`--init` as flags (with any other "automagically" added ones).
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
With a full attach, each attach was leaking 4 goroutines.
This updates attach to use errgroup instead of the hodge-podge of
waitgroups and channels.
In addition, the detach event was never being sent.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This makes it a bit simpler to remove this interface for v2 plugins
and not break external projects (libnetwork and swarmkit).
Note that before we remove the `Client()` interface from `CompatPlugin`
libnetwork and swarmkit must be updated to explicitly check for the v1
client interface as is done int his PR.
This is just a minor tweak that I realized is needed after trying to
implement the needed changes on libnetwork.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This fix adds `t.Name()` to tests in integration/network
so that services created in those tests have unique names.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
Begin to copy the data until the command to exit and any coping to
stdin or copy from stdout/stderr has completed.
Also adding defense code to trim the possible '\x00' null value.
Signed-off-by: Dennis Chen <dennis.chen@arm.com>
`${}`, `${:}` and so on are invalid because there's
no parameter within the brackets; fix detection for
this situation and add/update tests.
There were some existing test-cases that were testing
for the wrong behavior, which are now updated.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The builder did not detect syntax errors in substitusions in the
Dockerfile, causing those values to be processed incorrectly instead of
producing an error.
Example 1: missing `}`
docker build --no-cache -<<'EOF'
FROM busybox
ARG var=${aaa:-bbb
RUN echo $var
EOF
Before:
Step 3/3 : RUN echo $var
---> Running in f06571e77146
bbb
After:
Step 2/3 : ARG var=${aaa:-bbb
failed to process "${aaa:-bbb": syntax error: missing '}'
Example 2: missing closing `}`, no default value
docker build --no-cache -<<'EOF'
FROM busybox
ARG var=${aaa
RUN echo $var
EOF
Before:
Step 2/3 : ARG var=${aaa
failed to process "${aaa": missing ':' in substitution
After:
Step 2/3 : ARG var=${aaa
failed to process "${aaa": syntax error: missing '}'
Example 3: double opening bracket (`{`)
docker build --no-cache -<<'EOF'
FROM busybox
ARG var=${{aaa:-bbb}
RUN echo $var
EOF
Before:
Step 2/3 : ARG var=${{aaa:-bbb}
failed to process "${{aaa:-bbb}": missing ':' in substitution
After:
Step 2/3 : ARG var=${{aaa:-bbb}
failed to process "${{aaa:-bbb}": syntax error: bad substitution
Example 4: double opening bracket (`{`), no default value
docker build --no-cache -<<'EOF'
FROM busybox
ARG var=${{aaa}
RUN echo $var
EOF
Before:
Step 2/3 : ARG var=${{aaa}
failed to process "${{aaa}": missing ':' in substitution
After:
Step 2/3 : ARG var=${{aaa}
failed to process "${{aaa}": syntax error: bad substitution
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
If "ps" fails, in many cases it prints a meaningful error message
which a user can benefit from. Let's use it.
While at it, let's use errdefs.System to classify the error,
as well as errors.Wrap.
Before:
> $ docker top $CT <any bad ps options>
> Error response from daemon: Error running ps: exit status 1
After:
> $ docker top $CT auxm
> Error response from daemon: ps: error: thread display conflicts with forest display
or
> $ docker top $CT saur
> Error response from daemon: ps: error: conflicting format options
or, if there's no meaningful error on stderr, same as before:
> $ docker top $CT 1234
> Error response from daemon: ps: exit status 1
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Current ContainerTop (a.k.a. docker top) implementation uses "ps"
to get the info about *all* running processes, then parses it, then
filters the results to only contain PIDs used by the container.
Collecting data only to throw most of it away is inefficient,
especially on a system running many containers (or processes).
For example, "docker top" on a container with a single process
can take up to 0.5 seconds to execute (on a mostly idle system)
which is noticeably slow.
Since the containers PIDs are known beforehand, let's use ps's
"-q" option to provide it with a list of PIDs we want info about.
The problem with this approach is, some ps options can't be used
with "-q" (the only one I'm aware of is "f" ("forest view") but
there might be more). As the list of such options is not known,
in case ps fails, it is executed again without "q" (retaining
the old behavior).
Next, the data produced by "ps" is filtered in the same way as before.
The difference here is, in case "-q" worked, the list is much shorter.
I ran some benchmarks on my laptop, with about 8000 "sleep" processes
running to amplify the savings.
The improvement in "docker top" execution times is 5x to 10x (roughly
0.05s vs 0.5s).
The improvement in ContainerTop() execution time is up to 100x
(roughly 3ms vs 300ms).
I haven't measured the memory or the CPU time savings, guess those
are not that critical.
NOTE that busybox ps does not implement -q so the fallback is always
used, but AFAIK it is not usable anyway and Docker expects a normal
ps to be on the system (say the list of fields produced by
"busybox ps -ef" differs from normal "ps -ef" etc.).
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
This fix was not yet included in Docker 17.05, so
API version v1.37 was not the right selector (Docker
18.03, 18.04 and 18.05 all support API v1.37).
We should change these checks for engine versions,
or use a different method to skip tests when running
against older engines.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Sometimes docker-master CI fails on rhel4+selinux configuration,
like this:
--- FAIL: TestMount (0.12s)
--- FAIL: TestMount/none-remount,size=128k (0.01s)
mounter_linux_test.go:209: unexpected mount option "seclabel" expected "rw,size=128k"
--- FAIL: TestMount/none-remount,ro,size=128k (0.01s)
mounter_linux_test.go:209: unexpected mount option "seclabel" expected "ro,size=128k"
Earlier, commit 8bebd42df2 (PR #34965) fixed this failure,
but not entirely (i.e. the test is now flaky). It looks like
either selinux detection code is not always working (it won't
work in d-in-d), or the kernel might or might not add 'seclabel'
option).
As the subject of this test case is definitely not selinux,
it can just ignore the option added by it.
While at it, fix error messages:
- add missing commas;
- fix a typo;
- allow for clear distinction between mount
and vfs (per-superblock) options.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
The following failure is seen in CI from time to time:
> FAIL: docker_api_containers_test.go:435: DockerSuite.TestContainerAPITop
>
> docker_api_containers_test.go:453:
> c.Assert(top.Processes[0][10], checker.Equals, "/bin/sh -c top")
> ... obtained string = "top"
> ... expected string = "/bin/sh -c top"
The test case expects two processes in the output:
1. /bin/sh -c top
2. top
in the given order.
Now, "ps aux" output is sorted by PID*, and so since the "top" is a child
of "/bin/sh -c top" it has a higher PID and will come second as expected
by the test... unless the PIDs on the system are exhausted and PID rollover
happens, in which case PID of "top" will be lower than that of "/bin/sh".
Fix: sort output by process name.
* - in fact it is not sorted, but is being printed in the same order as
the kernel list PID entries in /proc directory, which appears to be
sorted by PID (see ls -1 -U /proc).
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
It does not make sense to keep looking for PID once
we found it, so let's give it a break.
The side effect of this patch is, if there's more than one column
titled "PID", the last (rightmost) column was used before, and now
the first (leftmost) column is used. Should make no practical
difference whatsoever.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Adds functionality to parse and return network attachment spec
information. Network attachment tasks are phony tasks created in
swarmkit to deal with unmanaged containers attached to swarmkit. Before
this change, attempting `docker inspect` on the task id of a network
attachment task would result in an empty task object. After this change,
a full task object is returned
Fixes#26548 the correct way.
Signed-off-by: Drew Erny <drew.erny@docker.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This prevents the following test case failures "go test" is run
as non-root in the daemon/ directory:
> --- FAIL: TestContainerInitDNS (0.02s)
> daemon_test.go:209: chown /tmp/docker-container-test-054812199/volumes: operation not permitted
>
> --- FAIL: TestDaemonReloadNetworkDiagnosticPort (0.00s)
> reload_test.go:525: mkdir /var/lib/docker/network/files/: permission denied
> --- FAIL: TestRootMountCleanup (0.00s)
> daemon_linux_test.go:240: assertion failed: error is not nil: operation not permitted
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Arrays expect a type to be set for items in the array.
This patch adds the "string" type, adds a short description,
and some example values.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
`arm64` needs get more time duration for the test to finish.
`pty.Start()` opens a file, so the caller should close it explicitly,
else the file I/O can result in unexpected data synchronization issue.
All those changes will not affect the test itself.
Signed-off-by: Dennis Chen <dennis.chen@arm.com>
Changes included:
- libnetwork#2147 Adding logs for ipam state
- libnetwork#2143 Fix race conditions in the overlay network driver
- possibly addresses moby#36743 services do not start: ingress-sbox is already present
- possibly addresses moby#30427 Flaky Test: TestSwarmPublishDuplicatePorts on s390
- possibly addresses moby#36501 Flaky tests: Service "port" tests
- libnetwork#2142 Add wait time into xtables lock warning
- libnetwork#2135 filter xtables lock warnings when firewalld is active
- libnetwork#2140 Switch from x/net/context to context
- libnetwork#2134 Adding a recovery mechanism for a split gossip cluster
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
`GetTimestamp()` "assumed" values it could not parse
to be a valid unix timestamp, and would use invalid
values ("hello world") as-is (even testing that
it did so).
This patch validates unix timestamp to be a valid
numeric value, and makes other values invalid.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
In case aufs driver is not supported because supportsAufs() said so,
it is not possible to get a real reason from the logs.
To fix, log the error returned.
Note we're not using WithError here as the error message itself is the
sole message we want to print (i.e. there's nothing to add to it).
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
The `deleteAllImages()` cleanup step in the integration suite was printing a
message for each image that was removed.
These messages were not very informative (given that `removeImage()` prints an
error if image removal failed), and made the test output harder to read
(more difficult to see error-conditions when scanning the output).
This patch removes the messages to make the output slightly less noisy.
Before this patch applied:
--- PASS: TestBuildMultiStageOnBuild (7.15s)
main_test.go:32: Removing image sha256:9db3ddbaaadd52804d8a417081f68db41fd4b8f80c85c1b4c4aee2d9b584c074
main_test.go:32: Removing image sha256:7eeb04d90b5e62a99ac6a5b2c10b9ba54b89b176fe2783e41461581c482852b3
main_test.go:32: Removing image sha256:adf42475eefff99b4a611c1a5d8353c4d0a011a7f7b9dc59a75d951cd54fa77f
main_test.go:32: Removing image sha256:c547a770806e0445f5dfc255683ced771a23be6157ba8d0617bb9ab55dcee6d1
main_test.go:32: Removing image sha256:b79659c3e6d34faf2a075f1df1ea2c805833982f112666b25a466177b5d1352d
main_test.go:32: Removing image sha256:29430078cdc927c19c87416cd1fb1ec386f167c5e201ee5dfb1644dcf268a3c2
=== RUN TestBuildUncleanTarFilenames
--- PASS: TestBuildUncleanTarFilenames (2.47s)
main_test.go:32: Removing image sha256:0a6418fb221dc2f25085a1a7e507e01c4a3938cc5c65f1cb85a8c0fb09d6814f
main_test.go:32: Removing image sha256:3d6e4bd0cce01ce5823b40dcb717cd16b3b4b769ff73dd86fa448aac49aa6d7c
main_test.go:32: Removing image sha256:98e3f335e874612668335b3a5f125a1e5cbd0f6c79a7c3f719529b69d0abf2a5
main_test.go:32: Removing image sha256:08919f344b382fd1447da7f3e8ffd2a7125f5f7d191ed7d33242736dbe3c59cf
main_test.go:32: Removing image sha256:d144b3c13838e841ec319a17e1046471d726bb2aa3211e167a6a53f766a2dcdc
main_test.go:32: Removing image sha256:7c768ec742d628020f50c99dc5af32400b78534ca9fc4c01a9f00ec0ab19193a
main_test.go:32: Removing image sha256:4c26c71d142045fbf3448aa1f6363d5a7a803cb438a78a4b20b7c847df03d50a
main_test.go:32: Removing image sha256:08dcd63c964f2dbb17ff2665b6b86993fb14c0d3e169da187ac48f078a560d25
main_test.go:32: Removing image sha256:c1743fab233f36f2d7f83cb13f8c10ff06bdbda8f8a218d25a3796d1bc2f9e84
=== RUN TestBuildMultiStageLayerLeak
--- PASS: TestBuildMultiStageLayerLeak (5.59s)
main_test.go:32: Removing image sha256:5e9974558276c34d7c9aab3fad408d433047b7b15bdae43ab5048adc58a15431
main_test.go:32: Removing image sha256:ac3c613c8c48794237c1e46ac0657ebbb1132910a240086bb2e9df9770fdc017
main_test.go:32: Removing image sha256:a936df268131ad427f7b4b66ce3dbb1e41866d7269a4d383cebcb1c5930d3346
main_test.go:32: Removing image sha256:5e613ea0ce7dbc908b0315c49585ae43ad6c34158e9e0b59a3dc93b00ef0ea41
main_test.go:32: Removing image sha256:e676f4ec41a42823b6d91e05e3290d3827f9175dea6fba5d8b769aa13aa7e082
main_test.go:32: Removing image sha256:93c8daab2703126b23d957d4d6b04f07949356f1cd95d4f8fdbededf4ab5c21e
With this patch applied:
=== RUN TestBuildMultiStageOnBuild
--- PASS: TestBuildMultiStageOnBuild (6.74s)
=== RUN TestBuildUncleanTarFilenames
--- PASS: TestBuildUncleanTarFilenames (2.49s)
=== RUN TestBuildMultiStageLayerLeak
--- PASS: TestBuildMultiStageLayerLeak (5.14s)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
> HasAvailableFlags checks if the command contains any flags (local
> plus persistent from the entire structure) which are not hidden or
> deprecated.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Use a tagged release of Cobra. All relevant PR's were merged, so the fork is
no longer needed.
Relevant changes:
- spf13/cobra#552 Add a field to disable [flags] in UseLine()
- spf13/cobra#567 Add `CalledAs` method to cobra.Command
- spf13/cobra#580 Update error message for missing required flags
- spf13/cobra#584 Add support for --version flag
- spf13/cobra#614 If user has a project in symlink, just use its destination folder and work there
- spf13/cobra#649 terminates the flags when -- is found in commandline
- spf13/cobra#662 Add support for ignoring parse errors
- spf13/cobra#686 doc: hide hidden parent flags
Also various improvements were added for generating Bash
completion scripts (currently not used by us)
Fixes usage output for dockerd;
Before this update:
dockerd --help
Usage: dockerd COMMAND
A self-sufficient runtime for containers.
After this update:
dockerd --help
Usage: dockerd [OPTIONS] [flags]
A self-sufficient runtime for containers.
Bump spf13/pflag to v1.0.1
Relevant changes:
- spf13/pflag#106 allow lookup by shorthand
- spf13/pflag#113 Add SortFlags option
- spf13/pflag#138 Generate flag error output for errors returned from the parseFunc
- spf13/pflag#141 Fixing Count flag usage string
- spf13/pflag#143 add int16 flag
- spf13/pflag#122 DurationSlice: implementation and tests
- spf13/pflag#115 Implement BytesHex type of argument
- spf13/pflag#150 Add uintSlice and boolSlice to name prettifier
- spf13/pflag#155 Add multiline wrapping support
- spf13/pflag#158 doc: clarify difference between string slice vs. array
- spf13/pflag#160 add ability to ignore unknown flags
- spf13/pflag#163 Allow Users To Show Deprecated Flags
Hide [flags] in usage output
Hides the [flags] in the usage output of commands (present in newer
versions of Cobra), using the `.DisableFlagsInUseLine` option.
Before this change:
dockerd --help
Usage: dockerd [OPTIONS] [flags]
A self-sufficient runtime for containers.
After this change:
dockerd --help
Usage: dockerd [OPTIONS]
A self-sufficient runtime for containers.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Â# modified: vendor/github.com/spf13/pflag/string_array.go
§
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
ext4 support d_type by default, but filetype feature is a tunable so
there is still a chance to disable it for some reasons. In this case,
print additional message to explicitly tell how to support d_type.
Signed-off-by: Chengguang Xu <cgxu519@gmx.com>
Since we use `NewLabelCommand()` instead of `addNodesForLabelOption()`
to create the 'LABEL' commands from '--label' options, so all the related
functions should be removed.
Signed-off-by: Dennis Chen <dennis.chen@arm.com>
Add a new test case `TestBuildLabelWithTargets` to cover the Docker builder
with both `--label` and `--target` options.
Signed-off-by: Dennis Chen <dennis.chen@arm.com>
This PR is tring to fix issue #36996.
Currently for multi-stage build, if `--target` specified, the `--label` option
will be ignored. The root cause is the last stage build will remove the `LABEL`
command(s) node created from the `--label` option. In order to address this issue,
we can create `LABEL` command(s) and add it/tem to the last stage.
Signed-off-by: Dennis Chen <dennis.chen@arm.com>
This issue has been reported by issue #36877.
The purpose of this test case is for the regression test of #12546,
so we only need to make sure the essential of the testing is still
in the way to check that while not disturbed by some testing noises,
which is exactly what this PR want to do.
Signed-off-by: Dennis Chen <dennis.chen@arm.com>
In cases where a logging plugin has crashed when the daemon tries to
copy the container stdio to the logging plugin it returns a broken pipe
error and any log entries that occurr while the plugin is down are lost.
Fix this by opening read+write in the daemon so logs are not lost while
the plugin is down.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
- Check errors.Cause(err) when comparing errors
- Fix bug where oldest log file is not actually removed. This in
particular causes issues when compression is enabled. On rotate it just
overwrites the data in the log file corrupting it.
- Use O_TRUNC to open new gzip files to ensure we don't corrupt log
files as happens without the above fix.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Closing the log driver was in a defer meanwhile logs are
collected asyncronously, so the log driver was being closed before reads
were actually finished.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
In TestDockerNetworkConnectAlias the network and container
names used are unique which are not preferred. This
fix address the issue by appending t.Name() so that
names are randomized.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
This message was missing the name of the plugin, resulting in
plugin_test.go:92: timeout hit after 30s: plugin %!q(MISSING) exists
On failing tests.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Embedded new lines aren't interpreted correctly in markdown renderers (they are treated as preformatted text instead). I removed the embedded newlines in the docker image spec.
Signed-off-by: Alex Goodman <wagoodman@gmail.com>
In version-history.md, the link for `Docker Engine API v1.37`
was pointed to `v1.36`.
This fix fixes the incorrect link.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
This fix re-orders the CHANGELOG so that `make validate` test
could pass. The validation requires that the date in the changelog
is always ordered.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
The overlay storage driver currently does not support any option, but was silently
ignoring any option that was passed.
This patch verifies that no options are passed, and if they are passed will produce
an error.
Before this change:
dockerd --storage-driver=overlay --storage-opt dm.thinp_percent=95
INFO[2018-05-11T11:40:40.996597152Z] libcontainerd: started new docker-containerd process pid=256
....
INFO[2018-05-11T11:40:41.135392535Z] Daemon has completed initialization
INFO[2018-05-11T11:40:41.141035093Z] API listen on /var/run/docker.sock
After this change:
dockerd --storage-driver=overlay --storage-opt dm.thinp_percent=95
INFO[2018-05-11T11:39:21.632610319Z] libcontainerd: started new docker-containerd process pid=233
....
Error starting daemon: error initializing graphdriver: overlay: unknown option dm.thinp_percent
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
These network operations really don't have anything to do with the
container but rather are setting up the networking.
Ideally these wouldn't get shoved into the daemon package, but doing
something else (e.g. extract a network service into a new package) but
there's a lot more work to do in that regard.
In reality, this probably simplifies some of that work as it moves all
the network operations to the same place.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
A recent optimization in getSourceMount() made it return an error
in case when the found mount point is "/". This prevented bind-mounted
volumes from working in such cases.
A (rather trivial but adeqate) unit test case is added.
Fixes: 871c957242 ("getSourceMount(): simplify")
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
The Partial property of the Logger message
was replaced by PLogMetaData, causing the build to fail.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This is an old script using tools that are no longer maintained or
recommended (and don't even work anymore).
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
These are no longer used and instead users should use the
`container-selinux` package on their distribution. Additionally, these
are unmaintained and untested.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This is left-over stuff from building Docker pacakges. These aren't
really maintained outside of bumping the golang version, and are never
tested.
These builders can be found at
https://github.com/docker/docker-ce-packaging where they are kept up to
date.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
It does not make sense to copy a slice element by element, then discard
the source one. Let's do copy in place instead which is way more
efficient.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
When this test fails, the error looks like this:
> FAIL: docker_api_attach_test.go:98: DockerSuite.TestPostContainersAttach
> docker_api_attach_test.go:211:
> c.Assert(actualStdout.Bytes(), checker.DeepEquals, []byte("hello\nsuccess"), check.Commentf("Attach didn't return the expected data from stdout"))
> ... obtained []uint8 = []byte{0x73, 0x75, 0x63, 0x63, 0x65, 0x73, 0x73}
> ... expected []uint8 = []byte{0x68, 0x65, 0x6c, 0x6c, 0x6f, 0xa, 0x73, 0x75, 0x63, 0x63, 0x65, 0x73, 0x73}
> ... Attach didn't return the expected data from stdout
Let's use strings for comparisons to make the output more readable.
While at it,
- get the container's stderr as well, and make sure it's empty;
- check that stdcopy.StdCopy() did not return an error, except for
the timeout which is expected;
- move/remove comments, simplify var names.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
1. As daemon.ContainerStop() documentation says,
> If a negative number of seconds is given, ContainerStop
> will wait for a graceful termination.
but since commit cfdf84d5d0 (PR #32237) this is no longer the case.
This happens because `context.WithTimeout(ctx, timeout)` is implemented
as `WithDeadline(ctx, time.Now().Add(timeout))`, resulting in a deadline
which is in the past.
To fix, don't use WithDeadline() if the timeout is negative.
2. Add a test case to validate the correct behavior and
as a means to prevent a similar regression in the future.
3. Fix/improve daemon.ContainerStop() and client.ContainerStop()
description for clarity and completeness.
4. Fix/improve DefaultStopTimeout description.
Fixes: cfdf84d5d0 ("Update Container Wait")
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
This PR is trying to fix issue #36956.
The stage name is case-insensitive by design, so we should use
`strings.EqualFold()` as the comparison method to eliminate the
case sensitive noise.
Also we need to return a pre-defined error code order to avoid below
message like:
"FIXME: Got an API for which error does not match any expected type!!!:
failed to reach build target dev in Dockerfile"
Signed-off-by: Dennis Chen <dennis.chen@arm.com>
vendored distribution is quite old, and current distribution contains an
API break, which means it's not possible to vendor a bugfixed
distribution and a docker/docker at the same time.
Signed-off-by: Mike Lundy <mike@fluffypenguin.org>
In pkg/term/proxy.go and pkg/term/proxy_test.go, check if escapeKeys is empty and if it is, return the one key read
Signed-off-by: Patrik Cyvoct <patrik@ptrk.io>
When then non-blocking mode is specified, awslogs will:
- No longer potentially block calls to logstream.Log(), instead will
return an error if the awslogs buffer is full. This has the effect of
dropping log messages sent to awslogs.Log() that are made while the
buffer is full.
- Wait to initialize the log stream until the first Log() call instead of in
New(). This has the effect of allowing the container to start in
the case where Cloudwatch Logs is unreachable.
Both of these changes require the --log-opt mode=non-blocking to be
explicitly set and do not modify the default behavior.
Signed-off-by: Cody Roseborough <crrosebo@amazon.com>
We really need to run those on the CI too at some point.
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
Signed-off-by: Tibor Vass <tibor@docker.com>
Since Go 1.7, context is a standard package. Since Go 1.9, everything
that is provided by "x/net/context" is a couple of type aliases to
types in "context".
Many vendored packages still use x/net/context, so vendor entry remains
for now.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
It should check `os.Geteuid` with `uid` instead of `os.Getegid`.
On the container (where the tests run), the uid and gid seems to be
the same, thus this doesn't fail.
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
This fix consists of some improvement in restart_test.go
by replacing Fatal with assert, so that they are consistent
with other tests in integration/container.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
govet complains (when using standard "context" package):
> the cancel function returned by context.WithTimeout should be called,
> not discarded, to avoid a context leak (vet)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
We are using interface in the api routers to not explicitely depend on
the daemon struct (`daemon.Daemon`), but somehow, we do depend on the
`daemon` package for the cluster functionalities.
This removes this dependency by defining the correct interfaces.
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
Updates swarmkit to 33d06bf5189881b4d1e371b5571f4d3acf832816, to bring in
docker/swarmkit#2610 (Don't use wrappers for grpc metadata)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This is not for the sake of test to run faster of course;
this is to simplify the code as well as have some more
testing for mount.SingleEntryFilter().
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
There is no need to parse mount table and iterate through the list of
mounts, and then call Unmount() which again parses the mount table and
iterates through the list of mounts.
It is totally OK to call Unmount() unconditionally.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Now, every Unmount() call takes a burden to parse the whole nine yards
of /proc/self/mountinfo to figure out whether the given mount point is
mounted or not (and returns an error in case parsing fails somehow).
Instead, let's just call umount() and ignore EINVAL, which results
in the same behavior, but much better performance.
Note that EINVAL is returned from umount(2) not only in the case when
`target` is not mounted, but also for invalid flags. As the flags are
hardcoded here, it can't be the case.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
The mountinfo parser implemented via `fmt.Sscanf()` is slower than the one
using `strings.Split()` and `strconv.Atoi()`. This rewrite helps to speed it
up to a factor of 8x, here is a result from go bench:
> BenchmarkParsingScanf-4 300 22294112 ns/op
> BenchmarkParsingSplit-4 3000 2780703 ns/op
I tried other approaches, such as using `fmt.Sscanf()` for the first
three (integer) fields and `strings.Split()` for the rest, but it slows
things down considerably:
> BenchmarkParsingMixed-4 1000 8827058 ns/op
Note the old code uses `fmt.Sscanf`, when a linear search for '-' field,
when a split for the last 3 fields. The new code relies on a single
split.
I have also added more comments to aid in future development.
Finally, the test data is fixed to now have white space before the first field.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
The flow of getSourceMount was:
1 get all entries from /proc/self/mountinfo
2 do a linear search for the `source` directory
3 if found, return its data
4 get the parent directory of `source`, goto 2
The repeated linear search through the whole mountinfo (which can have
thousands of records) is inefficient. Instead, let's just
1 collect all the relevant records (only those mount points
that can be a parent of `source`)
2 find the record with the longest mountpath, return its data
This was tested manually with something like
```go
func TestGetSourceMount(t *testing.T) {
mnt, flags, err := getSourceMount("/sys/devices/msr/")
assert.NoError(t, err)
t.Logf("mnt: %v, flags: %v", mnt, flags)
}
```
...but it relies on having a specific mount points on the system
being used for testing.
[v2: add unit tests for ParentsFilter]
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Use mount.SingleEntryFilter as we're only interested in a single entry.
Test case data of TestShouldUnmountRoot is modified accordingly, as
from now on:
1. `info` can't be nil;
2. the mountpoint check is not performed (as SingleEntryFilter
guarantees it to be equal to daemon.root).
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Functions `GetMounts()` and `parseMountTable()` return all the entries
as read and parsed from /proc/self/mountinfo. In many cases the caller
is only interested only one or a few entries, not all of them.
One good example is `Mounted()` function, which looks for a specific
entry only. Another example is `RecursiveUnmount()` which is only
interested in mount under a specific path.
This commit adds `filter` argument to `GetMounts()` to implement
two things:
1. filter out entries a caller is not interested in
2. stop processing if a caller is found what it wanted
`nil` can be passed to get a backward-compatible behavior, i.e. return
all the entries.
A few filters are implemented:
- `PrefixFilter`: filters out all entries not under `prefix`
- `SingleEntryFilter`: looks for a specific entry
Finally, `Mounted()` is modified to use `SingleEntryFilter()`, and
`RecursiveUnmount()` is using `PrefixFilter()`.
Unit tests are added to check filters are working.
[v2: ditch NoFilter, use nil]
[v3: ditch GetMountsFiltered()]
[v4: add unit test for filters]
[v5: switch to gotestyourself]
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
The unit test is checking that setting of non-default StopTimeout
works, but it checked the value of StopSignal instead.
Amazingly, the test was working since the default StopSignal is SIGTERM,
which has the numeric value of 15.
Fixes: commit e66d21089 ("Add config parameter to change ...")
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
This moves the platform specific stuff in a separate package and keeps
the `volume` package and the defined interfaces light to import.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
All `plugins` tests that require an `ExperimentalDaemon` are migrated
to `integration/plugin/*` and start an experimental daemon to test on
it.
The end goal being to remove the `experimental` build.
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
This makes sure that if the daemon root was already a self-binded mount
(thus meaning the daemonc only performed a remount) that the daemon does
not try to unmount.
Example:
```
$ sudo mount --bind /var/lib/docker /var/lib/docker
$ sudo dockerd &
```
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
The primary need for this is x/net/context now is just a type alias to
the stdlib context package.
This makes issues with conflicts between "golang.org/x/net/context" and
the stdib "context" go away (primarily a concern in interface
definitions/implementations).
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Instead of using a global store for volume drivers, scope the driver
store to the caller (e.g. the volume store). This makes testing much
simpler.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Since the volume store already provides this functionality, we should
just use it rather than duplicating it.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This unblocks the client to take other restore requests and makes sure
that a long/stuck request can't block the client forever.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Porting helpers from `integration-cli/daemon.Daemon` to this struct
and use the API instead of the cli.
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
- Move go package used by both `integration-cli` and `integration` to
`internal/test/fixtures`.
- Remove fixtures that are not used anymore (moved to `docker/cli` a
while ago) : deploy, notary, secrets.
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
This fix converts some `client.ContainerCreate` to `container.Create`,
and removes some unneeded `name` fields when test containers are created.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
- Move the code from `integration-cli` to `internal/test`.
- Use `testingT` and `assert` when creating the registry.
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
In Go 1.10.1, the Deps for pkg\tarsum\tarsum_test.go are empty ([]) and
the PowerShell script ends up setting its import list to a string value
of False instead of an empty array. This can be remedied by forcing the
result to be an array, by concatenating to an empty array (@() + ...)
Signed-off-by: John Stephens <johnstep@docker.com>
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit ec3f9230d70506c536a24e844da0f0b3af9b43f6)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 7b30fd0e1d8bc77f0556181c82f85d046b058f27)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Add synchronization around adding logs to a plugin
and reading those logs. Without the follow configuration,
a race occurs between go routines to add the logs into
the plugin and read the logs out of the plugin. This
adds a function to synchronize the action to avoid the
race.
Removes use of file for buffering, instead buffering whole
messages so log count can be checked discretely.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Prevent changing the tar output by setting the format to
PAX and keeping the times truncated.
Without this change the archiver will produce different tar
archives with different hashes with go 1.10.
The addition of the access and changetime timestamps would
also cause diff comparisons to fail.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Remove invalid flush commands, flush should only occur when file
has been completely written. This is already handle, remove these calls.
Ensure data gets written after EOF in correct order and before close.
Remove gname and uname from sum for hash compatibility.
Update tarsum tests for gname/uname removal.
Return valid length after eof.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Docker daemon has a 16K buffer for log messages. If a message length
exceeds 16K, it should be split by the logger and merged at the
endpoint.
This change adds `PartialLogMetaData` struct for enhanced partial support
- LastPartial (bool) : indicates if this is the last of all partials.
- ID (string) : unique 32 bit ID. ID is same across all partials.
- Ordinal (int starts at 1) : indicates the position of msg in the series of partials.
Also, the timestamps across partials in the same.
Signed-off-by: Anusha Ragunathan <anusha.ragunathan@docker.com>
When the authz response buffer limit is hit, perform a flush.
This prevents excessive buffer sizes, especially on large responses
(e.g. `/containers/<id>/archive` or `/containers/<id>/export`).
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
All `Ipvlan` related test on `DockerSuite` and `DockerNetworkSuite`
are migrated to `ipvlan_test.go`.
The end goal being to remove the `experimental` builds.
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
… and do not use the `docker` cli in it. One of the reason of this
move is to not make `integration` package using legacy
`integration-cli` package.
Next move will be to support swarm within this package *and* provide
some helper function using the api (compared to the one using cli in
`integration-cli/daemon` package).
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
Currently we hardcode the architecture to the `runtime.GOARCH` when
building a docker image, this will result in a confusing info if the
arch in the base image is different from the one on the host.
This PR takes use of the arch data from the base image during the build
process, thus we can get consistent arch info between the base image
and the finally built image.
Signed-off-by: Dennis Chen <dennis.chen@arm.com>
It does not make any sense to vary this based on whether the
rootfs is read only. We removed all the other mount dependencies
on read-only eg see #35344.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
We have seen a panic when re-joining a node to a swarm cluster. The
cause of the issue is unknown, so we just need to add a test for nil
objects and log when we get the condition. Hopefully this can prevent
the crash and we can recover the config at a later time.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
* libnetwork#2121: Retry other external DNS servers on ServFail
* libnetwork#2125: Fix README flag and expose orphan network peers
* libnetwork#2126: Adding goreport card
* libnetwork#2130: Modify awk to use cut in check_ip_overlap
* libnetwork#2117: [Carry 1534] Improve scalabiltiy of bridge network isolation rules
Full changes: 2bf63300c5...5c1218c956
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
Changes included:
- Ingress network should not be attachable
- [manager/state] Add fernet as an option for raft encryption
- Log GRPC server errors
- Log leadership changes at manager level
- [state/raft] Increase raft ElectionTick to 10xHeartbeatTick
- Remove the containerd executor
- agent: backoff session when no remotes are available
- [ca/manager] Remove root CA key encryption support entirely
- Fix agent logging race (fixes https://github.com/docker/swarmkit/issues/2576)
- Adding logic to restore networks in order
Also adds github.com/fernet/fernet-go as a new dependency
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This change fixes bug when $CURDIR contains special character e.g. '&&'.
Change-Id: I658eb60d97c2229cf8b4e0a61dbbecf3edc9d573
Signed-off-by: Fangming.Fang <fangming.fang@arm.com>
commit 617c352e92 "Don't create devices if in a user namespace"
introduced check, which meant to skip mknod operation when run
in user namespace, but instread skipped FIFO and socket files
copy.
Signed-off-by: Maxim Ivanov <ivanov.maxim@gmail.com>
There is no need to perform a separate losetup step; mount (even
the one in busybox!) is smart enough to set up a loopback device
all by itself (even without -o loop present!). More to say, while
doing this, it sets LO_FLAGS_AUTOCLEAR flag for the kernel to
delete the loopback device as soon as its fs is unmounted (this
is supposed to work since kernel 2.6.25).
Also, remove mount options (-t ext4, -o loop,rw) as they are
either defaults (rw) or mount is smart enough to figure out.
Leave -n so it won't do unnecessary write to container's /etc/mtab.
While at it, touch up some comments.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
All `docker build` tests that require an `ExperimentalDaemon` are
migrated to `integration/build` package and start an experimental
daemon to test on it.
The end goal being to remove the `experimental` builds.
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
A minor nit. `test01` never been created and used in
`TestDockerNetworkInspectCustomSpecified()` function, so correct it.
Signed-off-by: Dennis Chen <dennis.chen@arm.com>
Using the default PollSettings functions to adjust the timeout
value instead of changing the value each time when needed.
Signed-off-by: Dennis Chen <dennis.chen@arm.com>
Add the default function per resource to override the `pollSettings`
which will be re-used where it's needed.
Signed-off-by: Dennis Chen <dennis.chen@arm.com>
This call was added as part of commit a042e5a20 and at the time was
useful. sandbox.DisableService() basically calls
endpoint.deleteServiceInfoFromCluster() for every endpoint in the
sandbox. However, with the libnetwork change, endpoint.sbLeave()
invokes endpoint.deleteServiceInfoFromCluster(). The releaseNetwork()
call invokes sandbox.Delete() immediately after
sandbox.DisableService(). The sandbox.Delete() in turn ultimately
invokes endpoint.sbLeave() for every endpoint in the sandbox which thus
removes the endpoint's load balancing entry via
endpoint.deleteServiceInfoFromCluster(). So the call to
sandbox.DisableService() is now redundant.
It is noteworthy that, while redundant, the presence of the call would
not cause errors. It would just be sub-optimal. The DisableService()
call would cause libnetwork to down-weight the load balancing entries
while the call to sandbox.Delete() would cause it to remove the entries
immediately afterwards. Aside from the wasted computation, the extra
call would also propagate an extra state change in the networkDB gossip
messages. So, overall, it is much better to just avoid the extra
overhead.
Signed-off-by: Chris Telfer <ctelfer@docker.com>
This patch allows endpoints to complete servicing connections while
being removed from a service. The fix is entirely within libnetwork
and requires no changes to the moby codebase proper. It operates
by initially down-weighting a container endpoint in the load balancer
to 0 while keeping the endpoint present in the load balancer. This
allows traffic to continue to flow to the endpoint while preventing new
connections from going to the endpoint. This allows the container
to complete requests during the "stop_grace_period" and then exit when
finished without interruption of service.
This change requires propagating the status of disabled service
endpoints via the networkDB. Accordingly, the patch includes both code
to generate and handle service update messages. It also augments the
service structure with a ServiceDisabled boolean to convey whether an
endpoint should ultimately be removed or just disabled. This,
naturally, required a rebuild of the protocol buffer code.
The protocol buffer encoding is designed to support additions of fields
to messages in a backwards-compatible manner. Protocol buffer
unmarshalling code automatically skips past any fields that it isn't
aware of. As a result, an older moby daemon without this fix can
receive and will process correctly networkDB messages from newer moby
daemons with this patch.
As it turns out, the additional field is simply a bool that is otherwise
irrelevent on networkDB create and delete events. So its absence in
older moby daemon processing has no impact. However, the fix leverages
the "update" networkDB message which was previously unused in
libnetwork. Although older libnetwork implementations parse the message
cleanly, they will see the message as unexpected and as such issue a log
at error level indicating the receipt of such.
Other than this there should be no other negative impact for use of this
patch in mixed environments. (Although older mobys won't be able to
gracefully downgrade connections on their nodes of course.)
Signed-off-by: Chris Telfer <ctelfer@docker.com>
This fixes an issue where the containerd client is cached in a container
object in libcontainerd and becomes stale after containerd is restarted.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This does not bump the containerd binary.
Picks last commit before go1.10 switch, which is not currently supported
in moby.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This was added as part of a53930a04f with
the intent to sort the mounts in the plugin config, but this was sorting
*all* the mounts from the default OCI spec which is problematic.
In reality we don't need to sort this because we are only adding a
self-binded mount to flag it as rshared.
We may want to look at sorting the plugin mounts before they are added
to the OCI spec in the future, but for now I think the existing behavior
is fine since the plugin author has control of the order (except for the
propagated mount).
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
All `Macvlan` related test on `DockerSuite` and `DockerNetworkSuite`
are migrated to `macvlan_test.go`.
Also, as `macvlan` seems to be out of experimental, this removes
the *skip* when the run is not experimental (and doesn't start a
daemon with experimental either).
The end goal being to remove the `experimental` builds.
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
Now all of the storage drivers use the field "storage-driver" in their log
messages, which is set to name of the respective driver.
Storage drivers changed:
- Aufs
- Btrfs
- Devicemapper
- Overlay
- Overlay 2
- Zfs
Signed-off-by: Alejandro GonzÃlez Hevia <alejandrgh11@gmail.com>
Using `dpkg --print-architecture` instead of the `uname -m` to abstract
the architecture value from the container images, which the build process
is running inside, to match exactly the behavior specified by the following
Docker file while not 'passthru' to the host.
Signed-off-by: Dennis Chen <dennis.chen@arm.com>
Since the recent release of CRIU has already supported other
arches such as AArch64, ppc64le, and s390x, so we can enable
it now.
Signed-off-by: Dennis Chen <dennis.chen@arm.com>
TestServiceWithPredefinedNetwork test case was failing
at times. To fix the issue, added new API to check
for services after we clean up all services. Tested
multiple times and this sould fix flaky issue.
Signed-off-by: selansen <elango.siva@docker.com>
This moves some of the code that was conditionally
executed on Windows to a separate, windows-only file.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
As soon as the initial executable in the container is executed as a non root user,
permitted and effective capabilities are dropped. Drop them earlier than this, so
that they are dropped before executing the file. The main effect of this is that
if `CAP_DAC_OVERRIDE` is set (the default) the user will not be able to execute
files they do not have permission to execute, which previously they could.
The old behaviour was somewhat surprising and the new one is definitely correct,
but it is not in any meaningful way exploitable, and I do not think it is
necessary to backport this fix. It is unlikely to have any negative effects as
almost all executables have world execute permission anyway.
Use the bounding set not the effective set as the canonical set of capabilities, as
effective will now vary.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
This is a test case for issue https://github.com/moby/moby/issues/35123,
making sure we can set container's memory limit when using
`native.cgroupdriver=systemd`.
[v2: skip if no systemd present]
[v3: add --iptables=false to avoid flaky tests with t.Parallel()]
[v4: rebase after PR#36507 merge]
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Commit fd0e24b718 changed
the stats collection loop to use a `sleep()` instead
of `time.Tick()` in the for-loop.
This change caused a regression in situations where
no stats are being collected, or an error is hit
in the loop (in which case the loop would `continue`,
and the `sleep()` is not hit).
This patch puts the sleep at the start of the loop
to guarantee it's always hit.
This will delay the sampling, which is similar to the
behavior before fd0e24b718.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This PR adds support for compressibility of log file.
I added a new option conpression for the jsonfile log driver,
this option allows the user to specify compression algorithm to
compress the log files. By default, the log files will be
not compressed. At present, only support 'gzip'.
Signed-off-by: Yanqiang Miao <miao.yanqiang@zte.com.cn>
'docker logs' can read from compressed files
Signed-off-by: Yanqiang Miao <miao.yanqiang@zte.com.cn>
Add Metadata to the gzip header, optmize 'readlog'
Signed-off-by: Yanqiang Miao <miao.yanqiang@zte.com.cn>
This test case checks that a container created before start
of the currently running dockerd can be exported (as reported
in #36561). To satisfy this condition, either a pre-existing
container is required, or a daemon restart after container
creation.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Update libnetwork to 1b91bc94094ecfdae41daa465cc0c8df37dfb3dd to bring in a fix
for stale HNS endpoints on Windows:
When Windows Server 2016 is restarted with the Docker service running, it is
possible for endpoints to be deleted from the libnetwork store without being
deleted from HNS. This does not occur if the Docker service is stopped cleanly
first, or forcibly terminated (since the endpoints still exist in both). This
change works around the issue by removing any stale HNS endpoints for a network
when creating it.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This updates libnetwork to 8892d7537c67232591f1f3af60587e3e77e61d41 to bring in
IPAM fixes for duplicate IP addresses.
- IPAM tests (libnetwork PR 2104) (no changes in vendored files)
- Fix for Duplicate IP issues (libnetwork PR 2105)
Also bump golang/x/sync to match libnetwork (no code-changes, other
than the README being updated)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
connection does. If this isn't done, then a container listening on stdin won't
receive an EOF when the client closes the stream at their end.
Signed-off-by: Jim Minter <jminter@redhat.com>
Commit 7a7357dae1 ("LCOW: Implemented support for docker cp + build")
changed `container.BaseFS` from being a string (that could be empty but
can't lead to nil pointer dereference) to containerfs.ContainerFS,
which could be be `nil` and so nil dereference is at least theoretically
possible, which leads to panic (i.e. engine crashes).
Such a panic can be avoided by carefully analysing the source code in all
the places that dereference a variable, to make the variable can't be nil.
Practically, this analisys are impossible as code is constantly
evolving.
Still, we need to avoid panics and crashes. A good way to do so is to
explicitly check that a variable is non-nil, returning an error
otherwise. Even in case such a check looks absolutely redundant,
further changes to the code might make it useful, and having an
extra check is not a big price to pay to avoid a panic.
This commit adds such checks for all the places where it is not obvious
that container.BaseFS is not nil (which in this case means we do not
call daemon.Mount() a few lines earlier).
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
In case ContainerExport() is called for an unmounted container, it leads
to a daemon panic as container.BaseFS, which is dereferenced here, is
nil.
To fix, do not rely on container.BaseFS; use the one returned from
rwlayer.Mount().
Fixes: 7a7357dae1 ("LCOW: Implemented support for docker cp + build")
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
With the ticker this could end up just doing back-to-back checks, which
isn't really what we want here.
Instead use a sleep to ensure we actually sleep for the desired
interval.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
In info, we only need the number of images, but `CountImages` was
getting the whole map of images and then grabbing the length from that.
This causes a lot of unnecessary CPU usage and memory allocations, which
increases with O(n) on the number of images.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
I noticed this test failed on Windows:
> 17:46:24 docker_cli_run_test.go:4361:
> 17:46:24 c.Fatal("running container timed out") // cleanup in teardown
I also noticed that in general tests are running slower on Windows,
for example TestStartAttachSilent (which runs a container with
`busybox echo test` and then starts it again) took 29.763s.
This means a simple container start can easily take 15s, which
explains the above failure.
Double the timeout from 15s to 30s.
Fixes: 4e262f6387 ("Fix race on sending stdin close event")
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
I am not quite sure why but this test is sometimes failing like this:
> 15:21:41 --- FAIL: TestLinksEtcHostsContentMatch (0.53s)
> 15:21:41 assertions.go:226:
>
> Error Trace: links_linux_test.go:46
> 15:21:41
> Error: Not equal:
> 15:21:41
> expected: "127.0.0.1\tlocalhost\n::1\tlocalhost
> ip6-localhost
> ip6-loopback\nfe00::0\tip6-localnet\nff00::0\tip6-mcastprefix\nff02::1\tip6-allnodes\nff02::2\tip6-allrouters\n172.17.0.2\tf53feb6df161\n"
> 15:21:41
> received: ""
To eliminate some possible failures (like ignoring stderr from `cat` or
its exit code), let's use container.Exec() to read a file from a container.
Fixes: e6bd20edcb ("Migrate some integration-cli test to api tests")
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
As mentioned in commit 9e31938, test cases that use t.Parallel()
and start a docker daemon might step on each other toes as they
try to configure iptables during startup, resulting in flaky tests.
To avoid this, --iptables=false should be used while starting daemon.
Fixes: eaa5192856 ("Make container resource mounts unbindable")
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Ingress networks will no longer automatically remove their
load-balancing endpoint (and sandbox) automatically when the network is
otherwise upopulated. This is to prevent automatic removal of the
ingress networks when all the containers leave them. Therefore
explicit removal of an ingress network also requires explicit removal
of its load-balancing endpoint.
Signed-off-by: Chris Telfer <ctelfer@docker.com>
This PR prevents automatic removal of the load balancing sandbox
endpoint when the endpoint is the last one in the network but
the network is marked as ingress.
Signed-off-by: Chris Telfer <ctelfer@docker.com>
The commit https://github.com/moby/moby/pull/35422 had the result of
accidentally causing the removal of the ingress network when the
last member of a service left the network. This did not appear
in swarm instances because the swarm manager would still maintain
and return cluster state about the network even though it had
removed its sandbox and endpoint. This test verifies that after a
service gets added and removed that the ingress sandbox remains
in a functional state.
Signed-off-by: Chris Telfer <ctelfer@docker.com>
Since now we have only one Dockerfile, so the arch-specific suffix
of the Dockerfile is not needed anymore.
Signed-off-by: Dennis Chen <dennis.chen@arm.com>
Removing all the existing arch-specific Dockerfiles since we already
have a new multi-arch supported one as the replacement.
Signed-off-by: Dennis Chen <dennis.chen@arm.com>
This PR consolidates the existing arch-specific Dockerfiles into only
one file `Dockefile` to ease the code maintenance effort.
Signed-off-by: Dennis Chen <dennis.chen@arm.com>
These tests were enabled by changing a config option on the ci
machines, instead of from a patch, so let me disable them
for now on ppc64le and open up another patch to enable them, where I can find
out what the issues are with them.
Signed-off-by: Christopher Jones <tophj@linux.vnet.ibm.com>
It has been pointed out that if --read-only flag is given, /dev/shm
also becomes read-only in case of --ipc private.
This happens because in this case the mount comes from OCI spec
(since commit 7120976d74), and is a regression caused by that
commit.
The meaning of --read-only flag is to only have a "main" container
filesystem read-only, not the auxiliary stuff (that includes /dev/shm,
other mounts and volumes, --tmpfs, /proc, /dev and so on).
So, let's make sure /dev/shm that comes from OCI spec is not made
read-only.
Fixes: 7120976d74 ("Implement none, private, and shareable ipc modes")
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
The test case checks that in case of IpcMode: private and
ReadonlyRootfs: true (as in "docker run --ipc private --read-only")
the resulting /dev/shm mount is NOT made read-only.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
To avoid noise in sampling CPU usage metrics, we now sample the system
usage closer to the actual response from the underlying runtime. Because
the response from the runtime may be delayed, this makes the sampling
more resilient in loaded conditions. In addition to this, we also
replace the tick with a sleep to avoid situations where ticks can backup
under loaded conditions.
The trade off here is slightly more load reading the system CPU usage
for each container. There may be an optimization required for large
amounts of containers but the cost is on the order of 15 ms per 1000
containers. If this becomes a problem, we can time slot the sampling,
but the complexity may not be worth it unless we can test further.
Unfortunately, there aren't really any good tests for this condition.
Triggering this behavior is highly system dependent. As a matter of
course, we should qualify the fix with the users that are affected.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Sorting by mount point length can be implemented in a more
straightforward fashion since Go 1.8 introduced sort.Slice()
with an ability to provide a less() function in place.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
This makes `go test .` to pass if run as non-root user, skipping
those tests that require superuser privileges (for `mount`).
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Layer metadata storage has not been implemented outside of the layer
store and will be deprecated by containerd metadata storage. To prepare
for this and freeze the current metadata storage, remove the exported
interface and make it internal to the layer store.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
With the inclusion of PR 30897, creating service for host network
fails in 18.02. Modified IsPreDefinedNetwork check and return
NetworkNameError instead of errdefs.Forbidden to address this issue
Signed-off-by: selansen <elango.siva@docker.com>
While a `types.go` file is handly when there are a lot of record types,
it is completely obnoxious when used for concrete, utility types with a
struct, new function and method set in the same file. This change
removes the `types.go` file in favor of the simpler approach.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
The types defined in the errdefs package do not
satisfy the `error` interface, because they do not
implement `Error()`.
Instead of returning the matched interface, return
the original error.
When matching _multiple_ interfaces/types, Golang doesn't complain:
func getImplementer(err error) error {
switch e := err.(type) {
case
ErrNotFound,
ErrInvalidParameter:
return e
default:
return err
}
}
But matching a single interface/type:
func getImplementer(err error) error {
switch e := err.(type) {
case
ErrNotFound:
return e
default:
return err
}
}
Produces an error:
cannot use e (type ErrNotFound) as type error in return argument: ErrNotFound does not implement error (missing Error method)
Return the original `err` instead of the matched interface/type instead.
Also added some additional tests
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This fix addresses `expected` vs `actual` in integration tests
so that they match `assert.Equal(t, expected, actual)`
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
Move the "unmount and deactivate" code into a separate method, and
optimize it a bit:
1. Do not use filepath.Walk() as there's no requirement to recursively
go into every directory under home/mnt; a list of directories in mnt
is sufficient. With filepath.Walk(), in case some container will fail
to unmount, it'll go through the whole container filesystem which is
excessive and useless.
2. Do not use GetMounts() and check if a directory is mounted; just
unmount it and ignore "not mounted" error. Note the same error
is returned in case of wrong flags set, but as flags are hardcoded
we can safely ignore such case.
While at it, promote "can't unmount" log level from debug to warning.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
1. Make sure it's clear the error is from unmount.
2. Simplify the code a bit to make it more readable.
[v2: use errors.Wrap]
[v3: use errors.Wrapf]
[v4: lowercase the error message]
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
1. Replace EnsureRemoveAll() with Rmdir(), as here we are removing
the container's mount point, which is already properly unmounted
and is therefore an empty directory.
2. Ignore the Rmdir() error (but log it unless it's ENOENT). This
is a mount point, currently unmounted (i.e. an empty directory),
and an older kernel can return EBUSY if e.g. the mount was
leaked to other mount namespaces.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
This fix adds several improvement:
1. No need for explicit ContainerRemove as it has been handled in setupTest()
2. Added `container.WithImage` helper function and used it in commit tests.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
Exec processes do not automatically inherit AppArmor
profiles from the container.
This patch sets the AppArmor profile for the exec
process.
Before this change:
apparmor_parser -q -r <<EOF
#include <tunables/global>
profile deny-write flags=(attach_disconnected) {
#include <abstractions/base>
file,
network,
deny /tmp/** w,
capability,
}
EOF
docker run -dit --security-opt "apparmor=deny-write" --name aa busybox
docker exec aa sh -c 'mkdir /tmp/test'
(no error)
With this change applied:
docker exec aa sh -c 'mkdir /tmp/test'
mkdir: can't create directory '/tmp/test': Permission denied
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The canonical import comment was added some time ago, though several
newly added files do not have the comment. This fix adds the missing
canonical import comment to files in integration tests
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
This fix is a minor enhancement to replace several ContainerCreate with
helper funcs of `container.Create` in tests.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
Fix runc exec on big-endian, causing:
container_linux.go:265: starting container process caused "open /dev/pts/4294967296: no such file or directory"
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Release notes: https://github.com/opencontainers/runc/releases/tag/v1.0.0-rc5
Possibly relevant changes included:
- chroot when no mount namespaces is provided
- fix systemd slice expansion so that it could be consumed by cAdvisor
- libcontainer/capabilities_linux: Drop os.Getpid() call
- Update console dependency to fix runc exec on BE (causing: `container_linux.go:265: starting container process caused "open /dev/pts/4294967296: no such file or directory"`)
- libcontainer: setupUserNamespace is always called (fixes: Devices are mounted with wrong uid/gid)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Changes Details:
--------------
Fixes: #36395
Refactoring the code to do the following:
1. Add the method `errBindSourceDoesNotExist` inside `validate.go` to be in-line with the rest of error message
2. Utilised the new method inside `linux_parser.go`, `windows_parser.go` and `validate_test.go`
3. Change the format from `bind mount source path: '%s' does not exist` to `bind mount source path does not exist: %s`
4. Reflected the format change into the 2 unit tests, namely: `volume_test.go` and `validate_test.go`
5. Reflected the format change into `docker_api_containers_test.go` integration test
Signed-off-by: Amr Gawish <amr.gawish@gmail.com>
Remove TestBuildRenamedDockerfile and TestBuildDockerfileOutsideContext
that are cli-only tests (and already tested in the docker/cli
repository).
Also adds some comments on few tests that could be migrate to
docker/cli.
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
Using the `busybox:glibc` instead of `busybox:latest` to the
network related test cases (`ping` issue).
Signed-off-by: Dennis Chen <dennis.chen@arm.com>
This fix migrates some secret create tests to api tests,
and remove redundant TestConfigCreate.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
This fix update docker-py so that containers from the tests run
could be cleaned up during teardown:
```diff
-ENV DOCKER_PY_COMMIT 5e28dcaace5f7b70cbe44c313b7a3b288fa38916
+ENV DOCKER_PY_COMMIT 8b246db271a85d6541dc458838627e89c683e42f
```
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
This fix moves helper functions containerIsStopped and
containerIsInState to integration/internal/container,
so that they could be used outside of integration/container.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
We need to clean the resources created in some test cases, else
in some cases we'll get below error for other tests:
> FAIL: docker_experimental_network_test.go:37: DockerNetworkSuite.TestDockerNetworkMacvlanPersistance
> docker_experimental_network_test.go:44:
> ...
> Command: ip link add dm-dummy0 type dummy
> ExitCode: 2
> Error: exit status 2
> Stdout:
> Stderr: RTNETLINK answers: File exists
> ...
Logically, each test case should be independent, the failure of previous
test case should not have side-effect for the test cases followed.
Signed-off-by: Dennis Chen <dennis.chen@arm.com>
Signed-off-by: John Howard <jhoward@microsoft.com>
While debugging #32838, it was found (https://github.com/moby/moby/issues/32838#issuecomment-356005845) that the utility VM in some circumstances was crashing. Unfortunately, this was silently thrown away, and as far as the build step (also applies to docker run) was concerned, the exit code was zero and the error was thrown away. Windows containers operate differently to containers on Linux, and there can be legitimate system errors during container shutdown after the init process exits. This PR handles this and passes the error all the way back to the client, and correctly causes a build step running a container which hits a system error to fail, rather than blindly trying to keep going, assuming all is good, and get a subsequent failure on a commit.
With this change, assuming an error occurs, here's an example of a failure which previous was reported as a commit error:
```
The command 'powershell -Command $ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue'; Install-WindowsFeature -Name Web-App-Dev ; Install-WindowsFeature -Name ADLDS; Install-WindowsFeature -Name Web-Mgmt-Compat; Install-WindowsFeature -Name Web-Mgmt-Service; Install-WindowsFeature -Name Web-Metabase; Install-WindowsFeature -Name Web-Lgcy-Scripting; Install-WindowsFeature -Name Web-WMI; Install-WindowsFeature -Name Web-WHC; Install-WindowsFeature -Name Web-Scripting-Tools; Install-WindowsFeature -Name Web-Net-Ext45; Install-WindowsFeature -Name Web-ASP; Install-WindowsFeature -Name Web-ISAPI-Ext; Install-WindowsFeature -Name Web-ISAPI-Filter; Install-WindowsFeature -Name Web-Default-Doc; Install-WindowsFeature -Name Web-Dir-Browsing; Install-WindowsFeature -Name Web-Http-Errors; Install-WindowsFeature -Name Web-Static-Content; Install-WindowsFeature -Name Web-Http-Redirect; Install-WindowsFeature -Name Web-DAV-Publishing; Install-WindowsFeature -Name Web-Health; Install-WindowsFeature -Name Web-Http-Logging; Install-WindowsFeature -Name Web-Custom-Logging; Install-WindowsFeature -Name Web-Log-Libraries; Install-WindowsFeature -Name Web-Request-Monitor; Install-WindowsFeature -Name Web-Http-Tracing; Install-WindowsFeature -Name Web-Stat-Compression; Install-WindowsFeature -Name Web-Dyn-Compression; Install-WindowsFeature -Name Web-Security; Install-WindowsFeature -Name Web-Windows-Auth; Install-WindowsFeature -Name Web-Basic-Auth; Install-WindowsFeature -Name Web-Url-Auth; Install-WindowsFeature -Name Web-WebSockets; Install-WindowsFeature -Name Web-AppInit; Install-WindowsFeature -Name NET-WCF-HTTP-Activation45; Install-WindowsFeature -Name NET-WCF-Pipe-Activation45; Install-WindowsFeature -Name NET-WCF-TCP-Activation45;' returned a non-zero code: 4294967295: container shutdown failed: container ba9c65054d42d4830fb25ef55e4ab3287550345aa1a2bb265df4e5bfcd79c78a encountered an error during WaitTimeout: failure in a Windows system call: The compute system exited unexpectedly. (0xc0370106)
```
Without this change, it would be incorrectly reported such as in this comment: https://github.com/moby/moby/issues/32838#issuecomment-309621097
```
Step 3/8 : ADD buildtools C:/buildtools
re-exec error: exit status 1: output: time="2017-06-20T11:37:38+10:00" level=error msg="hcsshim::ImportLayer failed in Win32: The system cannot find the path specified. (0x3) layerId=\\\\?\\C:\\ProgramData\\docker\\windowsfilter\\b41d28c95f98368b73fc192cb9205700e21
6691495c1f9ac79b9b04ec4923ea2 flavour=1 folder=C:\\Windows\\TEMP\\hcs232661915"
hcsshim::ImportLayer failed in Win32: The system cannot find the path specified. (0x3) layerId=\\?\C:\ProgramData\docker\windowsfilter\b41d28c95f98368b73fc192cb9205700e216691495c1f9ac79b9b04ec4923ea2 flavour=1 folder=C:\Windows\TEMP\hcs232661915
```
This images is used to run integration and integration-cli tests on
anything that implements the docker api :). The image wasn't building
anywore :D
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
Originally I worked on this for the multi-stage build Dockerfile
changes. Decided to split this out as we are still waiting for
multi-stage to be available on CI and rebasing these is pretty annoying.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This fixes an issue where the container LogPath was empty when the
non-blocking logging mode was enabled. This change sets the LogPath on
the container as soon as the path is generated, instead of setting the
LogPath on a logger struct and then attempting to pull it off that
logger at a later point. That attempt to pull the LogPath off the logger
was error prone since it assumed that the logger would only ever be a
single type.
Prior to this change docker inspect returned an empty string for
LogPath. This caused issues with tools that rely on docker inspect
output to discover container logs, e.g. Kubernetes.
This commit also removes some LogPath methods that are now unnecessary
and are never invoked.
Signed-off-by: junzhe and mnussbaum <code@getbraintree.com>
Test made some bad assumptions about on-disk state of volume data.
This updates the test to only test based on what the volume API is
designed to provide.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
1. Use integration/internal/exec, removing the getContainerSysFSValue().
2. Avoid repeating magic numbers, use a variable for those.
3. Fix order of arguments to assert.Equal (first "expected", then "actual").
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
An implementation of exec in TestUpdateCPUQUota had a few issues,
including resource leaking and calling both ContainerExecAttach and
ContainerExecRun. The last one makes the test flaky:
update_linux_test.go:136: expected cgroup value 20000, got: Error: Exec
command f923baf709525f6b38f6511126addc5d9bb88fb477eeca1c22440551090fa2bb
is already running
Fix by using the integration/internal/exec package.
While at it, use require/assert to further improve code readability.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Some test cases might need an ability to execute a command inside a
container (in order to analyse its output and/or exit code). It is a bit
complicated operation to do so using engine API. The function provided
aims to hide this complexity, making exec almost as simple as 'docker
exec'.
NOTE that the exec is synchronous, and command's stdin is closed.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
When a container is created if "--network" is set to "host" all the
ports in the container are bound to the host.
Thus, adding "-p" or "--publish" to the command-line is meaningless.
Unlike "docker run" and "docker create", "docker service create" sends
an error message when network mode is host and port bindings are given
This patch however suggests to send a warning message to the client when
such a case occurs.
The warning message is added to "warnings" which are returned from
"verifyPlatformContainerSettings".
Signed-off-by: Boaz Shuster <ripcurld.github@gmail.com>
This updates the containerd/continuity package to d8fb8589b0e8e85b8c8bbaa8840226d0dfeb7371
which fixes builds failing on ARM 32-bit, after this dependency was added in
b3aab5e31f
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Makes sure if the user specifies an older API version that we don't pass
through templating options for versions that templating was not
supported.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
On unix, merge secrets/configs handling. This is important because
configs can contain secrets (via templating) and potentially a config
could just simply have secret information "by accident" from the user.
This just make sure that configs are as secure as secrets and de-dups a
lot of code.
Generally this makes everything simpler and configs more secure.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
The BurntSushi/toml code is now re-licensed as MIT. While
the vendored package was already updated, the tomlv binary
used was still using the old license type.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
dm_task_deferred_remove is not supported by all distributions, due to
out-dated versions of devicemapper. However, in the case where the
devicemapper library was updated without rebuilding Docker (which can
happen in some distributions) then we should attempt to dynamically load
the relevant object rather than try to link to it.
This can only be done if Docker was built dynamically, for obvious
reasons.
In order to avoid having issues arise when dlsym(3) was unnecessary,
gate the whole dlsym(3) logic behind a buildflag that we disable by
default (libdm_dlsym_deferred_remove).
Signed-off-by: Aleksa Sarai <asarai@suse.de>
These are now more in-line with wiki instructions. Also removes
broken/deprecated make target test-unit.
Signed-off-by: Brett Randall <javabrett@gmail.com>
Update and unify the `busybox` images on all arches to the `glibc` multi-arch
version and remove the temp workaround on amd64 which uses the old version
busybox (v1.26) before this PR to bypass the failure of those network related
test cases. Also, this PR will fix all the network related issues with `glibc`
version `busybox` image.
Signed-off-by: Dennis Chen <dennis.chen@arm.com>
When tailing a container log, if the log file is empty it will cause the
log stream to abort with an unexpected `EOF`.
Note that this only applies to the "current" log file as rotated files
cannot be empty.
This fix just skips adding the "current" file the log tail if it is
empty.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This fix is related to 36219
This fix updates runc to:
```
-RUNC_COMMIT=9f9c96235cc97674e935002fc3d78361b696a69e
+RUNC_COMMIT=6c55f98695e902427906eed2c799e566e3d3dfb5
-github.com/opencontainers/runc 9f9c96235cc97674e935002fc3d78361b696a69e
+github.com/opencontainers/runc 6c55f98695e902427906eed2c799e566e3d3dfb5
```
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
This fix is a sync up with 36266 so that relevant api tests
use the newly added container.Run/Create in helper package
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
The PKCS8 changes updated the encryption on the keys so that the
`x509.IsEncryptedPEMBlock` may no longer return true because it cannot
parse the PEM block. The `keyutils` module in SwarmKit can tell whether
it is encrypted either way.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This fix is a follow up to 36266 to update some api tests
to use the newly added container helper package.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
To help creating/running/… containers using the client for test integration.
This should make test more readable and reduce duplication a bit.
Usage example
```
// Create a default container named foo
id1 := container.Create(t, ctx, client, container.WithName("foo"))
// Run a default container with a custom command
id2 := container.Run(t, ctx, client, container.WithCmd("echo", "hello world"))
```
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
Both names have no real sense, but one allows to make sure these packages
aren't used outside of `integration`.
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
- Replace EC Private Key with PKCS#8 PEMs
- Fix IP overlap with empty EndpointSpec
- Add support for Support SCTP port mapping (depends on changes in libnetwork)
- [orchestrator/updater] Do not reschedule tasks if only placement constraints change and are satisfied by the assigned node
- Ensure task reaper stopChan is closed no more than once
- [manager/dispatcher] Synchronization fixes
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
A recent change accidently caused any TLS configuration in FromEnv to be
ignored. This change alters WithHost to create a new http client only if
one doesn't already exist, and otherwise applies the logic to the
transport on the existing client. This preserves the TLS configuration
that might already be on the client.
Signed-off-by: Drew Erny <drew.erny@docker.com>
When the daemon restores containers on daemon restart, it syncs up with
containerd to determine the existing state. For stopped containers it
then removes the container metadata from containerd.
In some cases this is not handled properly and causes an error when
someone attempts to start that container again.
In particular, this case is just a bad error check.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Since commit e9b9e4ace2 has landed, there is a chance that
container.RWLayer is nil (due to some half-removed container). Let's
check the pointer before use to avoid any potential nil pointer
dereferences, resulting in a daemon crash.
Note that even without the abovementioned commit, it's better to perform
an extra check (even it's totally redundant) rather than to have a
possibility of a daemon crash. In other words, better be safe than
sorry.
[v2: add a test case for daemon.getInspectData]
[v3: add a check for container.Dead and a special error for the case]
Fixes: e9b9e4ace2
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Service of inspect network can't be finished within 10s on AArch64 platform,
so we need to adjust the timeout value avoid to paper cover the real issue, plus
to make the integreation test can continue while not terminate with below error:
> === RUN TestInspectNetwork
> --- FAIL: TestInspectNetwork (27.65s)
> daemon.go:285: [de79880f4ed4a] waiting for daemon to start
> daemon.go:317: [de79880f4ed4a] daemon started
> inspect_test.go:57: timeout hit after 10s: waiting for tasks to enter run state
> daemon.go:275: [de79880f4ed4a] exiting daemon
> FAIL
> ---> Making bundle: .integration-daemon-stop (in bundles/test-integration)
> Removing test suite binaries
> Makefile:171: recipe for target 'test-integration' failed
Signed-off-by: Dennis Chen <dennis.chen@arm.com>
The imageRefs map was being popualted with containerID, and accessed
with an imageID which would never match.
Remove this broken code because: 1) it hasn't ever worked so isn't
necessary, and 2) because at best it would be racy
ImageDelete() should already handle preventing of removal of used
images.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
This fix migrates TestCreateTmpfsMountsTarget test to api test,
and removed integration-cli/docker_cli_create_unix_test.go
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
This allows to create a client with default values and override those
using functors. As an example, `NewEnvClient()` becomes
`NewClientWithOpts(FromEnv)` ; and if you want a different api version
for this client : `NewClientWithOpts(FromEnv, WithVersion("1.35"))`
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
This workaround for golang/go#15286 was added for Nano server TP5 in
fa82c0aa10, and should no longer be
needed
Due to a security fix in Go 1.9.4/1.8.7, loading the .dll is no longer
allowed, and produces an error:
.\docker_windows.go:9:3: //go:cgo_import_dynamic main.dummy CommandLineToArgvW%2 "shell32.dll" only allowed in cgo-generated code
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This fixes a vulnerability in `go get` (CVE-2018-6574, http://golang.org/issue/23672),
but shouldn't really affect our code, but it's good to keep in sync.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This reverts commit 0c2821d6f2.
Due to other changes this is no longer needed and resolves some other
issues with plugins.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Before this change, volume management was relying on the fact that
everything the plugin mounts is visible on the host within the plugin's
rootfs. In practice this caused some issues with mount leaks, so we
changed the behavior such that mounts are not visible on the plugin's
rootfs, but available outside of it, which breaks volume management.
To fix the issue, allow the plugin to scope the path correctly rather
than assuming that everything is visible in `p.Rootfs`.
In practice this is just scoping the `PropagatedMount` paths to the
correct host path.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Setting up the mounts on the host increases chances of mount leakage and
makes for more cleanup after the plugin has stopped.
With this change all mounts for the plugin are performed by the
container runtime and automatically cleaned up when the container exits.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Currently the metrics plugin uses a really hackish host mount with
propagated mounts to get the metrics socket into a plugin after the
plugin is alreay running.
This approach ends up leaking mounts which requires setting the plugin
manager root to private, which causes some other issues.
With this change, plugin subsystems can register a set of modifiers to
apply to the plugin's runtime spec before the plugin is ever started.
This will help to generalize some of the customization work that needs
to happen for various plugin subsystems (and future ones).
Specifically it lets the metrics plugin subsystem append a mount to the
runtime spec to mount the metrics socket in the plugin's mount namespace
rather than the host's and prevetns any leaking due to this mount.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
The goal of this refactor is to make it easier to integrate buildkit
and containerd snapshotters.
Commit is used from two places (api and build), each calls it
with distinct arguments. Refactored to pull out the common commit
logic and provide different interfaces for each consumer.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
This was added in #36047 just as a way to make sure the tree is fully
unmounted on shutdown.
For ZFS this could be a breaking change since there was no unmount before.
Someone could have setup the zfs tree themselves. It would be better, if
we really do want the cleanup to actually the unpacked layers checking
for mounts rather than a blind recursive unmount of the root.
BTRFS does not use mounts and does not need to unmount anyway.
These was only an unmount to begin with because for some reason the
btrfs tree was being moutned with `private` propagation.
For the other graphdrivers that still have a recursive unmount here...
these were already being unmounted and performing the recursive unmount
shouldn't break anything. If anyone had anything mounted at the
graphdriver location it would have been unmounted on shutdown anyway.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
By default, if a user requests a bind mount it uses private propagation.
When the source path is a path within the daemon root this, along with
some other propagation values that the user can use, causes issues when
the daemon tries to remove a mountpoint because a container will then
have a private reference to that mount which prevents removal.
Unmouting with MNT_DETATCH can help this scenario on newer kernels, but
ultimately this is just covering up the problem and doesn't actually
free up the underlying resources until all references are destroyed.
This change does essentially 2 things:
1. Change the default propagation when unspecified to `rslave` when the
source path is within the daemon root path or a parent of the daemon
root (because everything is using rbinds).
2. Creates a validation error on create when the user tries to specify
an unacceptable propagation mode for these paths...
basically the only two acceptable modes are `rslave` and `rshared`.
In cases where we have used the new default propagation but the
underlying filesystem is not setup to handle it (fs must hvae at least
rshared propagation) instead of erroring out like we normally would,
this falls back to the old default mode of `private`, which preserves
backwards compatibility.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Before this patch, when containerd is restarted (due to a crash, or
kill, whatever), the daemon would keep trying to process the event
stream against the old socket handles. This would lead to a CPU spin due
to the error handling when the client can't connect to containerd.
This change makes sure the containerd remote client is updated for all
registered libcontainerd clients.
This is not neccessarily the ideal fix which would likely require a
major refactor, but at least gets things to a working state with a
minimal patch.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
If a blank nodeID was previously passed in it resulted in a node list
request. The response would then fail to umarshal into a `Node`
type returning a JSON error.
This adds an extra validation to all inspect calls to check that the ID
that is required is provided and if not return an error.
Signed-off-by: Emil Davtyan <emil2k@gmail.com>
This fix is a follow up to 36198 by adding description
to TestContainerNetworkMountsNoChown so that it is clear
about the purpose of the test for ownership.
This fix is related to comment in 36198.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
As there is already a runSimpleContainer, I think it makes
sense to combine runSimpleContainer with runContainer for rename test
to reduce code duplication.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
This fix migrates TestContainersAPINetworkMountsNoChown from
integration-cli to api tests in integration.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
Current implementaion of docke daemon doesn't pass down the
`--oom-kill-disable` option specified by the end user to the containerd
when spawning a new docker instance with help from `runc` component, which
results in the `--oom-kill-disable` doesn't work no matter the flag is `true`
or `false`.
This PR will fix this issue reported by #36090
Signed-off-by: Dennis Chen <dennis.chen@arm.com>
Signed-off-by: Jianyong Wu <jianyong.wu@arm.com>
The plugin spec says that plugins can live in one of:
- /var/run/docker/plugins/<name>.sock
- /var/run/docker/plugins/<name>/<name>.sock
- /etc/docker/plugins/<name>.[json,spec]
- /etc/docker/plugins/<name>/<name>.<json,spec>
- /usr/lib/docker/plugins/<name>.<json,spec>
- /usr/lib/docker/plugins/<name>/<name>.<json,spec>
However, the plugin scanner which is used by the volume list API was
doing `filepath.Walk`, which will walk the entire tree for each of the
supported paths.
This means that even v2 plugins in
`/var/run/docker/plugins/<id>/<name>.sock` were being detected as a v1
plugin.
When the v1 plugin loader tried to load such a plugin it would log an
error that it couldn't find it because it doesn't match one of the
supported patterns... e.g. when in a subdir, the subdir name must match
the plugin name for the socket.
There is no behavior change as the error is only on the `Scan()` call,
which is passing names to the plugin registry when someone calls the
volume list API.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Previously adding files to vendor/ without adding to vendor.conf would not fail the
validation.
Also be consistent with indentation and use tabs.
Signed-off-by: Daniel Nephin <dnephin@gmail.com>
Attachable networks are networks created on the cluster which can then
be attached to by non-swarm containers. These networks are lazily
created on the node that wants to attach to that network.
When no container is currently attached to one of these networks on a
node, and then multiple containers which want that network are started
concurrently, this can cause a race condition in the network attachment
where essentially we try to attach the same network to the node twice.
To easily reproduce this issue you must use a multi-node cluster with a
worker node that has lots of CPUs (I used a 36 CPU node).
Repro steps:
1. On manager, `docker network create -d overlay --attachable test`
2. On worker, `docker create --restart=always --network test busybox
top`, many times... 200 is a good number (but not much more due to
subnet size restrictions)
3. Restart the daemon
When the daemon restarts, it will attempt to start all those containers
simultaneously. Note that you could try to do this yourself over the API,
but it's harder to trigger due to the added latency from going over
the API.
The error produced happens when the daemon tries to start the container
upon allocating the network resources:
```
attaching to network failed, make sure your network options are correct and check manager logs: context deadline exceeded
```
What happens here is the worker makes a network attachment request to
the manager. This is an async call which in the happy case would cause a
task to be placed on the node, which the worker is waiting for to get
the network configuration.
In the case of this race, the error ocurrs on the manager like this:
```
task allocation failure" error="failed during network allocation for task n7bwwwbymj2o2h9asqkza8gom: failed to allocate network IP for task n7bwwwbymj2o2h9asqkza8gom network rj4szie2zfauqnpgh4eri1yue: could not find an available IP" module=node node.id=u3489c490fx1df8onlyfo1v6e
```
The task is not created and the worker times out waiting for the task.
---
The mitigation for this is to make sure that only one attachment reuest
is in flight for a given network at a time *when the network doesn't
already exist on the node*. If the network already exists on the node
there is no need for synchronization because the network is already
allocated and on the node so there is no need to request it from the
manager.
This basically comes down to a race with `Find(network) ||
Create(network)` without any sort of syncronization.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This fix tries to address the issue raised in 36139 where
ExitCode and PID does not show up in Task.Status.ContainerStatus
The issue was caused by `json:",omitempty"` in PID and ExitCode
which interprate 0 as null.
This is confusion as ExitCode 0 does have a meaning.
This fix removes `json:",omitempty"` in ExitCode and PID,
but changes ContainerStatus to pointer so that ContainerStatus
does not show up at all if no content. If ContainerStatus
does have a content, then ExitCode and PID will show up (even if
they are 0).
This fix fixes 36139.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
ReleaseRWLayer can and should only be called once (unless it returns
an error), but might be called twice in case of a failure from
`system.EnsureRemoveAll(container.Root)`. This results in the
following error:
> Error response from daemon: driver "XXX" failed to remove root filesystem for YYY: layer not retained
The obvious fix is to set container.RWLayer to nil as soon as
ReleaseRWLayer() succeeds.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Moves builder/shell_parser and into its own subpackage at builder/shell since it
has no dependencies other than the standard library. This will make it
much easier to vendor for downstream libraries, without pulling all the
dependencies of builder/.
Fixes#36154
Signed-off-by: Matt Rickard <mrick@google.com>
This fix tries to address the issue raised in 36142 where
there are discrepancies between Swarm API and swagger.yaml.
This fix adds two recently added state `REMOVE` and `ORPHANED` to TaskState.
This fix fixes 36142.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
This fix tries to address the issue raised in 36042
where secret and config are not configured with the
specified file mode.
This fix update the file mode so that it is not impacted
with umask.
Additional tests have been added.
This fix fixes 36042.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
Daemon flags that can be specified multiple times use
singlar names for flags, but plural names for the configuration
file.
To make the daemon configuration know how to correlate
the flag with the corresponding configuration option,
`opt.NewNamedListOptsRef()` should be used instead of
`opt.NewListOptsRef()`.
Commit 6702ac590e attempted
to fix the daemon not corresponding the flag with the configuration
file option, but did so by changing the name of the flag
to plural.
This patch reverts that change, and uses `opt.NewNamedListOptsRef()`
instead.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
You don't need to resolve the symlink for the exec as long as the
process is to keep running during execution.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
During a plugin remove, docker performs an `os.Rename` to move the
plugin data dir to a new location before removing to acheive an atomic
removal.
`os.Rename` can return either a `NotExist` error if the source path
doesn't exist, or an `Exist` error if the target path already exists.
Both these cases can happen when there is an error on the final
`os.Remove` call, which is common on older kernels (`device or resource
busy`).
When calling rename, we can safely ignore these error types and proceed
to try and remove the plugin.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This change sets an explicit mount propagation for the daemon root.
This is useful for people who need to bind mount the docker daemon root
into a container.
Since bind mounting the daemon root should only ever happen with at
least `rlsave` propagation (to prevent the container from holding
references to mounts making it impossible for the daemon to clean up its
resources), we should make sure the user is actually able to this.
Most modern systems have shared root (`/`) propagation by default
already, however there are some cases where this may not be so
(e.g. potentially docker-in-docker scenarios, but also other cases).
So this just gives the daemon a little more control here and provides
a more uniform experience across different systems.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This fix tries to address the issue raised in 36083 where
`network inspect` does not show Created time if the network is
created in swarm scope.
The issue was that Created was not converted from swarm api.
This fix addresses the issue.
An unit test has been added.
This fix fixes 36083.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
This fix tries to address the issue raised in 33661 where
network alias does not work when connect to a network the second time.
This fix address the issue.
This fix fixes 33661.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
This fix tries to address the issue raised in 35752
where container start will trigger a crash if EndpointSettings is nil.
This fix adds the validation to make sure EndpointSettings != nil
This fix fixes 35752.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
When succesfully reloading the daemon configuration, print a message
in the logs with the active configuration:
INFO[2018-01-15T15:36:20.901688317Z] Got signal to reload configuration, reloading from: /etc/docker/daemon.json
INFO[2018-01-14T02:23:48.782769942Z] Reloaded configuration: {"mtu":1500,"pidfile":"/var/run/docker.pid","data-root":"/var/lib/docker","exec-root":"/var/run/docker","group":"docker","deprecated-key-path":"/etc/docker/key.json","max-concurrent-downloads":3,"max-concurrent-uploads":5,"shutdown-timeout":15,"debug":true,"hosts":["unix:///var/run/docker.sock"],"log-level":"info","swarm-default-advertise-addr":"","metrics-addr":"","log-driver":"json-file","ip":"0.0.0.0","icc":true,"iptables":true,"ip-forward":true,"ip-masq":true,"userland-proxy":true,"disable-legacy-registry":true,"experimental":false,"network-control-plane-mtu":1500,"runtimes":{"runc":{"path":"docker-runc"}},"default-runtime":"runc","oom-score-adjust":-500,"default-shm-size":67108864,"default-ipc-mode":"shareable"}
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This fix updates aws-sdk-go and go-ini to recent versions.
The aws-sdk-go used to be `v1.4.22` which was more than a
year old, and go-ini used to be pre-1.0 release.
This fix updates aws-sdk-go to v1.12.66 and go-ini to v1.25.4:
```
github.com/aws/aws-sdk-go v1.12.66
github.com/go-ini/ini v1.25.4
```
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
This fix carries PR 34248: Added tag log option to json-logger
This fix changes to use RawAttrs based on review feedback.
This fix fixes 19803, this fix closes 34248.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
Fixes#19803
Updated the json-logger to utilize the common log option
'tag' that can define container/image information to include
as part of logging.
When the 'tag' log option is not included, there is no change
to the log content via the json-logger. When the 'tag' log option
is included, the tag will be parsed as a template and the result
will be stored within each log entry as the attribute 'tag'.
Update: Removing test added to integration_cli as those have been deprecated.
Update: Using proper test calls (require and assert) in jsonfilelog_test.go based on review.
Update: Added new unit test configs for logs with tag. Updated unit test error checking.
Update: Cleanup check in jsonlogbytes_test.go to match pending changes in PR #34946.
Update: Merging to correct conflicts from PR #34946.
Signed-off-by: bonczj <josh.bonczkowski@gmail.com>
Signed-off-by: John Howard <jhoward@microsoft.com>
The re-coalesces the daemon stores which were split as part of the
original LCOW implementation.
This is part of the work discussed in https://github.com/moby/moby/issues/34617,
in particular see the document linked to in that issue.
The idea behind making the graphdrivers private is to prevent leaking
mounts into other namespaces.
Unfortunately this is not really what happens.
There is one case where this does work, and that is when the namespace
was created before the daemon's namespace.
However with systemd each system servie winds up with it's own mount
namespace. This causes a race betwen daemon startup and other system
services as to if the mount is actually private.
This also means there is a negative impact when other system services
are started while the daemon is running.
Basically there are too many things that the daemon does not have
control over (nor should it) to be able to protect against these kinds
of leakages. One thing is certain, setting the graphdriver roots to
private disconnects the mount ns heirarchy preventing propagation of
unmounts... new mounts are of course not propagated either, but the
behavior is racey (or just bad in the case of restarting services)... so
it's better to just be able to keep mount propagation in tact.
It also does not protect situations like `-v
/var/lib/docker:/var/lib/docker` where all mounts are recursively bound
into the container anyway.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Libcontainer no longer provides placeholders for
unsupported platforms, which cause the Windows
builds to fail.
This patch moves features that are not supported
to platform-specific files.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This PR contains a fix for moby/moby#30321. There was a moby/moby#31142
PR intending to fix the issue by adding a delay between disabling the
service in the cluster and the shutdown of the tasks. However
disabling the service was not deleting the service info in the cluster.
Added a fix to delete service info from cluster and verified using siege
to ensure there is zero downtime on rolling update of a service.In order
to support it and ensure consitency of enabling and disable service knob
from the daemon, we need to ensure we disable service when we release
the network from the container. This helps in making the enable and
disable service less racy. The corresponding part of libnetwork fix is
part of docker/libnetwork#1824
Signed-off-by: abhi <abhi@docker.com>
A linter (vet) found the following bug in the code:
> daemon/metrics.go:124::error: range variable p captured by func literal (vet)
Here a variable p is used in an async fashion by goroutine, and most
probably by the time of use it is set to the last element of a range.
For example, the following code
```go
for _, c := range []string{"here ", "we ", "go"} {
go func() {
fmt.Print(c)
}()
}
```
will print `gogogo` rather than `here we go` as one would expect.
Fixes: 0e8e8f0f31 ("Add support for metrics plugins")
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
1. Add = between the option and the argument, otherwise the argument
appears to be passed on to the linters directly, as in:
> DEBUG: [golint.8]: executing /home/kir/go/bin/golint
> -min_confidence 0.800000 ./10m ./api ./api/errdefs <...>
2. Fix setting the default for GOMETALINTER_OPTS -- the default
was -deadline (rather than --deadline).
Fixes: b96093fa56 ("gometalinter: add per-platform configurable options")
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Because we merge master into the branch before running tests, so the
actual git sha does not exist on any git remote.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
It's a common scenario for admins and/or monitoring applications to
mount in the daemon root dir into a container. When doing so all mounts
get coppied into the container, often with private references.
This can prevent removal of a container due to the various mounts that
must be configured before a container is started (for example, for
shared /dev/shm, or secrets) being leaked into another namespace,
usually with private references.
This is particularly problematic on older kernels (e.g. RHEL < 7.4)
where a mount may be active in another namespace and attempting to
remove a mountpoint which is active in another namespace fails.
This change moves all container resource mounts into a common directory
so that the directory can be made unbindable.
What this does is prevents sub-mounts of this new directory from leaking
into other namespaces when mounted with `rbind`... which is how all
binds are handled for containers.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
The `POST /volumes/create` expects a request body to be provided.
If no body was provided, a 500 status was returned. A 500 status
is incorrect, because the request is invalid (it's not a server
error).
Before this change:
$ curl --unix-socket /var/run/docker.sock -v -X POST http://localhost/volumes/create
* Trying /var/run/docker.sock...
* Connected to localhost (/Users/sebastiaan/Library/Containers/com.dock) port 80 (#0)
> POST /volumes/create HTTP/1.1
> Host: localhost
> User-Agent: curl/7.51.0
> Accept: */*
>
< HTTP/1.1 500 Internal Server Error
< Api-Version: 1.30
< Content-Length: 18
< Content-Type: application/json
< Date: Wed, 19 Jul 2017 11:29:26 GMT
< Docker-Experimental: true
< Ostype: linux
< Server: Docker/17.06.0-ce (linux)
<
{"message":"EOF"}
* Curl_http_done: called premature == 0
* Connection #0 to host localhost left intact
After this change:
$ curl --unix-socket /var/run/docker.sock -v -X POST http://localhost/volumes/create
* Trying /var/run/docker.sock...
* Connected to localhost (/var/run/docker.sock) port 80 (#0)
> POST /volumes/create HTTP/1.1
> Host: localhost
> User-Agent: curl/7.52.1
> Accept: */*
>
< HTTP/1.1 400 Bad Request
< Api-Version: 1.36
< Content-Type: application/json
< Docker-Experimental: false
< Ostype: linux
< Server: Docker/dev (linux)
< Date: Tue, 09 Jan 2018 15:00:13 GMT
< Content-Length: 42
<
{"message":"no body provided in request"}
* Curl_http_done: called premature == 0
* Connection #0 to host localhost left intact
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The Golang built-in gzip library is serialized, and fairly slow
at decompressing. It also only decompresses on demand, versus
pipelining decompression.
This change switches to using the pigz external command
for gzip decompression, as opposed to using the built-in
golang one. This code is not vendored, but will be used
if it autodetected as part of the OS.
This also switches to using context, versus a manually
managed channel to manage cancellations, and synchronization.
There is a little bit of weirdness around manually having
to cancel in the error cases.
Signed-off-by: Sargun Dhillon <sargun@sargun.me>
Fixes an issue where if cpu quota/period is sent via the update API, the
values are updated in the stored container data but not actually sent to
the running container.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
PR #36011 fixed almost all of the golint issues though
there is still one golint error:
https://goreportcard.com/report/github.com/docker/docker#golint
```
Golint is a linter for Go source code.
docker/daemon/reload.go
Line 64: warning: redundant if ...; err != nil check, just return error instead. (golint)
```
This fix fixes the last one.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
This fix is a follow up to 30397, with `FindUniqueNetwork`
changed to `FindNetwork` based on the review feedback.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
Some improvements to the test;
- Combine tests to reduce duplicated code
- Add test-cases for empty version in request using the default version
- Add test for valid versions in request actually setting the version
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Both lcow_parser.go and linux_parser.go are duplicating the error:
"invalid specification: destination can't be '/'"
This commit creates a new error called "ErrVolumeTargetIsRoot"
that is used by both linux_parser and lcow_parser and remove
the duplication in the code.
Signed-off-by: Boaz Shuster <ripcurld.github@gmail.com>
The pattern `echo str | grep -qE pattern` likes to fail on the z CI here for
an unknown reason. Use `grep -qE pattern <<< str` instead.
Signed-off-by: Christopher Jones <tophj@linux.vnet.ibm.com>
Now we only adjust the timeout value for `arm` while not `arm64`,
actually the avarage duration for this test is about 25s to crate
multiple services on arm64, else the integration test will terminate
with below error:
> --- FAIL: TestCreateServiceMultipleTimes (24.11s)
> daemon.go:285: [ddc3c7c1476c2] waiting for daemon to start
> daemon.go:317: [ddc3c7c1476c2] daemon started
> poll.go:121: timeout hit after 10s: task count at 4 waiting for 0
> daemon.go:275: [ddc3c7c1476c2] exiting daemon
> clean.go:108: Removing image sha256:e6a8d12d58602a19277ee5632b7ff9fa56a4ea52ba00eedf1d3f6f5a495fe761
> clean.go:108: Removing image sha256:876244cc2ecb8fe1b0b2e817e3b78709a2a735edb093bc6849f99aa6c18f3a01
This PR adjusts the timeout value for both `arm64` and `arm` to mitigate
this issue on those 2 platforms.
Signed-off-by: Dennis Chen <dennis.chen@arm.com>
Instead of having to create a bunch of custom error types that are doing
nothing but wrapping another error in sub-packages, use a common helper
to create errors of the requested type.
e.g. instead of re-implementing this over and over:
```go
type notFoundError struct {
cause error
}
func(e notFoundError) Error() string {
return e.cause.Error()
}
func(e notFoundError) NotFound() {}
func(e notFoundError) Cause() error {
return e.cause
}
```
Packages can instead just do:
```
errdefs.NotFound(err)
```
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
When a recursive unmount fails, don't bother parsing the mount table to check
if what we expected to be a mountpoint is still mounted. `EINVAL` is
returned when you try to unmount something that is not a mountpoint, the
other cases of `EINVAL` would not apply here unless everything is just
wrong. Parsing the mount table over and over is relatively expensive,
especially in the code path that it's in.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Using parallel tests is nice, however it can cause an issue with
multiple daemons trying to make changes to iptables at the same time
which causes flakey tests.
This just disables iptables for the set of tests since it is not
required.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
BusyBox v1.26.2 (2017-03-09 00:04:38 UTC) supports `-le` option
to get the full date and time information, while BusyBox v1.27.2
(2017-11-01 23:22:25 UTC, which is used by the official multi-arch
image, uses `--full-time` instead of `-e` to get the same data. As
a result, we will get below error for the `DockerSuite.TestBuildLastModified`
test case in case of multi-arch image used:
> docker_cli_build_test.go:446:
> out2 = cli.DockerCmd(c, "run", name, "ls", "-le", "/file").Combined()
> o/src/github.com/docker/docker/vendor/github.com/gotestyourself/gotestyourself/icmd/command.go:61:
> t.Fatalf("at %s:%d - %s\n", filepath.Base(file), line, err.Error())
> ... Error: at cli.go:33 -
> Command: /usr/local/bin/docker run testbuildlastmodified ls -le /file
> ExitCode: 1
> Error: exit status 1
> Stdout:
> Stderr: ls: invalid option -- e
> BusyBox v1.27.2 (2017-11-01 23:22:25 UTC) multi-call binary.
This PR tries to fix the above compatible issue for busybox image.
Signed-off-by: Dennis Chen <dennis.chen@arm.com>
We don't need the test image namespace anymore since we've already
upgrade those images to the latest multi-arch ones.
Signed-off-by: Dennis Chen <dennis.chen@arm.com>
Upgrade the frozen images to the multi-arch ones.
Since issue #35963 is not fixed yet on linux/amd64, so we keep the busybox
image on amd64 untouched.
Signed-off-by: Dennis Chen <dennis.chen@arm.com>
This fix updates doc for CopyFromContainer to explicitly
mention that the content received from the reader is
a TAR archive.
This fix closes 35965.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
Actually the specification was expecting a 'additionalProperties' for the Volumes data, where in fact it's expecting
a map of string pointing to empty object.
Signed-off-by: Joel Wurtz <joel.wurtz@gmail.com>
This fix is part of the effort to address 30242 where
issue arise because of the fact that multiple networks
may share the same name (within or across local/swarm scopes).
The focus of this fix is to allow creation of service
when a network in local scope has the same name as the
service network.
An integration test has been added.
This fix fixes 30242.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
This fix tries to address the issue raised in 35931 where
`before` and `since` filter for `docker ps` does not work
and returns an error
```
Error response from daemon: no such container <container_name>
```
The issue was that `before` and `since` filter are matched
with `view.Get()` which does not take into considerations
of name match.
This fix fixes the issue by adding additional logic for name
match.
This fix fixes 35931.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
This fix tries to address the issue raised in 35920 where the filter
of `docker ps` with `health=starting` always returns nothing.
The issue was that in container view, the human readable string (`HealthString()` => `Health.String()`)
of health status was used. In case of starting it is `"health: starting"`.
However, the filter still uses `starting` so no match returned.
This fix fixes the issue by using `container.Health.Status()` instead so that it matches
the string (`starting`) passed by filter.
This fix fixes 35920.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
This fix adds a test case for 35333: Devicemapper: ignore Nodata errors when delete thin device
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
if thin device is deteled and the metadata exists, you can not
delete related containers. This patch ignore Nodata errors for
thin device deletion
Signed-off-by: Liu Hua <sdu.liu@huawei.com>
The event filter used two separate filter-conditions for
"namespace" and "topic". As a result, both events matching
"topic" and events matching "namespace" were subscribed to,
causing events to be handled both by the "plugin" client, and
"container" client.
This patch rewrites the filter to match only if both namespace
and topic match.
Thanks to Stephen Day for providing the correct filter :)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Further to 355cf9483c which caught some
of these. This should fix the remainder in the contributing docs.
Signed-off-by: Brett Randall <javabrett@gmail.com>
According to https://github.com/golang/go/issues/5373, go recognizes
(and optimizes for) the following syntax:
```go
for i := range b {
b[i] = 0
}
```
so let's use it. Limited testing shows ~7.5x speed increase,
compared to the previously used syntax.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
I got the following test failure on power:
10:00:56
10:00:56
----------------------------------------------------------------------
10:00:56 FAIL: docker_cli_build_test.go:3521:
DockerSuite.TestBuildNotVerboseFailureRemote
10:00:56
10:00:56 docker_cli_build_test.go:3536:
10:00:56 c.Fatal(fmt.Errorf("Test[%s] expected that quiet stderr and
verbose stdout are equal; quiet [%v], verbose [%v]", name,
quietResult.Stderr(), result.Combined()))
10:00:56 ... Error: Test[quiet_build_wrong_remote] expected that quiet
stderr and verbose stdout are equal; quiet [
10:00:56 unable to prepare context: unable to download remote context
http://something.invalid: Get http://something.invalid: dial tcp: lookup
something.invalid on 172.29.128.11:53: no such host
10:00:56 ], verbose [unable to prepare context: unable to download
remote context http://something.invalid: Get http://something.invalid:
dial tcp: lookup something.invalid on 8.8.8.8:53: no such host
10:00:56 ]
10:00:56
10:00:56
10:00:56
----------------------------------------------------------------------
The reason is, either more than one name server is configured, or
nameserver was reconfigured in the middle of the test run. In any case,
different nameserver IP in an error messages should not be treated
as a failure, so let's strip those out.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
`TestCleanupMountsAfterDaemonAndContainerKill` was supposedly written
when the container mounts were visible from the host. Currently they
all live in their own mount namespace and the only visible mount is
the tmpfs one for shareable /dev/shm inside the container (i.e.
/var/lib/docker/containers/<ID>/shm), which will no longer be there
in case of `--default-ipc-mode private` is used, and so the test will
fail. Add a check if any container mounts are visible from the host,
and skip the test if there are none, as there's nothing to check.
`TestCleanupMountsAfterDaemonCrash`: fix in a similar way, keeping
all the other checks it does, and skipping the "mounts gone" check
if there were no mounts visible from the host.
While at it, also fix the tests to use `d.Kill()` in order to not
leave behind a stale `docker.pid` files.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
1. The functionality of this test is superceded by
`TestAPIIpcModeShareableAndContainer` (see
integration-cli/docker_api_ipcmode_test.go).
2. This test won't work with --default-ipc-mode private.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
The building machinery was being handed an uninitialized container
Config. This changes it to use the target container's Config.
Resolves#30538
Signed-off-by: Anthony Sottile <asottile@umich.edu>
The `Status` field was deprecated in favor of `Action`.
This patch updates the test to use the `Action` field,
but adds a check that both are set to the same value.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Commit 59d45c384a changed
the `eventsLimit` from 64 to 256, but did not update
the GoDoc accordingly.
This patch updates the GoDoc for `Subscribe` and `SubscribeTopic`
to match the actual limit.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This test was added a long time ago, and over the years has proven to be flaky,
and slow. To address those issues, it was modified to;
- cleanup containers afterwards
- take clock-skew into account
- improve performance by parallelizing the container runs
- _reduce_ parallelization to address platform issues on Windows (twice..)
- adjust the test to take new limits into account
- adjust the test to account for more events being generated by containers
The last change to this test (made in ddae20c032) actually
broke the test, as it's now testing that all events sent by containers
(`numContainers*eventPerContainer`) are received, but the number of events that
is generated (17 containers * 7 events = 119) is less than the limit (256 events).
The limit is already covered by the `TestLogEvents` unit-test, that was added in
8d056423f8, and tests that the number of events
is limited to `eventsLimit`.
This patch removes the test, because it's not needed.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
When compiling containerd binaries statically, linker rightfully
complains:
+ make BUILDTAGS=static_build 'EXTRA_FLAGS=-buildmode pie' 'EXTRA_LDFLAGS=-extldflags "-fno-PIC -static"'
🇩 bin/ctr
# github.com/containerd/containerd/cmd/ctr
/tmp/go-link-343047789/000000.o: In function `_cgo_b0c710f30cfd_C2func_getaddrinfo':
/tmp/go-build/net/_obj/cgo-gcc-prolog:46: warning: Using 'getaddrinfo'
in statically linked applications requires at runtime the shared
libraries from the glibc version used for linking
The same error appears for ctr, containerd, and containerd-stress
binaries.
The fix is to use Go's own DNS resolver functions, rather than
glibc's getaddrinfo() -- this option is turned on by `netgo` build tag.
See https://golang.org/pkg/net/ (look for "Name Resolution") for more
details.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
This is a fix to regression in vfs graph driver introduced by
commit 7a1618ced3 ("add quota support to VFS graphdriver").
On some filesystems, vfs fails to init with the following error:
> Error starting daemon: error initializing graphdriver: Failed to mknod
> /go/src/github.com/docker/docker/bundles/test-integration/d6bcf6de610e9/root/vfs/backingFsBlockDev:
> function not implemented
As quota is not essential for vfs, let's ignore (but log as a warning) any error
from quota init.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Validation of Mounts was only performed on container _creation_, not on
container _start_. As a result, if the host-path no longer existed
when the container was started, a directory was created in the given
location.
This is the wrong behavior, because when using the `Mounts` API, host paths
should never be created, and an error should be produced instead.
This patch adds a validation step on container start, and produces an
error if the host path is not found.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
When the containerd 1.0 runtime changes were made, we inadvertantly
removed the functionality where any running containers are killed on
startup when not using live-restore.
This change restores that behavior.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
If mknod() returns ENOSYS, it most probably means quota is not supported
here, so return the appropriate error.
This is a conservative* fix to regression in vfs graph driver introduced
by commit 7a1618ced3 ("add quota support to VFS graphdriver").
On some filesystems, vfs fails to init with the following error:
> Error starting daemon: error initializing graphdriver: Failed to mknod
> /go/src/github.com/docker/docker/bundles/test-integration/d6bcf6de610e9/root/vfs/backingFsBlockDev:
> function not implemented
Reported-by: Brian Goff <cpuguy83@gmail.com>
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Files that are suffixed with `_linux.go` or `_windows.go` are
already only built on Linux / Windows, so these build-tags
were redundant.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Follow the conventions for namespace naming set out by other projects,
such as linuxkit and cri-containerd. Typically, they are some sort of
host name, with a subdomain describing functionality of the namespace.
In the case of linuxkit, services are launched in `services.linuxkit`.
In cri-containerd, pods are launched in `k8s.io`, making it clear that
these are from kubernetes.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
With the contianerd 1.0 migration we now have strongly typed errors that
we can check for process not found.
We also had some bad error checks looking for `ESRCH` which would only
be returned from `unix.Kill` and never from containerd even though we
were checking containerd responses for it.
Fixes some race conditions around process handling and our error checks
that could lead to errors that propagate up to the user that should not.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Seen failing on Windows:
22:27:52 ----------------------------------------------------------------------
22:27:52 PANIC: docker_api_logs_test.go:152: DockerSuite.TestLogsAPIUntil
22:27:52
22:27:52 ... Panic: runtime error: index out of range (PC=0x45AC01)
22:27:52
22:27:52 d:/CI/CI-7caa30e89/go/src/runtime/asm_amd64.s:509
22:27:52 in call32
22:27:52 d:/CI/CI-7caa30e89/go/src/runtime/panic.go:491
22:27:52 in gopanic
22:27:52 d:/CI/CI-7caa30e89/go/src/runtime/panic.go:28
22:27:52 in panicindex
22:27:52 docker_api_logs_test.go:175
22:27:52 in DockerSuite.TestLogsAPIUntil
22:27:52 d:/CI/CI-7caa30e89/go/src/runtime/asm_amd64.s:509
22:27:52 in call32
22:27:52 d:/CI/CI-7caa30e89/go/src/reflect/value.go:434
22:27:52 in Value.call
22:27:52 d:/CI/CI-7caa30e89/go/src/reflect/value.go:302
22:27:52 in Value.Call
22:27:52 c:/gopath/src/github.com/docker/docker/vendor/github.com/go-check/check/check.go:816
22:27:52 in suiteRunner.forkTest.func1
22:27:52 c:/gopath/src/github.com/docker/docker/vendor/github.com/go-check/check/check.go:672
22:27:52 in suiteRunner.forkCall.func1
22:27:52 d:/CI/CI-7caa30e89/go/src/runtime/asm_amd64.s:2337
22:27:52 in goexit
22:27:54
22:27:54 ----------------------------------------------------------------------
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The `repository:shortid` syntax for referencing images is very little used,
collides with with tag references can be confused with digest references.
The `repository:shortid` notation was deprecated in Docker 1.13 through
5fc71599a0, and scheduled for removal
in Docker 17.12.
This patch removes the support for this notation.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Currently we only support 'application/vnd.docker.distribution.manifest.v2+json'
manifest images download, with more multi-arch images used, we need to support
download images with 'application/vnd.docker.distribution.manifest.list.v2+json'
format(aka "fat manifest"), else we will fail to download those multi-arch ones.
This PR adds 'application/vnd.docker.distribution.manifest.list.v2+json' manifest
support, thus we can download both multi-arch and legacy images.
Signed-off-by: Dennis Chen <dennis.chen@arm.com>
The overlay2 driver was not setting up the archive.TarOptions field
properly like other storage backend routes to "applyTarLayer"
functionality. The InUserNS field is populated now for overlay2 using
the same query function used by the other storage drivers.
Signed-off-by: Phil Estes <estesp@linux.vnet.ibm.com>
The previous bytes counter was moved out of scope was not counting the
total number of bytes in the batch. This type encapsulates the counter
and the batch for consideration and code ergonomics.
Signed-off-by: Jacob Vallejo <jakeev@amazon.com>
Also added back some of the maintainer processes that were in
MAINTAINERS but moved to docker/opensource repo. I believe this
project's governance should be disconnected from docker/opensource as
project's remaining under docker/opensource will not use the Moby TSC.
Signed-off-by: Phil Estes <estesp@linux.vnet.ibm.com>
We are planning to remove supports for non-Linux platform in
runc (https://github.com/opencontainers/runc/pull/1654). The current
import here is the only thing that i found in docker that is windows-related
so fixing this would remove the rest of windows code in runc.
This changes some functions in daemon_windows to be the same as
daemon_unix to use runtime-spec public API instead of runc.
Signed-off-by: Daniel Dao <dqminh89@gmail.com>
The `docker info` command compares the installed version
of containerd using a Git-sha. We currently use a tag for
this, but that tag is not returned by the version-API of
containerd, resulting in the `docker info` output to show:
containerd version: 89623f28b87a6004d4b785663257362d1658a729 (expected: v1.0.0)
This patch changes the `v1.0.0` tag to the commit that
corresponds with the tag, so that the `docker info` output
does not show the `expected:` string.
This should be considered a temporary workaround; the check
for the exact version of containerd that's installed was needed
when we still used the 0.2.x branch, because it did not have
stable releases yet.
With containerd reaching 1.0, and using SemVer, we can likely
do a comparison for "Major" version, or make this a "packaging"
issue, and remove the check entirely (we can still _print_ the
version that's installed if we think it's usefule).
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Replace those legacy docker images prefixed by 'aarch64/' and 'arm64v8/'
with official multi-arch ones.
Signed-off-by: Dennis Chen <dennis.chen@arm.com>
Commit ee594dcb7d removed the
`sleep 0.5` from this test, because sleep has a full-second
precision. However, in some cases, all three log-entries
are output at the same time, causing the `--until` filter
to fail.
This patch adds back a `sleep`, but uses 1 second instead.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Right now we only log source and destination (and demsg) if mount operation
fails. fstype and mount options are available easily. It probably is a good
idea to log these as well. Especially sometimes failures can happen due to
mount options.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Interacting with v1 registries was deprecated in Docker 1.8.3, disabled by default
in Docker 17.06, and scheduled for removal in Docker 17.12.
This patch disallows enabling V1 registry through the `--disable-legacy-registry`
option, and the `"disable-legacy-registry": false` option in the daemon configuration
file. The actual V1 registry code is still in place, and will be removed separately.
With this patch applied:
$ dockerd --disable-legacy-registry=false
ERROR: The '--disable-legacy-registry' flag has been removed. Interacting with legacy (v1) registries is no longer supported
Or, when setting through the `daemon.json` configuration file
$ mkdir -p /etc/docker/
$ echo '{"disable-legacy-registry":false}' > /etc/docker/daemon.json
$ dockerd
ERROR: The 'disable-legacy-registry' configuration option has been removed. Interacting with legacy (v1) registries is no longer supported
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The installation steps for go-swagger was a bit noisy, and not consistent with
other installation steps.
This patch makes it similar to other steps, which makes it less noisy, and
makes the image slightly smaller.
Before:
b53d7aac3200 14 minutes ago |1 APT_MIRROR=deb.debian.org /bin/sh -c git … 107MB
fa74acf32f99 2 hours ago /bin/sh -c #(nop) ENV GO_SWAGGER_COMMIT=c28… 0B
After:
6b2454f1a9a5 10 minutes ago |1 APT_MIRROR=deb.debian.org /bin/sh -c set … 35.2MB
fa74acf32f99 2 hours ago /bin/sh -c #(nop) ENV GO_SWAGGER_COMMIT=c28… 0B
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This sleep probably doesn't work because sleep typically takes only
whole numbers, to do < 1s you need to use usleep. It also is not really
needed as the loop has a very low bound that will not eat up too much
CPU.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Before this patch, if a splunk endpoint returned a non-OK status, the
splunk driver would read the entire response body. This could lead to
excessive memory allocations. Instead put a cap on the amount the driver
will read from the response body.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
If there is a package-rename forthcoming, it isn't currently evident on master, but seems to show in doc.
Changed doc to show actual docker/docker paths as they occur now instead of moby/moby.
Signed-off-by: Brett Randall <javabrett@gmail.com>
This change adds a Platform struct with a Name field and a general
Components field to the Version API type. This will allow API
consumers to show version information for the whole platform and
it will allow API providers to set the versions for the various
components of the platform.
All changes here are backwards compatible.
Signed-off-by: Tibor Vass <tibor@docker.com>
Add a new configuration option to allow the enabling
of the networkDB debug. The option is only parsed using the
reload event. This will protect the daemon on start or restart
if the option is left behind in the config file
Signed-off-by: Flavio Crisciani <flavio.crisciani@docker.com>
When the client closes websocket connections that sends container
output through websocket, an error message is displayed:
"Error attaching websocket: %!s(<nil>)"
This message is misleading. Thus, this change suggests to check
if error is nil and print the correct message accordingly.
Signed-off-by: Boaz Shuster <ripcurld.github@gmail.com>
Ensures that when a plugin is removed that it doesn't interfere with
other plugins mounts and also ensures its own mounts are cleaned up.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Even though it's highly discouraged, there are existing
installs that are running overlay/overlay2 on filesystems
without d_type support.
This patch allows the daemon to start in such cases, instead of
refusing to start without an option to override.
For fresh installs, backing filesystems without d_type support
will still cause the overlay/overlay2 drivers to be marked as
"unsupported", and skipped during the automatic selection.
This feature is only to keep backward compatibility, but
will be removed at some point.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Support for running overlay/overlay2 on a backing filesystem
without d_type support (most likely: xfs, as ext4 supports
this by default), was deprecated for some time.
Running without d_type support is problematic, and can
lead to difficult to debug issues ("invalid argument" errors,
or unable to remove files from the container's filesystem).
This patch turns the warning that was previously printed
into an "unsupported" error, so that the overlay/overlay2
drivers are not automatically selected when detecting supported
storage drivers.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The fsmagic check was always performed on "data-root" (`/var/lib/docker`),
not on the storage-driver's home directory (e.g. `/var/lib/docker/<somedriver>`).
This caused detection to be done on the wrong filesystem in situations
where `/var/lib/docker/<somedriver>` was a mount, and a different
filesystem than `/var/lib/docker` itself.
This patch checks if the storage-driver's home directory exists, and only
falls back to `/var/lib/docker` if it doesn't exist.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This commit is a set of fixes and improvement for zfs graph driver,
in particular:
1. Remove mount point after umount in `Get()` error path, as well
as in `Put()` (with `MNT_DETACH` flag). This should solve "failed
to remove root filesystem for <ID> .... dataset is busy" error
reported in Moby issue 35642.
To reproduce the issue:
- start dockerd with zfs
- docker run -d --name c1 --rm busybox top
- docker run -d --name c2 --rm busybox top
- docker stop c1
- docker rm c1
Output when the bug is present:
```
Error response from daemon: driver "zfs" failed to remove root
filesystem for XXX : exit status 1: "/sbin/zfs zfs destroy -r
scratch/docker/YYY" => cannot destroy 'scratch/docker/YYY':
dataset is busy
```
Output when the bug is fixed:
```
Error: No such container: c1
```
(as the container has been successfully autoremoved on stop)
2. Fix/improve error handling in `Get()` -- do not try to umount
if `refcount` > 0
3. Simplifies unmount in `Get()`. Specifically, remove call to
`graphdriver.Mounted()` (which checks if fs is mounted using
`statfs()` and check for fs type) and `mount.Unmount()` (which
parses `/proc/self/mountinfo`). Calling `unix.Unmount()` is
simple and sufficient.
4. Add unmounting of driver's home to `Cleanup()`.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Previously, the code would set the mtime on the directories before
creating files in the directory itself. This was problematic
because it resulted in the mtimes on the directories being
incorrectly set. This change makes it so that the mtime is
set only _after_ all of the files have been created.
Signed-off-by: Sargun Dhillon <sargun@sargun.me>
The error type libnetwork.ErrNoSuchNetwork is used in the controller
to retry the network creation as a managed network though the manager.
The change of the type was breaking the logic causing the network to
not being created anymore so that no new container on that network
was able to be launched
Added unit test
Signed-off-by: Flavio Crisciani <flavio.crisciani@docker.com>
There was a small issue here, where it copied the data using
traditional mechanisms, even when copy_file_range was successful.
Signed-off-by: Sargun Dhillon <sargun@sargun.me>
This change makes the VFS graphdriver use the kernel-accelerated
(copy_file_range) mechanism of copying files, which is able to
leverage reflinks.
Signed-off-by: Sargun Dhillon <sargun@sargun.me>
Previously, graphdriver/copy would improperly copy hardlinks as just regular
files. This patch changes that behaviour, and instead the code now keeps
track of inode numbers, and if it sees the same inode number again
during the copy loop, it hardlinks it, instead of copying it.
Signed-off-by: Sargun Dhillon <sargun@sargun.me>
The overlay2 storage-driver requires multiple lower dir
support for overlayFs. Support for this feature was added
in kernel 4.x, but some distros (RHEL 7.4, CentOS 7.4) ship with
an older kernel with this feature backported.
This patch adds feature-detection for multiple lower dirs,
and will perform this feature-detection on pre-4.x kernels
with overlayFS support.
With this patch applied, daemons running on a kernel
with multiple lower dir support will now select "overlay2"
as storage-driver, instead of falling back to "overlay".
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Plugin config can have Mounts without a 'Source' field. In such cases,
performing a 'plugin set' on the mount source will panic the daemon. Its
the same case for device paths as well. This detects the case and
returns error.
Signed-off-by: Anusha Ragunathan <anusha.ragunathan@docker.com>
The commit '0a13f827a10d3bf61744d9b3f7165c5885a39c5d' introduces an
import test for CVE-2017-14992, it uses a 8GB image to make sure we
don't revert CVE-2017-14992, but unfortunately this test can't finish
in 5-min on AArch64, as a fact, in most cases we have to crate a very
big image to make the test effective on AArch64, but this will result
in a test panic, so now we skip it order to avoid termination of others
tests followed.
Signed-off-by: Dennis Chen <dennis.chen@arm.com>
This subtle bug keeps lurking in because error checking for `Mkdir()`
and `MkdirAll()` is slightly different wrt to `EEXIST`/`IsExist`:
- for `Mkdir()`, `IsExist` error should (usually) be ignored
(unless you want to make sure directory was not there before)
as it means "the destination directory was already there"
- for `MkdirAll()`, `IsExist` error should NEVER be ignored.
Mostly, this commit just removes ignoring the IsExist error, as it
should not be ignored.
Also, there are a couple of cases then IsExist is handled as
"directory already exist" which is wrong. As a result, some code
that never worked as intended is now removed.
NOTE that `idtools.MkdirAndChown()` behaves like `os.MkdirAll()`
rather than `os.Mkdir()` -- so its description is amended accordingly,
and its usage is handled as such (i.e. IsExist error is not ignored).
For more details, a quote from my runc commit 6f82d4b (July 2015):
TL;DR: check for IsExist(err) after a failed MkdirAll() is both
redundant and wrong -- so two reasons to remove it.
Quoting MkdirAll documentation:
> MkdirAll creates a directory named path, along with any necessary
> parents, and returns nil, or else returns an error. If path
> is already a directory, MkdirAll does nothing and returns nil.
This means two things:
1. If a directory to be created already exists, no error is
returned.
2. If the error returned is IsExist (EEXIST), it means there exists
a non-directory with the same name as MkdirAll need to use for
directory. Example: we want to MkdirAll("a/b"), but file "a"
(or "a/b") already exists, so MkdirAll fails.
The above is a theory, based on quoted documentation and my UNIX
knowledge.
3. In practice, though, current MkdirAll implementation [1] returns
ENOTDIR in most of cases described in #2, with the exception when
there is a race between MkdirAll and someone else creating the
last component of MkdirAll argument as a file. In this very case
MkdirAll() will indeed return EEXIST.
Because of #1, IsExist check after MkdirAll is not needed.
Because of #2 and #3, ignoring IsExist error is just plain wrong,
as directory we require is not created. It's cleaner to report
the error now.
Note this error is all over the tree, I guess due to copy-paste,
or trying to follow the same usage pattern as for Mkdir(),
or some not quite correct examples on the Internet.
[1] https://github.com/golang/go/blob/f9ed2f75/src/os/path.go
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Standard golang's `os.MkdirAll()` function returns "not a directory" error
in case a directory to be created already exists but is not a directory
(e.g. a file). Our own `idtools.MkdirAs*()` functions do not replicate
the behavior.
This is a bug since all `Mkdir()`-like functions are expected to ensure
the required directory exists and is indeed a directory, and return an
error otherwise.
As the code is using our in-house `system.Stat()` call returning a type
which is incompatible with that of golang's `os.Stat()`, I had to amend
the `system` package with `IsDir()`.
A test case is also provided.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
This removes and recreates the merged dir with each umount/mount
respectively.
This is done to make the impact of leaking mountpoints have less
user-visible impact.
It's fairly easy to accidentally leak mountpoints (even if moby doesn't,
other tools on linux like 'unshare' are quite able to incidentally do
so).
As of recently, overlayfs reacts to these mounts being leaked (see
One trick to force an unmount is to remove the mounted directory and
recreate it. Devicemapper now does this, overlay can follow suit.
Signed-off-by: Euan Kemp <euan.kemp@coreos.com>
Adds a main_test for the image integration test, so we can download
frozen images, and clean up after the image test is ran
Signed-off-by: Christopher Jones <tophj@linux.vnet.ibm.com>
- Fix OOM event updating healthchecks and persisting container state
without locks
- Fix healthchecks being updated without locks on container stop
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
When starting the daemon, the `/var/lib/docker` directory
is scanned for existing directories, so that the previously
selected graphdriver will automatically be used.
In some situations, empty directories are present (those
directories can be created during feature detection of
graph-drivers), in which case the daemon refuses to start.
This patch improves detection, and skips empty directories,
so that leftover directories don't cause the daemon to
fail.
Before this change:
$ mkdir /var/lib/docker /var/lib/docker/aufs /var/lib/docker/overlay2
$ dockerd
...
Error starting daemon: error initializing graphdriver: /var/lib/docker contains several valid graphdrivers: overlay2, aufs; Please cleanup or explicitly choose storage driver (-s <DRIVER>)
With this patch applied:
$ mkdir /var/lib/docker /var/lib/docker/aufs /var/lib/docker/overlay2
$ dockerd
...
INFO[2017-11-16T17:26:43.207739140Z] Docker daemon commit=ab90bc296 graphdriver(s)=overlay2 version=dev
INFO[2017-11-16T17:26:43.208033095Z] Daemon has completed initialization
And on restart (prior graphdriver is still picked up):
$ dockerd
...
INFO[2017-11-16T17:27:52.260361465Z] [graphdriver] using prior storage driver: overlay2
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This is a fix to the following issue:
$ docker run --tmpfs /dev/shm busybox sh
docker: Error response from daemon: linux mounts: Duplicate mount point '/dev/shm'.
In current code (daemon.createSpec()), tmpfs mount from --tmpfs is added
to list of mounts (`ms`), when the mount from IpcMounts() is added.
While IpcMounts() is checking for existing mounts first, it does that
by using container.HasMountFor() function which only checks container.Mounts
but not container.Tmpfs.
Ultimately, the solution is to get rid of container.Tmpfs (moving its
data to container.Mounts). Current workaround is to add checking
of container.Tmpfs into container.HasMountFor().
A unit test case is included.
Unfortunately we can't call daemon.createSpec() from a unit test,
as the code relies a lot on various daemon structures to be initialized
properly, and it is hard to achieve. Therefore, we minimally mimick
the code flow of daemon.createSpec() -- barely enough to reproduce
the issue.
https://github.com/moby/moby/issues/35455
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
The code in question looks up mounts two times: first by using
HasMountFor(), and then directly by looking in container.MountPoints.
There is no need to do it twice.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Commit dbf580be57 removed
this helper script because it's no longer used in CI.
However, the "make test" target in the Makefile still
called this helper, resulting it to fail.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This route expects `application/json`. Sending a content type header of `application/octet-stream` results in an error.
Signed-off-by: Asad Saeeduddin <masaeedu@gmail.com>
Commit 7a1618ced3 regresses running Docker
in user namespaces. The new check for whether quota are supported calls
NewControl() which in turn calls makeBackingFsDev() which tries to
mknod(). Skip quota tests when we detect that we are running in a user
namespace and return ErrQuotaNotSupported to the caller. This just
restores the status quo.
Signed-off-by: Christian Brauner <christian.brauner@ubuntu.com>
Add a way to specify a custom graphdriver priority list
during build. This can be done with something like
go build -ldflags "-X github.com/docker/docker/daemon/graphdriver.priority=overlay2,devicemapper"
As ldflags are already used by the engine build process, and it seems
that only one (last) `-ldflags` argument is taken into account by go,
an envoronment variable `DOCKER_LDFLAGS` is introduced in order to
be able to append some text to `-ldflags`. With this in place,
using the feature becomes
make DOCKER_LDFLAGS="-X github.com/docker/docker/daemon/graphdriver.priority=overlay2,devicemapper" dynbinary
The idea behind this is, the priority list might be different
for different distros, so vendors are now able to change it
without patching the source code.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Adds a mutex to protect the status, as well. When running the race
detector with the unit test, we can see that the Status field is written
without holding this lock. Adding a mutex to read and set status
addresses the issue.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Make it possible to disable overlay and overlay2 separately.
With this commit, we now have `exclude_graphdriver_overlay` and
`exclude_graphdriver_overlay2` build tags for the engine, which
is in line with any other graph driver.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Before this change, if the splunk endpoint is blocked it will cause a
deadlock on `Close()`.
This sets a reasonable timeout for the http request to send a log batch.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This test case is checking that the built-in default size for /dev/shm
(which is used for `--ipcmode` being `private` or `shareable`)
is not overriding the size of user-defined tmpfs mount for /dev/shm.
In other words, this is a regression test case for issue #35271,
https://github.com/moby/moby/issues/35271
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
From https://www.kernel.org/doc/Documentation/filesystems/overlayfs.txt:
> The lower filesystem can be any filesystem supported by Linux and does
> not need to be writable. The lower filesystem can even be another
> overlayfs. The upper filesystem will normally be writable and if it
> is it must support the creation of trusted.* extended attributes, and
> must provide valid d_type in readdir responses, so NFS is not suitable.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
While this code was likely called from a single thread before, we have
now seen panics, indicating that it could be called in parallel. This
change adds a mutex to protect opening and closing of the channel. There
may be another root cause associated with this panic, such as something
that led to the calling of this in parallel, as this code is old and we
had seen this condition until recently.
This fix is by no means a permanent fix. Typically, bugs like this
indicate misplaced channel ownership. In idiomatic uses, the channel
should have a particular "owner" that coordinates sending and closure.
In this case, the owner of the channel is unclear, so it gets opened
lazily. Synchronizing this access is a decent solution, but a refactor
may yield better results.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Commit 7120976d74 ("Implement none, private, and shareable ipc
modes") introduces a bug: if a user-specified mount for /dev/shm
is provided, its size is overriden by value of ShmSize.
A reproducer is simple:
docker run --rm
--mount type=tmpfs,dst=/dev/shm,tmpfs-size=100K \
alpine df /dev/shm
This commit is an attempt to fix the bug, as well as optimize things
a but and make the code easier to read.
https://github.com/moby/moby/issues/35271
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Support for duplicate labels (but different values) was
deprecated in commit e4c9079d09
(Docker 1.13), and scheduled for removal in 17.12
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Go has supported PIC builds for a while now, and given the security
benefits of using PIC binaries we should really enable them. There also
appears to be some indication that non-PIC builds have been interacting
oddly on ppc64le (the linker cannot load some shared libraries), and
using PIC builds appears to solve this problem.
Signed-off-by: Aleksa Sarai <asarai@suse.de>
Before this, if a volume exists in a driver but not in the local cache,
the store would just return a bare volume. This means that if a user
supplied options or labels, they will not get stored.
Instead only return early if we have the volume stored locally. Note
this could still have an issue with labels/opts passed in by the user
differing from what is stored, however this isn't really a new problem.
This fixes a problem where if there is a shared storage backend between
two docker nodes, a create on one node will have labels stored and a
create on the other node will not.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
The test was passing previously because the preamble was already buffered. After
the change to return Scanner.Err() the final read error on the buffer was no
longer being ignored.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
In order to avoid reverting our fix for mount leakage in devicemapper,
add a test which checks that devicemapper's Get() and Put() cycle can
survive having a command running in an rprivate mount propagation setup
in-between. While this is quite rudimentary, it should be sufficient.
We have to skip this test for pre-3.18 kernels.
Signed-off-by: Aleksa Sarai <asarai@suse.de>
To ensure that we don't revert CVE-2017-14992, add a test that is quite
similar to that upstream tar-split test (create an empty archive with
lots of junk and make sure the daemon doesn't crash).
Signed-off-by: Aleksa Sarai <asarai@suse.de>
This helper acts like /dev/zero (outputs \x00 indefinitely) in an
OS-independent fashion. This ensures we don't need to special-case
around Windows in tests that want to open /dev/zero.
Signed-off-by: Aleksa Sarai <asarai@suse.de>
Update to the latest version of tar-split, which includes a change to
fix a memory exhaustion issue where a malformed image could cause the
Docker daemon to crash.
* tar: asm: store padding in chunks to avoid memory exhaustion
Fixes: CVE-2017-14992
Signed-off-by: Aleksa Sarai <asarai@suse.de>
This patch adds the capability for the VFS graphdriver to use
XFS project quotas. It reuses the existing quota management
code that was created by overlay2 on XFS.
It doesn't rely on a filesystem whitelist, but instead
the quota-capability detection code.
Signed-off-by: Sargun Dhillon <sargun@sargun.me>
This allows much of the read logic to be shared for other things,
especially for the new log driver proposed in
https://github.com/moby/moby/issues/33475
The only logic for reads in the json logger is around decoding log
messages, which gets passed into the log file object.
This also helps with implementing compression as it allows us to
simplify locking strategies.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Make the `*RotateFileWriter` specifically about writing
`logger.Message`'s, which is what it's used for.
This allows for future changes where the log writer can cache details
about log entries such as (e.g.) the timestamps included in a particular
log file, which can be used to optimize reads.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This is writeable, and can be used to remove devices. Containers do
not need to know about scsi devices.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
This largely removes references to Docker where possible.
The HOWTO guides are still Docker docs and refer to Docker not
moby, so the next step is to rework these as Moby docs and put
them in `docs/` in this repo.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
The `docker info` code was shelling out to obtain the
version of containerd (using the `--version` flag).
Parsing the output of this version string is error-prone,
and not needed, as the containerd API can return the
version.
This patch adds a `Version()` method to the containerd Client
interface, and uses this to get the containerd version.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
/dev is mounted on a tmpfs inside of a container. Processes inside of containers
some times need to create devices nodes, or to setup a socket that listens on /dev/log
Allowing these containers to run with the --readonly flag makes sense. Making a tmpfs
readonly does not add any security to the container, since there is plenty of places
where the container can write tmpfs content.
I have no idea why /dev was excluded.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
Commit 3ba1dda191 bumped
the API version, but forgot to actually bump the version
in code.
This patch fixes the version to match those changes :-)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The Swagger file contained a version matrix to
find which API version is used by which version
of Docker.
Given that Docker is a downstream of the Moby project,
we should not be maintaining such a matrix in this
repository.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Commit d91c5f42eb added
support for "Isolation" mode for services, but didn't
get merged before API 1.34.
This patch moves the description in the API version
history to the correct API version (1.35), and does
a slight rewording of the functionality.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Remove Dockerfile.solaris reference in `hack/make/.detect-daemon-osarch`
as `Dockerfile.solaris` has been removed.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
If the user specifies a mountpath from the host, we should not be
attempting to chown files outside the daemon's metadata directory
(represented by `daemon.repository` at init time).
This forces users who want to use user namespaces to handle the
ownership needs of any external files mounted as network files
(/etc/resolv.conf, /etc/hosts, /etc/hostname) separately from the
daemon. In all other volume/bind mount situations we have taken this
same line--we don't chown host file content.
Docker-DCO-1.1-Signed-off-by: Phil Estes <estesp@linux.vnet.ibm.com>
The `.integration-test-helpers` script was sourced by `/etc/bash/bash.rc`.
However, the `$SCRIPTDIR` environment variable is set through `hack/make.sh`,
so will not be set when calling the `.integration-test-helpers` script directly.
Before this patch;
make BIND_DIR=. shell
...
bash: /make/.go-autogen: No such file or directory
After this patch, the warning is no longer printed
Also removed sourcing `.go-autogen` from test-integration and build-integration-test-binary,
as they already sourced `.integration-test-helpers` (which sources
`.go-autogen`).
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Instead of duplicating the same if condition per plugin manager directory,
use one if condition and a for-loop.
Signed-off-by: Boaz Shuster <ripcurld.github@gmail.com>
These scripts have not been used for a while now, and should not be used again
because they are for releasing docker, not moby
Signed-off-by: Daniel Nephin <dnephin@docker.com>
This vendoring fixes two issues.
1. When a user specifies an SELinux MCS Label (level) to override moby picking
an unigue MCS label, the code currently picks a label then overrides with the
user selected. This works fine, except the unique MCS Label is leaked and will
not be used until the daemon is restarted.
2. The override label, is not reserved. This could potentially cause an issue
where the daemon could pick the same MCS Label again for a different container.
(~ 1/500,000 Chance).
The updated selinux go bindings, now release the overriden unigue label, and reserve
the one specified by the user.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Update golang.org/x/sys to 95c6576299259db960f6c5b9b69ea52422860fce in
order to get the unix.Utsname with byte array instead of int8/uint8
members.
This allows to use simple byte slice to string conversions instead of
using charsToString or its open-coded version.
Also see golang/go#20753 for details.
Signed-off-by: Tobias Klauser <tklauser@distanz.ch>
Log drivers may have an internal buffer size that can be accommodated
by the copier as it is more effective to buffer and send fewer though
larger messages that the log driver can consume.
This eliminates the need for Partial handling for drivers that do not
support the concept (ie: awslogs, which can only have events up to
service limits).
Signed-off-by: Jacob Vallejo <jakeev@amazon.com>
This adds a mechanism (read-only) to check for project quota support
in a standard way. This mechanism is leveraged by the tests, which
test for the following:
1. Can we get a quota controller?
2. Can we set the quota for a particular directory?
3. Is the quota being over-enforced?
4. Is the quota being under-enforced?
5. Can we retrieve the quota?
Signed-off-by: Sargun Dhillon <sargun@sargun.me>
go1.8.5 (released 2017/10/25) includes fixes to the compiler, linker, runtime,
documentation, go command, and the crypto/x509 and net/smtp packages. It
includes a fix to a bug introduced in Go 1.8.4 that broke go get of non-Git
repositories under certain conditions. See the Go 1.8.5 milestone on our issue
tracker for details:
https://github.com/golang/go/issues?q=milestone%3AGo1.8.5
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Currently, if a container removal has failed for some reason,
any client waiting for removal (e.g. `docker run --rm`) is
stuck, waiting for removal to succeed while it has failed already.
For more details and the reproducer, please check
https://github.com/moby/moby/issues/34945
This commit addresses that by allowing `ContainerWait()` with
`container.WaitCondition == "removed"` argument to return an
error in case of removal failure. The `ContainerWaitOKBody`
stucture returned to a client is amended with a pointer to `struct Error`,
containing an error message string, and the `Client.ContainerWait()`
is modified to return the error, if any, to the client.
Note that this feature is only available for API version >= 1.34.
In order for the old clients to be unstuck, we just close the connection
without writing anything -- this causes client's error.
Now, docker-cli would need a separate commit to bump the API to 1.34
and to show an error returned, if any.
[v2: recreate the waitRemove channel after closing]
[v3: document; keep legacy behavior for older clients]
[v4: convert Error from string to pointer to a struct]
[v5: don't emulate old behavior, send empty response in error case]
[v6: rename legacy* vars to include version suffix]
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
To ensure that namesgenerator binary outputs random name
by initializing Seed.
Signed-off-by: Mizuki Urushida <z11111001011@gmail.com>
not use init function.
Signed-off-by: Mizuki Urushida <z11111001011@gmail.com>
This changeset allows Docker's VFS, and Overlay to take advantage of
Linux's zerocopy APIs.
The copy function first tries to use the ficlone ioctl. Reason being:
- they do not allow partial success (aka short writes)
- clones are expected to be a fast metadata operation
See: http://oss.sgi.com/archives/xfs/2015-12/msg00356.html
If the clone fails, we fall back to copy_file_range, which internally
may fall back to splice, which has an upper limit on the size
of copy it can perform. Given that, we have to loop until the copy
is done.
For a given dirCopy operation, if the clone fails, we will not try
it again during any other file copy. Same is true with copy_file_range.
If all else fails, we fall back to traditional copy.
Signed-off-by: Sargun Dhillon <sargun@sargun.me>
Checks + pulls down the frozen images in the integration-cli
tests too. Fixes an issue where they wouldn't be pulled if running
against just integration-cli tests, e.g. through TESTFLAGS
Signed-off-by: Christopher Jones <tophj@linux.vnet.ibm.com>
The shutdown timeout for containers in insufficient on Windows. If the daemon is shutting down, and a container takes longer than expected to shut down, this can cause the container to remain in a bad state after restart, and never be able to start again. Increasing the timeout makes this less likely to occur.
Signed-off-by: Darren Stahl <darst@microsoft.com>
Probably a good idea to treat `*.Dockerfile` as dockerfile format as well. In general, it's better to use the `Dockerfile` part as an extension rather than a basename.
Signed-off-by: Andrew Pennebaker <andrew.pennebaker@gmail.com>
Before this commit if "--config-file" wasn't set the daemon would use
the default configuration file which is "/etc/docker/daemon.json".
When attempting to reload the daemon if that file didn't exist
and error message would display.
This behaviour is changed in a way that if the default configuration
file does not exist and no other configuration file is set
the daemon uses an empty configuration which later will be updated
and reloaded using the "reload" function given as an argument in Reload.
However, if the "--config-file" is set and the file is removed
or renamed an error message will be displayed and no reload will be done.
Signed-off-by: Boaz Shuster <ripcurld.github@gmail.com>
In some circumstances we were not properly releasing plugin references,
leading to failures in removing a plugin with no way to recover other
than restarting the daemon.
1. If volume create fails (in the driver)
2. If a driver validation fails (should be rare)
3. If trying to get a plugin that does not match the passed in capability
Ideally the test for 1 and 2 would just be a unit test, however the
plugin interfaces are too complicated as `plugingetter` relies on
github.com/pkg/plugin/Client (a concrete type), which will require
spinning up services from within the unit test... it just wouldn't be a
unit test at this point.
I attempted to refactor this a bit, but since both libnetwork and
swarmkit are reliant on `plugingetter` as well, this would not work.
This really requires a re-write of the lower-level plugin management to
decouple these pieces.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
In some cases (e.g. NFS), a chown may technically be a no-op but still
return `EPERM`, so only call `chown` when neccessary.
This is particularly problematic for docker users bind-mounting an NFS
share into a container.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Update golang.org/x/sys to 8dbc5d05d6edcc104950cc299a1ce6641235bc86 in
order to get the Major, Minor and Mkdev functions for every unix-like
OS. Use them instead of the locally defined versions which currently use
the Linux specific device major/minor encoding.
This means that the device number should now be properly encoded on e.g.
Darwin, FreeBSD or Solaris.
Also, the SIGUNUSED constant was removed from golang.org/x/sys/unix in
https://go-review.googlesource.com/61771 as it is also removed from the
respective glibc headers.
Remove it from signal.SignalMap as well after the golang.org/x/sys
re-vendoring.
Signed-off-by: Tobias Klauser <tklauser@distanz.ch>
Describe more how host port allocation is done when
container is stopped/started in "PublishAllPorts".
Signed-off-by: Boaz Shuster <ripcurld.github@gmail.com>
With `rprivate` there exists a race where a reference to a mount has
propagated to the new namespace, when `rprivate` is set the parent
namespace is not able to remove the mount due to that reference.
With `rslave` unmounts will propagate correctly into the namespace and
prevent the sort of transient errors that are possible with `rprivate`.
This is a similar fix to 117c92745b
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
When runc is bind-mounting a particular path "with options", it has to
do so by first creating a bind-mount and the modifying the options of
said bind-mount via remount. However, in a user namespace, there are
restrictions on which flags you can change with a remount (due to
CL_UNPRIVILEGED being set in this instance). Docker historically has
ignored this, and as a result, internal Docker mounts (such as secrets)
haven't worked with --userns-remap. Fix this by preserving
CL_UNPRIVILEGED mount flags when Docker is spawning containers with user
namespaces enabled.
Ref: https://github.com/opencontainers/runc/pull/1603
Signed-off-by: Aleksa Sarai <asarai@suse.de>
If a 400 error is returned due to an API version mismatch, no
version and server-identification headers were returned by the API.
All information in these headers is "static", so there is no
reason to omit the information in case of an error being
returned.
This patch updates the version middleware to always
return the headers.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
If a network is created with a name that matches another
network's ID, the network with that name was masking the
other network's ID.
As a result, it was not possible to remove the network
with a given ID.
This patch changes the order in which networks are
matched to be what we use for other cases;
1. Match on full ID
2. Match on full Name
3. Match on Partial ID
Before this patch:
$ docker network create foo
336717eac9eaa3da6557042a04efc803f7e8862ce6cf96f6b9565265ba5c618b
$ docker network create 336717eac9eaa3da6557042a04efc803f7e8862ce6cf96f6b9565265ba5c618b
4a698333f1197f20224583abce14876d7f25fdfe416a8545927006c315915a2a
$ docker network ls
NETWORK ID NAME DRIVER SCOPE
4a698333f119 336717eac9eaa3da6557042a04efc803f7e8862ce6cf96f6b9565265ba5c618b bridge local
d1e40d43a2c0 bridge bridge local
336717eac9ea foo bridge local
13cf280a1bbf host host local
d9e4c03728a0 none null local
$ docker network rm 336717eac9eaa3da6557042a04efc803f7e8862ce6cf96f6b9565265ba5c618b
4a698333f1197f20224583abce14876d7f25fdfe416a8545927006c315915a2a
$ docker network ls
NETWORK ID NAME DRIVER SCOPE
d1e40d43a2c0 bridge bridge local
336717eac9ea foo bridge local
13cf280a1bbf host host local
d9e4c03728a0 none null local
After this patch:
$ docker network create foo
2d1791a7def4e2a1ef0f6b83c6add333df0bb4ced2f196c584cb64e6bd94b835
$ docker network create 2d1791a7def4e2a1ef0f6b83c6add333df0bb4ced2f196c584cb64e6bd94b835
6cbc749a529cd2d9d3b10566c84e56c4203dd88b67417437b5fc7a6e955dd48f
$ docker network ls
NETWORK ID NAME DRIVER SCOPE
6cbc749a529c 2d1791a7def4e2a1ef0f6b83c6add333df0bb4ced2f196c584cb64e6bd94b835 bridge local
166c943dbeb5 bridge bridge local
2d1791a7def4 foo bridge local
6c45b8aa6d8e host host local
b11c96b51ea7 none null local
$ docker network rm 2d1791a7def4e2a1ef0f6b83c6add333df0bb4ced2f196c584cb64e6bd94b835
2d1791a7def4e2a1ef0f6b83c6add333df0bb4ced2f196c584cb64e6bd94b835
$ docker network ls
NETWORK ID NAME DRIVER SCOPE
6cbc749a529c 2d1791a7def4e2a1ef0f6b83c6add333df0bb4ced2f196c584cb64e6bd94b835 bridge local
166c943dbeb5 bridge bridge local
6c45b8aa6d8e host host local
b11c96b51ea7 none null local
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
83c2152de5 sets the kernel param for
fs.may_detach_mounts, but this is not neccessary for the daemon to
operate. Instead of erroring out (and thus aborting startup) just log
the error.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Fixes previously flaky TestAPISwarmRaftQuorum.
In the test we are shutting down 2 of 3 nodes, all of which are managers.
When this happens the remaining node prepares to step down from being
a leader because there is no longer an active quorum. This test was
written before that change was implemented, and assumes that the
remaining node can still handle requests to create a service (it can't)
There was further flakiness depending on whether or not the node has
stepped down or was in the process of doing so. Fix this by waiting
for the node to completely step down and checking to see if it errors
out like it should.
Signed-off-by: Christopher Jones <tophj@linux.vnet.ibm.com>
As part of the Moby transition (see #35115), update the Roadmap to
reflect the new priorities. Also just update it as it was written
a while back, and we made some progress in areas such as `containerd`.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
Commit e98e4a7111 implemented API version
negotiation using the `/_ping` endpoint. In that change, URL validation for the
maximum supported API version was removed from the API server (validation for
the _minimum_ version was kept in place).
With this feature, clients that support version negotiation would negotiate the
maximum version supported by the daemon, and downgrade to an older API version
if the client's default API version is not supported.
However, clients that do _not_ support version negotiation can call API versions
that are higher than the maximum supported version. Due to the missing version
check, this is silently ignored, and the daemon's default API version is used.
This is a problem, because the actual API version in use is non-deterministic;
for example, calling `/v9999.9999/version` on a daemon that runs API v1.34 will
use API v1.34, but calling the same URL on an older daemon may use API version
v1.24.
This patch reverts the removal of the API check for maximum supported versions.
The documentation has been updated accordingly
Before this patch is applied, the daemon returns a 200 (success):
$ curl -v --unix-socket /var/run/docker.sock http://localhost/v9999.9999/version
* Trying /var/run/docker.sock...
* Connected to localhost (/Users/sebastiaan/Library/Containers/com.dock) port 80 (#0)
> GET /v9999.9999/version HTTP/1.1
> Host: localhost
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Api-Version: 1.32
< Content-Length: 240
< Content-Type: application/json
< Date: Tue, 10 Oct 2017 09:11:29 GMT
< Docker-Experimental: true
< Ostype: linux
< Server: Docker/17.09.0-ce (linux)
<
{"Version":"17.09.0-ce","ApiVersion":"1.32","MinAPIVersion":"1.12","GitCommit":"afdb6d4","GoVersion":"go1.8.3","Os":"linux","Arch":"amd64","KernelVersion":"4.9.49-moby","Experimental":true,"BuildTime":"2017-09-26T22:45:38.000000000+00:00"}
* Connection #0 to host localhost left intact
After this patch is applied, a 400 (Bad Request) is returned:
$ curl -v --unix-socket /var/run/docker.sock http://localhost/v9999.9999/version
* Trying /var/run/docker.sock...
* Connected to localhost (/var/run/docker.sock) port 80 (#0)
> GET /v9999.9999/info HTTP/1.1
> Host: localhost
> User-Agent: curl/7.52.1
> Accept: */*
>
< HTTP/1.1 400 Bad Request
< Content-Type: application/json
< Date: Tue, 10 Oct 2017 08:08:34 GMT
< Content-Length: 89
<
{"message":"client version 9999.9999 is too new. Maximim supported API version is 1.34"}
* Curl_http_done: called premature == 0
* Connection #0 to host localhost left intact
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
- Call the function that create an event entry while volumes are
pruning.
- Pass volume.Volume type on volumeRm instead of a name. Volume lookup is done
on the exported VolumeRm function.
- Skip volume deletion when force option used and it does not exists.
Signed-off-by: Nicolas Sterchele <sterchele.nicolas@gmail.com>
As described in #35115 Completing the Moby Transition, we would like to complete the transition
stage that we began a while back. This tries to clarify what and who this project is for.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
ContainerExecAttach used `types.ExecConfig` instead of `types.ExecStartCheck`,
which is the type that's expected by the `/exec/execid/start` API endpoint.
Investigating when this inconsistency was introduced, I found that the client has
sent the additional properties since its first imlpementation in
c786a8ee5e.
The `postContainerExecStart()` at that time used the "jobs" package, which
only took the information from the body that was needed (`Detach` and `Tty`).
Commit 24425021d2 refactored the Exec commands
to remove the "jobs", and introduced the `ExecStartCheck` type, but failed to
update the `cli.hijack()` call with the new type.
The change in this patch should not affect compatibility with older clients,
as the additional information from the `ExecConfig` type is not used (the
API server already decodes to the `ExecStartCheck` type).
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: John Howard <jhoward@microsoft.com>
This PR has the API changes described in https://github.com/moby/moby/issues/34617.
Specifically, it adds an HTTP header "X-Requested-Platform" which is a JSON-encoded
OCI Image-spec `Platform` structure.
In addition, it renames (almost all) uses of a string variable platform (and associated)
methods/functions to os. This makes it much clearer to disambiguate with the swarm
"platform" which is really os/arch. This is a stepping stone to getting the daemon towards
fully multi-platform/arch-aware, and makes it clear when "operating system" is being
referred to rather than "platform" which is misleadingly used - sometimes in the swarm
meaning, but more often as just the operating system.
These files don't exist under proc so this rule does nothing.
They are protected against by docker's default cgroup devices since they're
both character devices and not explicitly allowed.
Signed-off-by: Tycho Andersen <tycho@docker.com>
Update logic to choose manifest from manifest list to check
for os version on Windows. Separate the logic for windows
and unix to keep unix logic the same.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Commit ebcb7d6b40 removed string checking
for error messages, in favor of typed errors.
In this change, the status code for conflicting container names
changed from 409 to 400 (validationError).
This patch add a `nameConflictError`, changing the status code to
409 as it was in older versions.
With this change applied, the correct 409 status is returned:
```bash
$ docker create --name c1 busybox
```
```bash
$ curl --unix-socket /var/run/docker.sock -v -XPOST -H"Content-Type: application/json" -d'{"Image":"busybox"}' http://localhost/containers/create?name=c1
Note: Unnecessary use of -X or --request, POST is already inferred.
* Trying /var/run/docker.sock...
* Connected to localhost (/var/run/docker.sock) port 80 (#0)
> POST /containers/create?name=c1 HTTP/1.1
> Host: localhost
> User-Agent: curl/7.52.1
> Accept: */*
> Content-Type: application/json
> Content-Length: 19
>
* upload completely sent off: 19 out of 19 bytes
< HTTP/1.1 409 Conflict
< Api-Version: 1.33
< Content-Type: application/json
< Docker-Experimental: false
< Ostype: linux
< Server: Docker/17.06.0-dev (linux)
< Date: Thu, 28 Sep 2017 15:07:23 GMT
< Content-Length: 229
<
{"message":"Conflict. The container name \"/c1\" is already in use by container \"ed2efdc806c1883954e677eb9ab8cbc7e286c9c5934ef6724fd5d93c56744923\". You have to remove (or rename) that container to be able to reuse that name."}
* Curl_http_done: called premature == 0
* Connection #0 to host localhost left intact
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
s390x node-1 has kernel 4.6.0, kernel.CompareKernelVersion()
returns 0 if the kernels are equal, so include that.
Full logic for CompareKernelVersion() is
a > b ret 1,
a == b ret 0,
a < b ret -1
Signed-off-by: Christopher Jones <tophj@linux.vnet.ibm.com>
The missing console mode constants were added to go-ansiterm in
Azure/go-ansiterm#23. Use these constants instead of defining them
locally.
Signed-off-by: Tobias Klauser <tklauser@distanz.ch>
Re-vendor go-ansiterm to d6e3b3328b783f23731bc4d058875b0371ff8109 in
order to get the newly added console mode constants.
Signed-off-by: Tobias Klauser <tklauser@distanz.ch>
Changed push message from "The push refers to a repository" to "The push refers to the repository" to make it clearer that this is just an information about the local source and not a distinction between pushing single images or whole repositories.
Signed-off-by: Pascal <pascalgn@users.noreply.github.com>
When use overlay2 as the graphdriver and the kernel enable
`CONFIG_OVERLAY_FS_REDIRECT_DIR=y`, rename a dir in lower layer
will has a xattr to redirct its dir to source dir. This make the
image layer unportable. This patch fallback to use naive diff driver
when kernel enable CONFIG_OVERLAY_FS_REDIRECT_DIR
Signed-off-by: Lei Jitang <leijitang@huawei.com>
libdm currently has a fairly substantial DoS bug that makes certain
operations fail on a libdm device if the device has active references
through mountpoints. This is a significant problem with the advent of
mount namespaces and MS_PRIVATE, and can cause certain --volume mounts
to cause libdm to no longer be able to remove containers:
% docker run -d --name testA busybox top
% docker run -d --name testB -v /var/lib/docker:/docker busybox top
% docker rm -f testA
[fails on libdm with dm_task_run errors.]
This also solves the problem of unprivileged users being able to DoS
docker by using unprivileged mount namespaces to preseve mounts that
Docker has dropped.
Signed-off-by: Aleksa Sarai <asarai@suse.de>
2017-09-06 20:11:01 +10:00
4247 changed files with 366753 additions and 110061 deletions
@@ -5,6 +5,28 @@ information on the list of deprecated flags and APIs please have a look at
https://docs.docker.com/engine/deprecated/ where target removal dates can also
be found.
## 17.03.2-ce (2017-05-29)
### Networking
- Fix a concurrency issue preventing network creation [#33273](https://github.com/moby/moby/pull/33273)
### Runtime
- Relabel secrets path to avoid a Permission Denied on selinux enabled systems [#33236](https://github.com/moby/moby/pull/33236) (ref [#32529](https://github.com/moby/moby/pull/32529)
- Fix cases where local volume were not properly relabeled if needed [#33236](https://github.com/moby/moby/pull/33236) (ref [#29428](https://github.com/moby/moby/pull/29428))
- Fix an issue while upgrading if a plugin rootfs was still mounted [#33236](https://github.com/moby/moby/pull/33236) (ref [#32525](https://github.com/moby/moby/pull/32525))
- Fix an issue where volume wouldn't default to the `rprivate` propagation mode [#33236](https://github.com/moby/moby/pull/33236) (ref [#32851](https://github.com/moby/moby/pull/32851))
- Fix a panic that could occur when a volume driver could not be retrieved [#33236](https://github.com/moby/moby/pull/33236) (ref [#32347](https://github.com/moby/moby/pull/32347))
+ Add a warning in `docker info` when the `overlay` or `overlay2` graphdriver is used on a filesystem without `d_type` support [#33236](https://github.com/moby/moby/pull/33236) (ref [#31290](https://github.com/moby/moby/pull/31290))
- Fix an issue with backporting mount spec to older volumes [#33207](https://github.com/moby/moby/pull/33207)
- Fix issue where a failed unmount can lead to data loss on local volume remove [#33120](https://github.com/moby/moby/pull/33120)
### Swarm Mode
- Fix a case where tasks could get killed unexpectedly [#33118](https://github.com/moby/moby/pull/33118)
- Fix an issue preventing to deploy services if the registry cannot be reached despite the needed images being locally present [#33117](https://github.com/moby/moby/pull/33117)
# set the graph driver as the current graphdriver if not set
DOCKER_GRAPHDRIVER:=$(if$(DOCKER_GRAPHDRIVER),$(DOCKER_GRAPHDRIVER),$(shell docker info 2>&1| grep "Storage Driver"| sed 's/.*: //'))
@@ -16,6 +16,13 @@ export DOCKER_GITCOMMIT
# env vars passed through directly to Docker's build scripts
# to allow things like `make KEEPBUNDLE=1 binary` easily
# `project/PACKAGERS.md` have some limited documentation of some of these
#
# DOCKER_LDFLAGS can be used to pass additional parameters to -ldflags
# option of "go build". For example, a built-in graphdriver priority list
# can be changed during build time like this:
#
# make DOCKER_LDFLAGS="-X github.com/docker/docker/daemon/graphdriver.priority=overlay2,devicemapper" dynbinary
#
DOCKER_ENVS:=\
-e DOCKER_CROSSPLATFORMS \
-e BUILD_APT_MIRROR \
@@ -24,6 +31,7 @@ DOCKER_ENVS := \
-e DOCKER_BUILD_ARGS \
-e DOCKER_BUILD_GOGC \
-e DOCKER_BUILD_PKGS \
-e DOCKER_BUILDKIT \
-e DOCKER_BASH_COMPLETION_PATH \
-e DOCKER_CLI_PATH \
-e DOCKER_DEBUG \
@@ -31,10 +39,12 @@ DOCKER_ENVS := \
-e DOCKER_GITCOMMIT \
-e DOCKER_GRAPHDRIVER \
-e DOCKER_INCREMENTAL_BINARY \
-e DOCKER_LDFLAGS \
-e DOCKER_PORT \
-e DOCKER_REMAP_ROOT \
-e DOCKER_STORAGE_OPTS \
-e DOCKER_USERLANDPROXY \
-e DOCKERD_ARGS \
-e TEST_INTEGRATION_DIR \
-e TESTDIRS \
-e TESTFLAGS \
@@ -45,7 +55,10 @@ DOCKER_ENVS := \
-e http_proxy \
-e https_proxy \
-e no_proxy \
-e VERSION
-e VERSION\
-e PLATFORM \
-e DEFAULT_PRODUCT_LICENSE \
-e PRODUCT
# note: we _cannot_ add "-e DOCKER_BUILDTAGS" here because even if it's unset in the shell, that would shadow the "ENV DOCKER_BUILDTAGS" set in our Dockerfile, which is very important for our official builds
# to allow `make BIND_DIR=. shell` or `make BIND_DIR= test`
# This allows the test suite to be able to run without worrying about the underlying fs used by the container running the daemon (e.g. aufs-on-aufs), so long as the host running the container is running a supported fs.
# The volume will be cleaned up when the container is removed due to `--rm`.
# Note that `BIND_DIR` will already be set to `bundles` if `DOCKER_HOST` is not set (see above BIND_DIR line), in such case this will do nothing since `DOCKER_MOUNT` will already be set.
### Docker users, see [Moby and Docker](https://mobyproject.org/#moby-and-docker) to clarify the relationship between the projects
### Docker maintainers and contributors, see [Transitioning to Moby](#transitioning-to-moby) for more details
The Moby Project
================
Moby is an open-source project created by Docker to advance the software containerization movement.
It provides a “Lego set” of dozens of components, the framework for assembling them into custom container-based systems, and a place for all container enthusiasts to experiment and exchange ideas.
Moby is an open-source project created by Docker to enable and accelerate software containerization.
# Moby
## Overview
At the core of Moby is a framework to assemble specialized container systems.
It provides:
- A library of containerized components for all vital aspects of a container system: OS, container runtime, orchestration, infrastructure management, networking, storage, security, build, image distribution, etc.
- Tools to assemble the components into runnable artifacts for a variety of platforms and architectures: bare metal (both x86 and Arm); executables for Linux, Mac and Windows; VM images for popular cloud and virtualization providers.
- A set of reference assemblies which can be used as-is, modified, or used as inspiration to create your own.
All Moby components are containers, so creating new components is as easy as building a new OCI-compatible container.
It provides a "Lego set" of toolkit components, the framework for assembling them into custom container-based systems, and a place for all container enthusiasts and professionals to experiment and exchange ideas.
Components include container build tools, a container registry, orchestration tools, a runtime and more, and these can be used as building blocks in conjunction with other tools and projects.
## Principles
Moby is an open project guided by strong principles, but modular, flexible and without too strong an opinion on user experience, so it is open to the community to help set its direction.
The guiding principles are:
Moby is an open project guided by strong principles, aiming to be modular, flexible and without too strong an opinion on user experience.
It is open to the community to help set its direction.
- Modular: the project includes lots of components that have well-defined functions and APIs that work together.
- Batteries included but swappable: Moby includes enough components to build fully featured container system, but its modular architecture ensures that most of the components can be swapped by different implementations.
- Usable security: Moby will provide secure defaults without compromising usability.
-Container centric: Moby is built with containers, for running containers.
With Moby, you should be able to describe all the components of your distributed application, from the high-level configuration files down to the kernel you would like to use and build and deploy it easily.
Moby uses [containerd](https://github.com/containerd/containerd) as the default container runtime.
- Usable security: Moby provides secure defaults without compromising usability.
-Developer focused: The APIs are intended to be functional and useful to build powerful tools.
They are not necessarily intended as end user tools but as components aimed at developers.
Documentation and UX is aimed at developers not end users.
## Audience
Moby is recommended for anyone who wants to assemble a container-based system. This includes:
The Moby Project is intended for engineers, integrators and enthusiasts looking to modify, hack, fix, experiment, invent and build systems based on containers.
It is not for people looking for a commercially supported system, but for people who want to work and learn with open source code.
- Hackers who want to customize or patch their Docker build
- System engineers or integrators building a container system
- Infrastructure providers looking to adapt existing container systems to their environment
- Container enthusiasts who want to experiment with the latest container tech
- Open-source developers looking to test their project in a variety of different systems
- Anyone curious about Docker internals and how it’s built
## Relationship with Docker
Moby is NOT recommended for:
The components and tools in the Moby Project are initially the open source components that Docker and the community have built for the Docker Project.
New projects can be added if they fit with the community goals. Docker is committed to using Moby as the upstream for the Docker Product.
However, other projects are also encouraged to use Moby as an upstream, and to reuse the components in diverse ways, and all these uses will be treated in the same way. External maintainers and contributors are welcomed.
- Application developers looking for an easy way to run their applications in containers. We recommend Docker CE instead.
- Enterprise IT and development teams looking for a ready-to-use, commercially supported container platform. We recommend Docker EE instead.
- Anyone curious about containers and looking for an easy way to learn. We recommend the [docker.com](https://www.docker.com/) website instead.
# Transitioning to Moby
Docker is transitioning all of its open source collaborations to the Moby project going forward.
During the transition, all open source activity should continue as usual.
We are proposing the following list of changes:
- splitting up the engine into more open components
- removing the docker UI, SDK etc to keep them in the Docker org
- clarifying that the project is not limited to the engine, but to the assembly of all the individual components of the Docker platform
- open-source new tools & components which we currently use to assemble the Docker product, but could benefit the community
- defining an open, community-centric governance inspired by the Fedora project (a very successful example of balancing the needs of the community with the constraints of the primary corporate sponsor)
The Moby project is not intended as a location for support or feature requests for Docker products, but as aplace for contributors to work on open source code, fix bugs, and make the code more useful.
The releases are supported by the maintainers, community and users, on a best efforts basis only, and are not intended for customers who want enterprise or commercial support; Docker EE is the appropriate product for these use cases.
-----
@@ -82,7 +50,6 @@ violate applicable laws.
For more information, please see https://www.bis.doc.gov
Licensing
=========
Moby is licensed under the Apache License, Version 2.0. See
@@ -10,7 +10,7 @@ It consists of various components in this repository:
-`client/` The Go client used by the command-line client. It can also be used by third-party Go programs.
-`daemon/` The daemon, which serves the API.
##Swagger definition
##Swagger definition
The API is defined by the [Swagger](http://swagger.io/specification/) definition in `api/swagger.yaml`. This definition can be used to:
@@ -20,7 +20,7 @@ The API is defined by the [Swagger](http://swagger.io/specification/) definition
## Updating the API documentation
The API documentation is generated entirely from `api/swagger.yaml`. If you make updates to the API, you'll need to edit this file to represent the change in the documentation.
The API documentation is generated entirely from `api/swagger.yaml`. If you make updates to the API, edit this file to represent the change in the documentation.
The file is split into two main sections:
@@ -29,9 +29,9 @@ The file is split into two main sections:
To make an edit, first look for the endpoint you want to edit under `paths`, then make the required edits. Endpoints may reference reusable objects with `$ref`, which can be found in the `definitions` section.
There is hopefully enough example material in the file for you to copy a similar pattern from elsewhere in the file (e.g. adding new fields or endpoints), but for the full reference, see the [Swagger specification](https://github.com/docker/docker/issues/27919)
There is hopefully enough example material in the file for you to copy a similar pattern from elsewhere in the file (e.g. adding new fields or endpoints), but for the full reference, see the [Swagger specification](https://github.com/docker/docker/issues/27919).
`swagger.yaml` is validated by `hack/validate/swagger` to ensure it is a valid Swagger definition. This is useful for when you are making edits to ensure you are doing the right thing.
`swagger.yaml` is validated by `hack/validate/swagger` to ensure it is a valid Swagger definition. This is useful when making edits to ensure you are doing the right thing.
returnfmt.Sprintf("client version %s is too old. Minimum supported API version is %s, please upgrade your client to a newer version",e.version,e.minVersion)
ife.minVersion!=""{
returnfmt.Sprintf("client version %s is too old. Minimum supported API version is %s, please upgrade your client to a newer version",e.version,e.minVersion)
}
returnfmt.Sprintf("client version %s is too new. Maximum supported API version is %s",e.version,e.maxVersion)
// No need to check the ID collision here as we are still in
// local scope and the network ID is unique in this scope.
listByFullName[network.ID]=network
}
ifstrings.HasPrefix(network.ID,term){
// No need to check the ID collision here as we are still in
// local scope and the network ID is unique in this scope.
listByPartialID[network.ID]=network
}
}
nr,_:=n.cluster.GetNetworks(filter)
for_,network:=rangenr{
ifnetwork.ID==term{
returnnetwork,nil
}
ifnetwork.Name==term{
// Check the ID collision as we are in swarm scope here, and
// the map (of the listByFullName) may have already had a
// network with the same ID (from local scope previously)
if_,ok:=listByFullName[network.ID];!ok{
listByFullName[network.ID]=network
}
}
ifstrings.HasPrefix(network.ID,term){
// Check the ID collision as we are in swarm scope here, and
// the map (of the listByPartialID) may have already had a
// network with the same ID (from local scope previously)
if_,ok:=listByPartialID[network.ID];!ok{
listByPartialID[network.ID]=network
}
}
}
// Find based on full name, returns true only if no duplicates
iflen(listByFullName)==1{
for_,v:=rangelistByFullName{
returnv,nil
}
}
iflen(listByFullName)>1{
returntypes.NetworkResource{},errdefs.InvalidParameter(errors.Errorf("network %s is ambiguous (%d matches found based on name)",term,len(listByFullName)))
}
// Find based on partial ID, returns true only if no duplicates
iflen(listByPartialID)==1{
for_,v:=rangelistByPartialID{
returnv,nil
}
}
iflen(listByPartialID)>1{
returntypes.NetworkResource{},errdefs.InvalidParameter(errors.Errorf("network %s is ambiguous (%d matches found based on ID prefix)",term,len(listByPartialID)))
The API is usually changed in each release of Docker, so API calls are versioned to ensure that clients don't break.
The API is usually changed in each release, so API calls are versioned to
ensure that clients don't break. To lock to a specific version of the API,
you prefix the URL with its version, for example, call `/v1.30/info` to use
the v1.30 version of the `/info` endpoint. If the API version specified in
the URL is not supported by the daemon, a HTTP `400 Bad Request` error message
is returned.
For Docker Engine 17.09, the API version is 1.32. To lock to this version, you prefix the URL with `/v1.32`. For example, calling `/info` is the same as calling `/v1.32/info`.
If you omit the version-prefix, the current version of the API (v1.39) is used.
For example, calling `/info` is the same as calling `/v1.39/info`. Using the
API without a version-prefix is deprecated and will be removed in a future release.
Engine releases in the near future should support this version of the API, so your client will continue to work even if it is talking to a newer Engine.
Engine releases in the near future should support this version of the API,
so your client will continue to work even if it is talking to a newer Engine.
In previous versions of Docker, it was possible to access the API without providing a version. This behaviour is now deprecated will be removed in a future version of Docker.
The API uses an open schema model, which means server may add extra properties
to responses. Likewise, the server will ignore any extra query parameters and
request body properties. When you write clients, you need to ignore additional
properties in responses to ensure they do not break when talking to newer
daemons.
The API uses an open schema model, which means server may add extra properties to responses. Likewise, the server will ignore any extra query parameters and request body properties. When you write clients, you need to ignore additional properties in responses to ensure they do not break when talking to newer Docker daemons.
This documentation is for version 1.33 of the API. Use this table to find documentation for previous versions of the API:
description:"Host IP address that the container's port is mapped to"
PrivatePort:
type:"integer"
format:"uint16"
@@ -176,7 +168,7 @@ definitions:
Type:
type:"string"
x-nullable:false
enum:["tcp","udp"]
enum:["tcp","udp","sctp"]
example:
PrivatePort:8080
PublicPort:80
@@ -263,6 +255,7 @@ definitions:
properties:
Propagation:
description:"A propagation mode with the value `[r]private`, `[r]shared`, or `[r]slave`."
type:"string"
enum:
- "private"
- "rprivate"
@@ -341,6 +334,7 @@ definitions:
Memory:
description:"Memory limit in bytes."
type:"integer"
format:"int64"
default:0
# Applicable to UNIX platforms
CgroupParent:
@@ -450,6 +444,10 @@ definitions:
OomKillDisable:
description:"Disable OOM Killer for the container."
type:"boolean"
Init:
description:"Run an init inside the container that forwards signals and reaps processes. This field is omitted if empty, and the default (as configured on the daemon) is used."
type:"boolean"
x-nullable:true
PidsLimit:
description:"Tune a container's pids limit. Set -1 for unlimited."
type:"integer"
@@ -615,17 +613,7 @@ definitions:
description:"Network mode to use for this container. Supported standard values are: `bridge`, `host`, `none`, and `container:<name|id>`. Any other value is taken
as a custom network's name to which this container should connect to."
PortBindings:
type:"object"
description:"A map of exposed container ports and the host port they should map to."
additionalProperties:
type:"object"
properties:
HostIp:
type:"string"
description:"The host IP address"
HostPort:
type:"string"
description:"The host port number, as a string"
$ref:"#/definitions/PortMap"
RestartPolicy:
$ref:"#/definitions/RestartPolicy"
AutoRemove:
@@ -719,7 +707,15 @@ definitions:
description:"Gives the container full access to the host."
PublishAllPorts:
type:"boolean"
description:"Allocates a random host port for all of a container's exposed ports."
description:|
Allocates an ephemeral host port for all of a container's
exposed ports.
Ports are de-allocated when the container stops and allocated when the container starts.
The allocated port might be changed when restarting the container.
The port is selected from the ephemeral port range that depends on the kernel.
For example, on Linux the range is defined by `/proc/sys/net/ipv4/ip_local_port_range`.
ReadonlyRootfs:
type:"boolean"
description:"Mount the container's root filesystem as read only."
@@ -776,6 +772,16 @@ definitions:
- "default"
- "process"
- "hyperv"
MaskedPaths:
type:"array"
description:"The list of paths to be masked inside the container (this overrides the default set of paths)"
items:
type:"string"
ReadonlyPaths:
type:"array"
description:"The list of paths to be set as read-only inside the container (this overrides the default set of paths)"
items:
type:"string"
ContainerConfig:
description:"Configuration for a container that is portable between hosts"
@@ -806,7 +812,7 @@ definitions:
description:|
An object mapping ports to an empty object in the form:
`{"<port>/<tcp|udp>": {}}`
`{"<port>/<tcp|udp|sctp>": {}}`
type:"object"
additionalProperties:
type:"object"
@@ -833,9 +839,7 @@ definitions:
type:"string"
Cmd:
description:"Command to run specified as a string or an array of strings."
type:
- "array"
- "string"
type:"array"
items:
type:"string"
Healthcheck:
@@ -849,12 +853,11 @@ definitions:
Volumes:
description:"An object mapping mount point paths inside the container to empty objects."
type:"object"
properties:
additionalProperties:
type:"object"
enum:
- {}
default:{}
additionalProperties:
type:"object"
enum:
- {}
default:{}
WorkingDir:
description:"The working directory for commands to run in."
type:"string"
@@ -863,9 +866,7 @@ definitions:
The entry point for the container as a string or an array of strings.
If the array consists of exactly one empty string (`[""]`) then the entry point is reset to system default (i.e., the entry point used by docker when there is no `ENTRYPOINT` instruction in the `Dockerfile`).
type:
- "array"
- "string"
type:"array"
items:
type:"string"
NetworkDisabled:
@@ -1076,8 +1077,8 @@ definitions:
container's port-number and protocol as key in the format `<port>/<protocol>`,
for example, `80/udp`.
If a container's port is mapped for both `tcp` and `udp`, two separate
entries are added to the mapping table.
If a container's port is mapped for multiple protocols, separate entries
description:"Protocol to use for clients connecting to the plugin."
enum:
- ""
- "moby.plugins.http/v1"
Entrypoint:
type:"array"
items:
@@ -2445,7 +2466,15 @@ definitions:
properties:
PluginSpec:
type:"object"
description:"Invalid when specified with `ContainerSpec`. *(Experimental release only.)*"
description:|
Plugin spec for the service. *(Experimental release only.)*
<p><br /></p>
> **Note**: ContainerSpec, NetworkAttachmentSpec, and PluginSpec are
> mutually exclusive. PluginSpec is only used when the Runtime field
> is set to `plugin`. NetworkAttachmentSpec is used when the Runtime
> field is set to `attachment`.
properties:
Name:
description:"The name or 'alias' to use for the plugin."
@@ -2472,7 +2501,15 @@ definitions:
type:"string"
ContainerSpec:
type:"object"
description:"Invalid when specified with `PluginSpec`."
description:|
Container spec for the service.
<p><br /></p>
> **Note**: ContainerSpec, NetworkAttachmentSpec, and PluginSpec are
> mutually exclusive. PluginSpec is only used when the Runtime field
> is set to `plugin`. NetworkAttachmentSpec is used when the Runtime
> field is set to `attachment`.
properties:
Image:
description:"The image name to use for the container"
@@ -2677,7 +2714,33 @@ definitions:
ConfigName is the name of the config that this references, but this is just provided for
lookup/display purposes. The config in the reference will be identified by its ID.
type:"string"
Isolation:
type:"string"
description:"Isolation technology of the containers running the service. (Windows only)"
enum:
- "default"
- "process"
- "hyperv"
Init:
description:"Run an init inside the container that forwards signals and reaps processes. This field is omitted if empty, and the default (as configured on the daemon) is used."
type:"boolean"
x-nullable:true
NetworkAttachmentSpec:
description:|
Read-only spec type for non-swarm containers attached to swarm overlay
networks.
<p><br /></p>
> **Note**: ContainerSpec, NetworkAttachmentSpec, and PluginSpec are
> mutually exclusive. PluginSpec is only used when the Runtime field
> is set to `plugin`. NetworkAttachmentSpec is used when the Runtime
> field is set to `attachment`.
type:"object"
properties:
ContainerID:
description:"ID of the container represented by this task"
type:"string"
Resources:
description:"Resource requirements which apply to each individual container created as part of the service."
type:"object"
@@ -2795,6 +2858,8 @@ definitions:
- "shutdown"
- "failed"
- "rejected"
- "remove"
- "orphaned"
Task:
type:"object"
@@ -3035,12 +3100,31 @@ definitions:
enum:
- "tcp"
- "udp"
- "sctp"
TargetPort:
description:"The port inside the container."
type:"integer"
PublishedPort:
description:"The port on the swarm hosts."
type:"integer"
PublishMode:
description:|
The mode in which port is published.
<p><br /></p>
- "ingress" makes the target port accessible on on every node,
regardless of whether there is a task for the service running on
that node or not.
- "host" bypasses the routing mesh and publish the port directly on
the swarm node where that service is running.
type:"string"
enum:
- "ingress"
- "host"
default:"ingress"
example:"ingress"
EndpointSpec:
description:"Properties that can be configured to access and load balance a service."
@@ -3309,6 +3393,13 @@ definitions:
Driver:
description:"Name of the secrets driver used to fetch the secret's value from an external secret store"
$ref:"#/definitions/Driver"
Templating:
description:|
Templating driver, if applicable
Templating controls whether and how to evaluate the config payload as
a template. If no driver is set, no templating is used.
description:"OK response to ContainerTop operation"
properties:
Titles:
description:"The ps column titles"
@@ -4934,6 +5065,11 @@ paths:
description:"Only return logs since this time, as a UNIX timestamp"
type:"integer"
default:0
- name:"until"
in:"query"
description:"Only return logs before this time, as a UNIX timestamp"
type:"integer"
default:0
- name:"timestamps"
in:"query"
description:"Add timestamps to every log line"
@@ -4965,6 +5101,8 @@ paths:
items:
type:"object"
x-go-name:"ContainerChangeResponseItem"
title:"ContainerChangeResponseItem"
description:"change item in response to ContainerChanges operation"
required:[Path, Kind]
properties:
Path:
@@ -5038,9 +5176,9 @@ paths:
This endpoint returns a live stream of a container’s resource usage
statistics.
The `precpu_stats` is the CPU statistic of last read, which is used
for calculating the CPU usage percentage. It is not the same as the
`cpu_stats` field.
The `precpu_stats` is the CPU statistic of the *previous* read, and is
used to calculate the CPU usage percentage. It is not an exact copy
of the `cpu_stats` field.
If either `precpu_stats.online_cpus` or `cpu_stats.online_cpus` is
nil then for compatibility with older daemons the length of the
@@ -5315,6 +5453,13 @@ paths:
examples:
application/json:
message:"No such container: c2ada9df5af8"
409:
description:"container is not running"
schema:
$ref:"#/definitions/ErrorResponse"
examples:
application/json:
message:"Container d37cde0fe4ad63c3a7252023b2f9800282894247d145cb5933ddf6e52cc03a28 is not running"
500:
description:"server error"
schema:
@@ -5343,6 +5488,8 @@ paths:
description:"The container has been updated."
schema:
type:"object"
title:"ContainerUpdateResponse"
description:"OK response to ContainerUpdate operation"
properties:
Warnings:
type:"array"
@@ -5696,12 +5843,21 @@ paths:
description:"The container has exit."
schema:
type:"object"
title:"ContainerWaitResponse"
description:"OK response to ContainerWait operation"
required:[StatusCode]
properties:
StatusCode:
description:"Exit code of the container"
type:"integer"
x-nullable:false
Error:
description:"container waiting error, if any"
type:"object"
properties:
Message:
description:"Details of an error"
type:"string"
404:
description:"no such container"
schema:
@@ -5787,7 +5943,7 @@ paths:
headers:
X-Docker-Container-Path-Stat:
type:"string"
description:"TODO"
description:"A base64 - encoded JSON object with some filesystem header information about the path"
400:
description:"Bad parameter"
schema:
@@ -5934,6 +6090,7 @@ paths:
description:"No error"
schema:
type:"object"
title:"ContainerPruneResponse"
properties:
ContainersDeleted:
description:"Container IDs that were deleted"
@@ -6113,8 +6270,19 @@ paths:
type:"integer"
- name:"buildargs"
in:"query"
description:"JSON map of string pairs for build-time variables. Users pass these values at build-time. Docker uses the buildargs as the environment context for commands run via the `Dockerfile` RUN instruction, or for variable expansion in other `Dockerfile` instructions. This is not meant for passing secret values. [Read more about the buildargs instruction.](https://docs.docker.com/engine/reference/builder/#arg)"
type:"integer"
description:>
JSON map of string pairs for build-time variables. Users pass these values at build-time. Docker
uses the buildargs as the environment context for commands run via the `Dockerfile` RUN
instruction, or for variable expansion in other `Dockerfile` instructions. This is not meant for
passing secret values.
For example, the build arg `FOO=bar` would become `{"FOO":"bar"}` in JSON. This would result in the
the query parameter `buildargs={"FOO":"bar"}`. Note that `{"FOO":"bar"}` should be URI component encoded.
[Read more about the buildargs instruction.](https://docs.docker.com/engine/reference/builder/#arg)
type:"string"
- name:"shmsize"
in:"query"
description:"Size of `/dev/shm` in bytes. The size must be greater than 0. If omitted the system uses 64MB."
@@ -6162,6 +6330,16 @@ paths:
Only the registry domain name (and port if not the default 443) are required. However, for legacy reasons, the Docker Hub registry must be specified with both a `https://` prefix and a `/v1/` suffix even though Docker will prefer to use the v2 registry API.
type:"string"
- name:"platform"
in:"query"
description:"Platform in the format os[/arch[/variant]]"
type:"string"
default:""
- name:"target"
in:"query"
description:"Target build stage"
type:"string"
default:""
responses:
200:
description:"no error"
@@ -6185,6 +6363,7 @@ paths:
description:"No error"
schema:
type:"object"
title:"BuildPruneResponse"
properties:
SpaceReclaimed:
description:"Disk space reclaimed in bytes"
@@ -6243,6 +6422,11 @@ paths:
in:"header"
description:"A base64-encoded auth configuration. [See the authentication section for details.](#section/Authentication)"
type:"string"
- name:"platform"
in:"query"
description:"Platform in the format os[/arch[/variant]]"
type:"string"
default:""
tags:["Image"]
/images/{name}/json:
get:
@@ -6365,6 +6549,8 @@ paths:
items:
type:"object"
x-go-name:HistoryResponseItem
title:"HistoryResponseItem"
description:"individual image layer information in response to ImageHistory operation"
Images report these events: `delete`, `import`, `load`, `pull`, `push`, `save`, `tag`, and `untag`
@@ -6883,6 +7098,7 @@ paths:
description:"no error"
schema:
type:"object"
title:"SystemEventsResponse"
properties:
Type:
description:"The type of object emitting the event"
@@ -6950,7 +7166,7 @@ paths:
- `network=<string>` network name or ID
- `node=<string>` node ID
- `plugin`=<string> plugin name or ID
- `scope`＝<string> local or swarm
- `scope`=<string> local or swarm
- `secret=<string>` secret name or ID
- `service=<string>` service name or ID
- `type=<string>` object to filter by, one of `container`, `image`, `volume`, `network`, `daemon`, `plugin`, `node`, `service`, `secret` or `config`
@@ -6966,6 +7182,7 @@ paths:
description:"no error"
schema:
type:"object"
title:"SystemDataUsageResponse"
properties:
LayersSize:
type:"integer"
@@ -7223,6 +7440,9 @@ paths:
User:
type:"string"
description:"The user, and optionally, group to run the exec process inside the container. Format is one of: `user`, `user:group`, `uid`, or `uid:gid`."
WorkingDir:
type:"string"
description:"The working directory for the exec process inside the container."
example:
AttachStdin:false
AttachStdout:true
@@ -7321,7 +7541,12 @@ paths:
description:"No error"
schema:
type:"object"
title:"ExecInspectResponse"
properties:
CanRemove:
type:"boolean"
DetachKeys:
type:"string"
ID:
type:"string"
Running:
@@ -7387,6 +7612,8 @@ paths:
description:"Summary volume data that matches the query"
schema:
type:"object"
title:"VolumeListResponse"
description:"Volume list response"
required:[Volumes, Warnings]
properties:
Volumes:
@@ -7463,6 +7690,8 @@ paths:
description:"Volume configuration"
schema:
type:"object"
description:"Volume configuration"
title:"VolumeConfig"
properties:
Name:
description:"The new volume's name. If not specified, Docker generates a name."
@@ -7568,6 +7797,7 @@ paths:
description:"No error"
schema:
type:"object"
title:"VolumePruneResponse"
properties:
VolumesDeleted:
description:"Volumes that were deleted"
@@ -7746,6 +7976,7 @@ paths:
description:"No error"
schema:
type:"object"
title:"NetworkCreateResponse"
properties:
Id:
description:"The ID of the created network."
@@ -7846,7 +8077,7 @@ paths:
summary:"Connect a container to a network"
operationId:"NetworkConnect"
consumes:
- "application/octet-stream"
- "application/json"
responses:
200:
description:"No error"
@@ -7948,6 +8179,7 @@ paths:
description:"No error"
schema:
type:"object"
title:"NetworkPruneResponse"
properties:
NetworksDeleted:
description:"Networks that were deleted"
@@ -7999,6 +8231,7 @@ paths:
items:
description:"Describes a permission the user has to accept upon installing the plugin."
type:"object"
title:"PluginPrivilegeItem"
properties:
Name:
type:"string"
@@ -8548,14 +8781,28 @@ paths:
nodes in order to reach the containers running on this node. Using this parameter it is possible to
separate the container data traffic from the management traffic of the cluster.
type:"string"
DefaultAddrPool:
description:|
Default Address Pool specifies default subnet pools for global scope networks.
type:"array"
items:
type:"string"
example:["10.10.0.0/16","20.20.0.0/16"]
ForceNewCluster:
description:"Force creation of a new swarm."
type:"boolean"
SubnetSize:
description:|
SubnetSize specifies the subnet size of the networks created from the default subnet pool
type:"integer"
format:"uint32"
Spec:
$ref:"#/definitions/SwarmSpec"
example:
ListenAddr:"0.0.0.0:2377"
AdvertiseAddr:"192.168.1.1:2377"
DefaultAddrPool:["10.10.0.0/8","20.20.0.0/8"]
SubnetSize:24
ForceNewCluster:false
Spec:
Orchestration:{}
@@ -8701,6 +8948,7 @@ paths:
description:"no error"
schema:
type:"object"
title:"UnlockKeyResponse"
properties:
UnlockKey:
description:"The swarm's unlock key."
@@ -8792,6 +9040,7 @@ paths:
description:"no error"
schema:
type:"object"
title:"ServiceCreateResponse"
properties:
ID:
description:"The ID of the created service."
@@ -9463,13 +9712,7 @@ paths:
201:
description:"no error"
schema:
type:"object"
properties:
ID:
description:"The ID of the created secret."
type:"string"
example:
ID:"ktnbjxoalbkvbvedmg1urrz8h"
$ref:"#/definitions/IdResponse"
409:
description:"name conflicts with an existing object"
schema:
@@ -9668,13 +9911,7 @@ paths:
201:
description:"no error"
schema:
type:"object"
properties:
ID:
description:"The ID of the created config."
type:"string"
example:
ID:"ktnbjxoalbkvbvedmg1urrz8h"
$ref:"#/definitions/IdResponse"
409:
description:"name conflicts with an existing object"
@@ -16,7 +16,6 @@ type ContainerCreateConfig struct {
HostConfig*container.HostConfig
NetworkingConfig*network.NetworkingConfig
AdjustCPUSharesbool
Platformstring
}
// ContainerRmConfig holds arguments for the container remove
@@ -26,19 +25,6 @@ type ContainerRmConfig struct {
ForceRemove,RemoveVolume,RemoveLinkbool
}
// ContainerCommitConfig contains build configs for commit operation,
// and is used when making a commit with the current state of the container.
typeContainerCommitConfigstruct{
Pausebool
Repostring
Tagstring
Authorstring
Commentstring
// merge container config into commit config before commit
MergeConfigsbool
Config*container.Config
}
// ExecConfig is a small subset of the Config struct that holds the configuration
// for the exec feature of docker.
typeExecConfigstruct{
@@ -51,6 +37,7 @@ type ExecConfig struct {
Detachbool// Execute in detach mode
DetachKeysstring// Escape keys for detach
Env[]string// Environment variables
WorkingDirstring// Working directory
Cmd[]string// Execution commands and args
}
@@ -68,3 +55,10 @@ type PluginEnableConfig struct {
typePluginDisableConfigstruct{
ForceDisablebool
}
// NetworkListConfig stores the options available for listing networks
typeNetworkListConfigstruct{
// TODO(@cpuguy83): naming is hard, this is pulled from what was being used in the router before moving here
Detailedbool
Verbosebool
}
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
