The validate step in CI was broken, due to a combination of
086b4541cf, fbdd437d29,
and 85733620eb being merged to master.
```
api/types/filters/parse.go:39:1: exported method `Args.Keys` should have comment or be unexported (golint)
func (args Args) Keys() []string {
^
daemon/config/builder.go:19:6: exported type `BuilderGCFilter` should have comment or be unexported (golint)
type BuilderGCFilter filters.Args
^
daemon/config/builder.go:21:1: exported method `BuilderGCFilter.MarshalJSON` should have comment or be unexported (golint)
func (x *BuilderGCFilter) MarshalJSON() ([]byte, error) {
^
daemon/config/builder.go:35:1: exported method `BuilderGCFilter.UnmarshalJSON` should have comment or be unexported (golint)
func (x *BuilderGCFilter) UnmarshalJSON(data []byte) error {
^
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 9d726f1c18)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
If anything marshals the daemon config now or in the future
this commit ensures the correct canonical form for the builder
GC policies' filters.
Signed-off-by: Tibor Vass <tibor@docker.com>
(cherry picked from commit 85733620eb)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
For backwards compatibility, the old incorrect object format for
builder.GC.Rule.Filter still works but is deprecated in favor of array of
strings akin to what needs to be passed on the CLI.
Signed-off-by: Tibor Vass <tibor@docker.com>
(cherry picked from commit fbdd437d29)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
buildkit supports entitlements like network-host and security-insecure.
this patch aims to make it configurable through daemon.json file.
by default network-host is enabled & secuirty-insecure is disabled.
Signed-off-by: Kunal Kushwaha <kunal.kushwaha@gmail.com>
(cherry picked from commit 8b7bbf180f)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Commit 7019b60d0d added these
env-vars to other stages, but forgot to update the DCO stage,
which also does a diff to validate commits that are in a PR.
Also adding openssh-client, for situations where the upstream
needs to be accessed through an ssh connection.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 7c5fd83c22)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Before this change we just accept that any error is "not found" and it
could be something else, but even if it it is just a "not found" kind of
error this should be dealt with from the container store and not the
event processor.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit 54e30a62d3)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Previously we were re-using schema2.DeserializedManifest to handle oci
manifests. The issue lies in the fact that distribution started
validating the media type string during json deserialization. This
change broke our usage of that type.
Instead distribution now provides direct support for oci schemas, so use
that instead of our custom handlers.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit e443512ce4)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This change caused a regression, causing the DOCKER-USER chain
to not be created, despite iptables being enabled on the daemon.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This is a continuation of 2a08f33166247da9d4c09d4c6c72cbb8119bf8df;
When running CI in other repositories (e.g. Docker's downstream
docker/engine repository), or other branches, the validation
scripts were calculating the list of changes based on the wrong
information.
This lead to weird failures in CI in a branch where these values
were not updated ':-) (CI on a pull request failed because it detected
that new tests were added to the deprecated `integration-cli` test-suite,
but the pull request did not actually make changes in that area).
This patch uses environment variables set by Jenkins to sets the
correct target repository (and branch) to compare to.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 7019b60d0d)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The patch made in docker/libnetwork#2450 caused a breaking change in the
networking behaviour, causing Kubernetes installations on Docker Desktop
(and possibly other setups) to fail.
Rolling back this change in the 19.03 branch while we investigate if there
are alternatives.
diff: 45c710223c...96bcc0dae8
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
When slirp4netns v0.4.0+ is used, now slirp4netns is hardened using
mount namespace ("sandbox") and seccomp to mitigate potential
vulnerabilities.
bump up rootlesskit: 2fcff6ceae...791ac8cb20
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
(cherry picked from commit e20b7323fb)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Also updated the related docs.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 5175ed54e5)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The docker-py tests were broken, because the version of
py-test that was used, used a dependency that had a new
major release with a breaking change.
Unfortunately, it was not pinned to a specific version,
so when the dependency did the release, py-test broke;
```
22:16:47 Traceback (most recent call last):
22:16:47 File "/usr/local/bin/pytest", line 10, in <module>
22:16:47 sys.exit(main())
22:16:47 File "/usr/local/lib/python3.6/site-packages/_pytest/config/__init__.py", line 61, in main
22:16:47 config = _prepareconfig(args, plugins)
22:16:47 File "/usr/local/lib/python3.6/site-packages/_pytest/config/__init__.py", line 182, in _prepareconfig
22:16:47 config = get_config()
22:16:47 File "/usr/local/lib/python3.6/site-packages/_pytest/config/__init__.py", line 156, in get_config
22:16:47 pluginmanager.import_plugin(spec)
22:16:47 File "/usr/local/lib/python3.6/site-packages/_pytest/config/__init__.py", line 530, in import_plugin
22:16:47 __import__(importspec)
22:16:47 File "/usr/local/lib/python3.6/site-packages/_pytest/tmpdir.py", line 25, in <module>
22:16:47 class TempPathFactory(object):
22:16:47 File "/usr/local/lib/python3.6/site-packages/_pytest/tmpdir.py", line 35, in TempPathFactory
22:16:47 lambda p: Path(os.path.abspath(six.text_type(p)))
22:16:47 TypeError: attrib() got an unexpected keyword argument 'convert'
```
docker-py master has a fix for this (bumping the version of
`py-test`), but it's not in a release yet, and the docker cli that's used
in our CI is pinned to 17.06, which doesn't support building from a remote
git repository from a specific git commit.
To fix the immediate situation, this patch switches the docker-py
tests to run from the master branch.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 48353e16fe)
Signed-off-by: Andrew Hsu <andrewhsu@docker.com>
The `/session` endpoint left experimental in API V1.39 through
239047c2d3 and
01c9e7082e, but the API reference
was not updated accordingly.
This updates the API documentation to match the change.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 6756f5f378)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
full diff: https://github.com/golang/go/compare/go1.12.9...go1.12.10
```
Hi gophers,
We have just released Go 1.13.1 and Go 1.12.10 to address a recently reported security issue. We recommend that all affected users update to one of these releases (if you're not sure which, choose Go 1.13.1).
net/http (through net/textproto) used to accept and normalize invalid HTTP/1.1 headers with a space before the colon, in violation of RFC 7230. If a Go server is used behind an uncommon reverse proxy that accepts and forwards but doesn't normalize such invalid headers, the reverse proxy and the server can interpret the headers differently. This can lead to filter bypasses or request smuggling, the latter if requests from separate clients are multiplexed onto the same upstream connection by the proxy. Such invalid headers are now rejected by Go servers, and passed without normalization to Go client applications.
The issue is CVE-2019-16276 and Go issue golang.org/issue/34540.
Thanks to Andrew Stucki, Adam Scarr (99designs.com), and Jan Masarik (masarik.sh) for discovering and reporting this issue.
Downloads are available at https://golang.org/dl for all supported platforms.
Alla prossima,
Filippo on behalf of the Go team
```
From the patch: 6e6f4aaf70
```
net/textproto: don't normalize headers with spaces before the colon
RFC 7230 is clear about headers with a space before the colon, like
X-Answer : 42
being invalid, but we've been accepting and normalizing them for compatibility
purposes since CL 5690059 in 2012.
On the client side, this is harmless and indeed most browsers behave the same
to this day. On the server side, this becomes a security issue when the
behavior doesn't match that of a reverse proxy sitting in front of the server.
For example, if a WAF accepts them without normalizing them, it might be
possible to bypass its filters, because the Go server would interpret the
header differently. Worse, if the reverse proxy coalesces requests onto a
single HTTP/1.1 connection to a Go server, the understanding of the request
boundaries can get out of sync between them, allowing an attacker to tack an
arbitrary method and path onto a request by other clients, including
authentication headers unknown to the attacker.
This was recently presented at multiple security conferences:
https://portswigger.net/blog/http-desync-attacks-request-smuggling-reborn
net/http servers already reject header keys with invalid characters.
Simply stop normalizing extra spaces in net/textproto, let it return them
unchanged like it does for other invalid headers, and let net/http enforce
RFC 7230, which is HTTP specific. This loses us normalization on the client
side, but there's no right answer on the client side anyway, and hiding the
issue sounds worse than letting the application decide.
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
```
docker/integration-cli/checker/checker.go
Line 12: warning: exported type Compare should have comment or be unexported (golint)
Line 14: warning: exported function False should have comment or be unexported (golint)
Line 20: warning: exported function True should have comment or be unexported (golint)
Line 26: warning: exported function Equals should have comment or be unexported (golint)
Line 32: warning: exported function Contains should have comment or be unexported (golint)
Line 38: warning: exported function Not should have comment or be unexported (golint)
Line 52: warning: exported function DeepEquals should have comment or be unexported (golint)
Line 58: warning: exported function HasLen should have comment or be unexported (golint)
Line 64: warning: exported function IsNil should have comment or be unexported (golint)
Line 70: warning: exported function GreaterThan should have comment or be unexported (golint)
Line 76: warning: exported function NotNil should have comment or be unexported (golint)
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 6397dd4d31)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The `-test.timeout=5m` was glued directly after the current `TESTFLAGS`,
causing them to be non-functional;
Before:
make TESTDEBUG=1 TESTDIRS='github.com/docker/docker/pkg/filenotify' TESTFLAGS='-test.run TestPollerEvent' test-unit
+ mkdir -p bundles
+ gotestsum --format=standard-quiet --jsonfile=bundles/go-test-report.json --junitfile=bundles/junit-report.xml -- -tags 'netgo seccomp libdm_no_deferred_remove' -cover -coverprofile=bundles/profile.out -covermode=atomic -test.run TestPollerEvent-test.timeout=5m github.com/docker/docker/pkg/filenotify
testing: warning: no tests to run
ok github.com/docker/docker/pkg/filenotify 0.003s coverage: 0.0% of statements [no tests to run]
DONE 0 tests in 0.298s
After:
make TESTDEBUG=1 TESTDIRS='github.com/docker/docker/pkg/filenotify' TESTFLAGS='-test.run TestPollerEvent' test-unit
+ mkdir -p bundles
+ gotestsum --format=standard-quiet --jsonfile=bundles/go-test-report.json --junitfile=bundles/junit-report.xml -- -tags 'netgo seccomp libdm_no_deferred_remove' -cover -coverprofile=bundles/profile.out -covermode=atomic -test.run TestPollerEvent -test.timeout=5m github.com/docker/docker/pkg/filenotify
ok github.com/docker/docker/pkg/filenotify 0.608s coverage: 44.7% of statements
DONE 1 tests in 0.922s
This was introduced in 42f0a0db75
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 0620990307)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Replaced using a bit of grep-ing;
```
find . -name "*_test.go" -exec sed -E -i 's#assert.Assert\((.*), fmt.Sprintf\((.*)\)\)$#assert.Assert\(\1, \2\)#g' '{}' \;
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 0fabf3e41e)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
After the commit faaffd5d6d ("Windows:Disable 2 restart test when
Hyper-V") some tests became skipped on linux:
SKIP: docker_cli_restart_test.go:167: DockerSuite.TestRestartContainerSuccess (unmatched requirement IsolationIsProcess)
SKIP: docker_cli_restart_test.go:240: DockerSuite.TestRestartPolicyAfterRestart (unmatched requirement IsolationIsProcess)
But AFAIU it is highly unlikely that we actually meant to skip them on linux.
https://github.com/moby/moby/issues/39625
Signed-off-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>
(cherry picked from commit b469933b06)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
- remove -check.* flags
- use (per-test) -timeout flag
- allow user to override TEST_SKIP_* regardless of TESTFLAGS
- remove test-imports validation
Signed-off-by: Tibor Vass <tibor@docker.com>
(cherry picked from commit 7cd028f2d0)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
go run rm-gocheck.go redress '[^/]\bcheck\.Suite\(.*\{\s*$' \
"integration-cli/check_test.go" "integration-cli/docker_cli_external_volume_driver_unix_test.go" "integration-cli/docker_cli_network_unix_test.go"
Signed-off-by: Tibor Vass <tibor@docker.com>
(cherry picked from commit 6a8a9738ec)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The following "rm-gocheck:"-prefixed commits were generated by
go run rm-gocheck.go --commit
Signed-off-by: Tibor Vass <tibor@docker.com>
(cherry picked from commit 8f64611c83)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Tests fixed in this patch used to compile and pass successfully,
despite checking if non-nullable types are not nil.
These would have become compile errors once go-check is removed.
About TestContainerAPIPsOmitFields:
Basically what happened is that this test got refactored to start using the API types
and API client library instead of custom types and stdlib's http functions.
This test used to test an API regression which could possibly be a unit test.
However because PublicPort and IP are not nullable types, this test became useless.
Signed-off-by: Tibor Vass <tibor@docker.com>
(cherry picked from commit e07a3f2917)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Don't use two-stage mount in TestContainersAPICreateMountsCreate();
apparently it was written before mount.Mount() could accept propagation
flags.
While at it, remove rw as this is the default.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 1cfdb2ffb8)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The only option we supply is either BIND or a mount propagation flag,
so it makes sense to specify the flag value directly, rather than using
parseOptions() every time.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit ec248fe61d)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
It is sufficient to add "rprivate" to mount flags.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit a6773f69f2)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
For bind mounts, fstype argument to mount(2) is ignored.
Usual convention is either empty string or "none".
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 4e65b17ac4)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Eliminate double call to parseOptions() from Mount()
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 80fce834ad)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Current code in MakeMount parses /proc/self/mountinfo twice:
first in call to Mounted(), then in call to Mount(). Use
ForceMount() to eliminate such double parsing.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit aa60541877)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
TestLogBlocking is intended to test that the Log method blocks by
default. It does this by mocking out the internals of the
awslogs.logStream and replacing one of its internal channels with one
that is controlled by the test. The call to Log occurs inside a
goroutine. Go may or may not schedule the goroutine immediately and the
blocking may or may not be observed outside the goroutine immediately
due to decisions made by the Go runtime. This change adds a small
timeout for test failure so that the Go runtime has the opportunity to
run the goroutine before the test fails.
Signed-off-by: Samuel Karp <skarp@amazon.com>
(cherry picked from commit fd94bae0b8)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
I've seen this test fail a number of times recently on RS1
Looking at failures, the test is taking a long time ro run (491.77s, which is
more than 8 minutes), so perhaps it's just too slow on RS1, which may be
because we switch to a different base image, or because we're now running
on different machines.
Compared to RS5 (still slow, but a lot faster);
```
--- PASS: Test/DockerSuite/TestAPIImagesSaveAndLoad (146.25s)
```
```
--- FAIL: Test/DockerSuite/TestAPIImagesSaveAndLoad (491.77s)
cli.go:45: assertion failed:
Command: d:\CI-5\CI-93d2cf881\binary\docker.exe inspect --format {{.Id}} sha256:69e7c1ff23be5648c494294a3808c0ea3f78616fad67bfe3b10d3a7e2be5ff02
ExitCode: 1
Error: exit status 1
Stdout:
Stderr: Error: No such object: sha256:69e7c1ff23be5648c494294a3808c0ea3f78616fad67bfe3b10d3a7e2be5ff02
Failures:
ExitCode was 1 expected 0
Expected no error
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 5adaf52953)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This `ENV` was added to the Dockerfile in b96093fa56,
when the repository used per-architecture Dockerfiles, and some architectures needed
a different configuration.
Now that we use a multi-arch Dockerfile, and CI uses a Jenkinsfile, we can remove
this `ENV` from the Dockerfile, and set it in CI instead if needed.
Also updated the wording and fixed linting issues in hack/validate/gometalinter
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit a464a3d51f)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This test frequently fails on Windows RS1 (mainly), so skipping it
for now on Windows;
```
ok github.com/docker/docker/daemon/logger 0.525s coverage: 43.0% of statements
time="2019-09-09T20:37:35Z" level=info msg="Trying to get region from EC2 Metadata"
time="2019-09-09T20:37:36Z" level=info msg="Log stream already exists" errorCode=ResourceAlreadyExistsException logGroupName= logStreamName= message= origError="<nil>"
--- FAIL: TestLogBlocking (0.02s)
cloudwatchlogs_test.go:313: Expected to be able to read from stream.messages but was unable to
time="2019-09-09T20:37:36Z" level=error msg=Error
time="2019-09-09T20:37:36Z" level=error msg="Failed to put log events" errorCode=InvalidSequenceTokenException logGroupName=groupName logStreamName=streamName message="use token token" origError="<nil>"
time="2019-09-09T20:37:36Z" level=error msg="Failed to put log events" errorCode=DataAlreadyAcceptedException logGroupName=groupName logStreamName=streamName message="use token token" origError="<nil>"
time="2019-09-09T20:37:36Z" level=info msg="Data already accepted, ignoring error" errorCode=DataAlreadyAcceptedException logGroupName=groupName logStreamName=streamName message="use token token"
FAIL
coverage: 78.2% of statements
FAIL github.com/docker/docker/daemon/logger/awslogs 0.630s
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 6c75c86240)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Starting the daemon should not load the busybox image again
in most cases, so add a new `StartNodeWithBusybox` function
to be clear that this one loads the busybox image, and use
`StartNode()` for cases where loading the busybox image is
not needed.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit ead3f4e7c8)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The daemon was already created and started with the busybox
image loaded, so there's no need to load the image again.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 8fc23588f1)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Noticed this test container not exiting correctly while debugging
another issue. Before this change, signals were being eaten by bash, now
they are hanlded by top. This cuts the test time in half since it
doesn't have to wait for docker to SIGKILL it.
Old:
PASS: docker_cli_swarm_test.go:840: DockerSwarmSuite.TestSwarmServiceTTY 18.997s
New:
PASS: docker_cli_swarm_test.go:840: DockerSwarmSuite.TestSwarmServiceTTY 6.293s
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit e6c5563ae9)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Instead of using the locally crafted `serviceContainerCount()` utility
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit f874f8b6fd)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The Windows Dockerfile downloads the Go binaries, which (unlike
the Golang images) do not have a trailing `.0` in their version.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 61450a651b)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This was an oversight when changing the Dockerfile to use a build-arg;
the Windows Dockerfile downloads the Go binaries, which never have a
trailing `.0`.
This patch makes sure that the trailing zero (if any) is removed.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit c5bd6e3dc7)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
There's already a step in "Nuke Everything", but lets' stop it
after stopping the daemon as well
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit e1636ad5fa)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Hyper-V isolated containers do not allow file-operations on a
running container. This test currently uses `docker cp` to verify
that the WORKDIR was automatically created, which cannot be done
while the container is running.
```
FAIL: docker_cli_create_test.go:302: DockerSuite.TestCreateWithWorkdir
assertion failed:
Command: d:\CI-7\CI-f3768a669\binary\docker.exe cp foo:c:\home\foo\bar c:\tmp
ExitCode: 1
Error: exit status 1
Stdout:
Stderr: Error response from daemon: filesystem operations against a running Hyper-V container are not supported
Failures:
ExitCode was 1 expected 0
Expected no error
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit ac9ef840ef)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This change will cause Jenkins to only build the
latest HEAD of a PR branch, cancelling any
previous builds that may already be in progress.
This will decrease feedback time and help mitigate
resource contention.
Signed-off-by: Peter Salvatore <peter@psftw.com>
(cherry picked from commit 85bcc524ea)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
CI already stores the logs of the test daemon, so we might as well
store them as artifacts
```
[2019-09-03T12:49:39.835Z] INFO: Tidying up at end of run
[2019-09-03T12:49:39.835Z] INFO: Saving daemon under test log (d:\CI-2\CI-3593e7622\dut.out) to C:\windows\TEMP\CIDUT.out
[2019-09-03T12:49:39.835Z] INFO: Saving daemon under test log (d:\CI-2\CI-3593e7622\dut.err) to C:\windows\TEMP\CIDUT.err
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 6ee61f5493)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Trying to see if this helps with the cleanup step exiting in CI, but
Jenkins continuing to wait for the script to end afterwards.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 8e8c52c4ab)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Seen failing a couple of times:
```
[2019-09-02T08:40:15.796Z] =================================== FAILURES ===================================
[2019-09-02T08:40:15.796Z] __________________ AttachContainerTest.test_attach_no_stream ___________________
[2019-09-02T08:40:15.796Z] tests/integration/api_container_test.py:1250: in test_attach_no_stream
[2019-09-02T08:40:15.796Z] assert output == 'hello\n'.encode(encoding='ascii')
[2019-09-02T08:40:15.796Z] E AssertionError: assert b'' == b'hello\n'
[2019-09-02T08:40:15.796Z] E Right contains more items, first extra item: 104
[2019-09-02T08:40:15.796Z] E Use -v to get the full diff
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit ce77a804b8)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
`TestBuildBuildTimeArgEnv` and `TestBuildBuildTimeArgEmptyValVariants` were
using non-standard comparisons.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit dbde4786e4)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
- Add quotes to prevent word splitting in `cp` statement (SC2046)
- Replace legacy back tics with `$()`
- Replace `which` with `command -v` (SC2230)
- Fix incorrect (`==`) comparison
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 70d3677825)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Instead of using a `initDispatchTestCases()` function, declare the test-table
inside `TestDispatch` itself, and run the tests as subtests.
```
[2019-08-27T15:14:51.072Z] builder/dockerfile/evaluator_test.go:18:2: `name` is unused (structcheck)
[2019-08-27T15:14:51.072Z] name, expectedError string
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit a3f9cb5b63)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Some integration tests are known to fail if Windows Defender is
enabled. On the machines that run our CI, defender is disabled
for that reason.
Contributors likely will have defender enabled, and because of
that are currently not able to run the integration tests.
This patch changes the ERROR into a WARNING, so that contributors
can still run (a limited set of) the integration tests, but get
informed that some may fail.
We should make this requirement more specific, and only skip
tests that are known to require defender to be disabled, but
while that's not yet in place, let's print a warning instead.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 31885181fc)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
- use subtests to make it clearer what the individual test-cases
are, and to prevent tests from depending on values set by the
previous test(s).
- remove redundant messages in assert (gotest.tools already prints
a useful message if assertions fail).
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 556d26c07d)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
TestBuildMulitStageResetScratch testcase was actually meant to be
TestBuildMulitStageResetScratch
Signed-off-by: Vitaly Ostrosablin <tmp6154@yandex.ru>
(cherry picked from commit c266d8fe56)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
New features
- build CRIU with Android NDK
- C/R of
- IP RAW sockets
- lsm: dump and restore any SELinux process label
- support restoring ghost files on readonly mounts
Bugfixes
- Do not lock network if running in the host network namespace
- Fix RPC configuration file handling
- util: don't leak file descriptors to third-party tools
- small fixes here and there
Improvements
- travis: switch to the Ubuntu Xenial
- travis-ci: Enable ia32 tests
- Many improvements and bug fixes in the libcriu
- Changes in the API and ABI (SONAME increased from 1 to 2)
full diff: https://github.com/checkpoint-restore/criu/compare/v3.11...v3.12
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 00ad0222ce)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Generate more unique names, based on architecture and test-suite name.
Clean up the path to this integration test to create a useful package name.
"$dir" can be either absolute (/go/src/github.com/docker/docker/integration/foo)
or relative (./integration/foo). To account for both, first we strip the
absolute path, then any leading periods and slashes.
For the package-name, we use periods as separator instead of slashes, to be more
in-line with Java package names (which is what junit.xml was originally designed
for).
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit f007b0150a)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Without these options set, test2json does not include a `Time`
field in the generated JSON;
{"Action":"run","Test":"TestCgroupNamespacesBuild"}
{"Action":"output","Test":"TestCgroupNamespacesBuild","Output":"=== RUN TestCgroupNamespacesBuild\n"}
{"Action":"output","Test":"TestCgroupNamespacesBuild","Output":"--- PASS: TestCgroupNamespacesBuild (1.70s)\n"}
...
{"Action":"pass","Test":"TestCgroupNamespacesBuild"}
As a result, `gotestsum` was not able to calculate test-duration, and
reported `time="0.000000"` for all tests;
<testcase classname="amd64.integration.build" name="TestCgroupNamespacesBuild" time="0.000000"></testcase>
With this patch applied:
{"Time":"2019-08-23T22:42:41.644361357Z","Action":"run","Package":"amd64.integration.build","Test":"TestCgroupNamespacesBuild"}
{"Time":"2019-08-23T22:42:41.644367647Z","Action":"output","Package":"amd64.integration.build","Test":"TestCgroupNamespacesBuild","Output":"=== RUN TestCgroupNamespacesBuild\n"}
{"Time":"2019-08-23T22:42:44.926933252Z","Action":"output","Package":"amd64.integration.build","Test":"TestCgroupNamespacesBuild","Output":"--- PASS: TestCgroupNamespacesBuild (3.28s)\n"}
...
{"Time":"2019-08-23T22:42:44.927003836Z","Action":"pass","Package":"amd64.integration.build","Test":"TestCgroupNamespacesBuild","Elapsed":3.28}
Which now correctly reports the test's duration:
<testcase classname="amd64.integration.build" name="TestCgroupNamespacesBuild" time="3.280000"></testcase>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit d2e00d62e2)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
'Namespace' parallel runs by bind-mounting a different directory
in the container, instead of making the tests running inside
the container aware of the namespaced location.
This makes it transparent to the tests, and slightly reduces
complexity.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 3262a69be6)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The inContainer check isn't really useful anymore.
Even though it was said that we shouldn't rely on its existence back in
2016, we're now in 2019 and this thing still exists so we should just
rely on it now to check whether or not we're in a container.
Signed-off-by: Eli Uriegas <eli.uriegas@docker.com>
(cherry picked from commit f5cd8fdd44)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Call helper for starting swarm agents and add some logging with daemon
id's when joining the swarm.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit b0fe0dff7a)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The `DockerDaemonSuite.SetUpTest` already checks for Linux and a local daemon;
```
func (s *DockerDaemonSuite) SetUpTest(c *check.C) {
testRequires(c, DaemonIsLinux, testEnv.IsLocalDaemon)
s.d = daemon.New(c, dockerBinary, dockerdBinary, testdaemon.WithEnvironment(testEnv.Execution))
}
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 7f37d99ef5)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Code retrying service update operations when receiving "update out of
sequence" errors was removed because of a misunderstanding, which has
made tests flaky. This re-adds the "CmdRetryOutOfSequence" method, and
uses it in TestSwarmPublishAdd to avoid flaky behavior.
Signed-off-by: Drew Erny <drew.erny@docker.com>
(cherry picked from commit 1de914695b)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The session endpoint is no longer experimental since
01c9e7082e, so we don't
need to start an experimental daemon.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit becd29c665)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
A bit too quick on the trigger on some text completion I think...
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit 5d818213ff)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This is just noise due to timing. I picked `> 2` just based on
logs from tests I've seen there's always 1 or 2.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit 15675e28f1)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The directory used for storage was either changed or new directories
were added.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 6a64a4deec)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This allows our tests, which all share a containerd instance, to be a
bit more isolated by setting the containerd namespaces to the generated
daemon ID's rather than the default namespaces.
This came about because I found in some cases we had test daemons
failing to start (really very slow to start) because it was (seemingly)
processing events from other tests.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit 24ad2f486d)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Removes some test functions that were unused:
- bridgeNfIP6tables
- ambientCapabilities (added to support #26979, which was reverted in #27737)
- overlay2Supported
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit c887b09abc)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Both Linux and Windows now use busybox, so no need to keep a
constant for this.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 27f432ca57)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
All current versions of Docker support exec, so no need
to check for this.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 7204341950)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The daemon requires kernel 3.10 or up to start, so there's no need
to check if the daemon is kernel 3.8 or up.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 691eb14256)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
IPVLAN no longer is experimental since 3ab093d567,
so there's no need to set this option.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit dae9bac675)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
These tests require a local daemon, and are not built on Windows
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 1e4bd2623a)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Their inclusion is no longer defensible.
closes#39981
Signed-off-by: Harald Niesche <harald@niesche.de>
(cherry picked from commit 77d3c68f97)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
In case jsonlogfile is used with max-file=1 and max-size set,
the log rotation is not perfomed; instead, the log file is closed
and re-open with O_TRUNC.
This situation is not handled by the log reader in follow mode,
leading to an issue of log reader being stuck forever.
This situation (file close/reopen) could be handled in waitRead(),
but fsnotify library chose to not listen to or deliver this event
(IN_CLOSE_WRITE in inotify lingo).
So, we have to handle this by checking the file size upon receiving
io.EOF from the log reader, and comparing the size with the one received
earlier. In case the new size is less than the old one, the file was
truncated and we need to seek to its beginning.
Fixes#39235.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 9cd24ba605)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
1.This commit replaces serviceRunningCount with
swarm.RunningTasksCount to accurately check if the
service is running with the accurate number of instances
or not. serviceRunningCount was only checking the ServiceList
and was not checking if the tasks were running or not
This adds a safe barrier to execute docker network inspect
commands for overlay networks which get created
asynchronously via Swarm
2. Make sure client connections are closed
3. Make sure every service and network name is unique
4. Make sure services and networks are cleaned up
Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>
(cherry picked from commit f3a3ea0d3c)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Looks like TestServiceWithDefaultAddressPoolInit is failing
randomly in CI. I am not able to reproduce the issue locally
but this has been reported few times. So I tried to modify
code and see if I can fix the random failure.
Signed-off-by: selansen <elango.siva@docker.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 88578aa9e9)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Line 441: warning: context.Context should be the first parameter of a function (golint)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit dac5710b68)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Line 30: warning: context.Context should be the first parameter of a function (golint)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 123e29f44a)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Line 59: warning: context.Context should be the first parameter of a function (golint)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 9f9b4290b9)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
```
Line 25: warning: context.Context should be the first parameter of a function (golint)
Line 44: warning: context.Context should be the first parameter of a function (golint)
Line 52: warning: context.Context should be the first parameter of a function (golint)
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit b4c46b0dac)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Line 30: warning: context.Context should be the first parameter of a function (golint)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit caec45a37f)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This prevents restarting event processing in a tight loop.
You can see this with the following steps:
```terminal
$ containerd &
$ dockerd --containerd=/run/containerd/containerd.sock &
$ pkill -9 containerd
```
At this point you will be spammed with logs such as:
```
ERRO[2019-07-12T22:29:37.318761400Z] failed to get event error="rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial unix /run/containerd/containerd.sock: connect: connection refused\"" module=libcontainerd namespace=plugins.moby
```
Without this change you can quickly end up with gigabytes of log data.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit 1acaf2aabe)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This improves the select logic for the restart tests or starting the
daemon in general. With the way the ticker and select was setup, it was
possible for only the timeout to be displayed and not the wait errors.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
(cherry picked from commit 402433a5e4)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Reported by govet linter:
> daemon/monitor.go:57:9: lostcancel: the cancel function returned by context.WithTimeout should be called, not discarded, to avoid a context leak (govet)
> ctx, _ := context.WithTimeout(context.Background(), 2*time.Second)
> ^
> daemon/monitor.go:128:9: lostcancel: the cancel function returned by context.WithTimeout should be called, not discarded, to avoid a context leak (govet)
> ctx, _ := context.WithTimeout(context.Background(), 2*time.Second)
> ^
Fixes: b5f288 ("Handle blocked I/O of exec'd processes")
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 53cbf1797b)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
By convention, containers spawned by jenkins jobs have the name:
docker-pr${BUILD_NUMBER}
That works fine for jobs with a single container. This commit cleans up
when multiple containers are spawned with the convention that their names
share the same "docker-pr${BUILD_NUMBER}-" prefix.
Signed-off-by: Tibor Vass <tibor@docker.com>
(cherry picked from commit f470698c2c)
Signed-off-by: Andrew Hsu <andrewhsu@docker.com>
The Blue Ocean UI truncates names, which makes it possible to distinguish
which Windows stage is RS1 or RS5. This patch shortens those names so that they
fit in the Blue Ocean UI.
Other stages and parameters were renamed as well to better reflect what they're running;
Before | After
-------------------|--------------------------------
janky | amd64
power | ppc64le
power-master | ppc64le integration-cli
windowsRS1 | win-RS1
windowsRS5-process | win-RS5
z | s390x
z-master | x390x integration-cli
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
WIP renames
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit c18f793f40)
Signed-off-by: Andrew Hsu <andrewhsu@docker.com>
full diff: bbe341867e...f35d9100f2
changes included:
- docker/swarmkit#2891 [19.03 backport] Remove hardcoded IPAM config subnet value for ingress network
- backport of docker/swarmkit#2890 Remove hardcoded IPAM config subnet value for ingress network
- fixes [ENGORC-2651] Specifying --default-addr-pool for docker swarm init is not picked up by ingress network
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The warning on pull was incorrectly asking to contact registry admins.
It is kept on push however.
Pulling manifest lists with v2 schema1 manifests will not be supported thus
there is a warning for those, but wording changed to suggest repository author
to upgrade.
Finally, a milder warning on regular pull is kept ONLY for DockerHub users
in order to incite moving away from schema1.
Signed-off-by: Tibor Vass <tibor@docker.com>
(cherry picked from commit 647dfe99a5)
Signed-off-by: Tibor Vass <tibor@docker.com>
full diffs:
- fc5a7d91d5...62a13ae87c
- b2de5d10e3...v1.0.0
- 604eaf189e...13995c7128ccc8e51e9a6bd2b551020a27180abd
notable changes in libnetwork:
- docker/libnetwork#2366 Bump vishvananda/netlink to 1.0.0
- docker/libnetwork#2339 controller: Check if IPTables is enabled for arrangeUserFilterRule
- addresses docker/libnetwork#2158 dockerd when run with --iptables=false modifies iptables by adding DOCKER-USER
- addresses moby/moby#35777 With iptables=false dockerd still creates DOCKER-USER chain and rules
- addresses docker/for-linux#136 dockerd --iptables=false adds DOCKER-USER chain and modify FORWARD chain anyway
- docker/libnetwork#2394 Make DNS records and queries case-insensitive
- addresses moby/moby#28689 Embedded DNS is case-sensitive
- addresses moby/moby#21169 hostnames with new networking are case-sensitive
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 344b093258)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
In situations where the containerd error is consumed directly
and not received over gRPC, errors were not translated.
This patch converts containerd errors to the correct HTTP
status code.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 4a516215e2)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
While investigating a test failure, I found this in the logs:
```
time="2019-07-04T15:06:32.622506760Z" level=warning msg="Error while setting daemon root propagation, this is not generally critical but may cause some functionality to not work or fallback to less desirable behavior" dir=/go/src/github.com/docker/docker/bundles/test-integration/d1285b8250308/root error="error writing file to signal mount cleanup on shutdown: open /tmp/dxr/d1285b8250308/unmount-on-shutdown: no such file or directory"
```
This path is generated from the daemon's exec-root, which appears to not
exist yet. This change just makes sure it exists before we try to write
a file.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit 7725b88edc)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Fixes#39427
This always sends the exec exit events even when the exec fails to find
the binary. A standard 127 exit status is sent in this situation.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
(cherry picked from commit c08d4da6e5)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Trying to start a container that is already running is not an
error condition, so a `304 Not Modified` should be returned instead
of a `409 Conflict`.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit c030885e7a)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The API does not filter images on platform, so searching on
Windows should work as well.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 3d1850d10d)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The `--stars` flag was deprecated, and was replaced by `--filter stars=xx`
Integration tests run with a fixed version of the CLI, and the new
(`--filter`) option is already tested in this test, so there's no need
to verify the old flags.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 85d6fb888c)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The `--stars` and `--automated` flags have been deprecated, and were
replaced by `--filter stars=xx` and `--filter is-automated=true`.
Integration tests run with a fixed version of the CLI, and the new
(`--filter`) option is already tested in this test, so there's no need
to verify the old flags.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit b38c71bfe0)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Both `--help` and `--no-trunc` are implemented in the CLI. There's
no need to verify them here because the integration tests use a
fixed version of the CLI.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit a78b9a3726)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
TestSwarmClusterRotateUnlockKey had been identified as a flaky test. It
turns out that the test code was wrong: where we should have been
checking the string output of a command, we were instead checking the
value of the error. This means that the error case we were expecting was
not being matched, and the test was failing when it should have just
retried.
Signed-off-by: Drew Erny <drew.erny@docker.com>
(cherry picked from commit b79adac339)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
kr/pty was moved to creak/pty and the old location was
archived.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 0595c01718)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
From the release notes: https://github.com/containerd/containerd/releases/tag/v1.2.7
> Welcome to the v1.2.7 release of containerd!
>
> The seventh patch release for containerd 1.2 introduces OCI image
> descriptor annotation support and contains fixes for containerd shim logs,
> container stop/deletion, cri plugin and selinux.
>
> It also contains several important bug fixes for goroutine and file
> descriptor leakage in containerd and containerd shims.
>
> Notable Updates
>
> - Support annotations in the OCI image descriptor, and filtering image by annotations. containerd/containerd#3254
> - Support context timeout in ttrpc which can help avoid containerd hangs when a shim is unresponsive. containerd/ttrpc#31
> - Fix a bug that containerd shim leaks goroutine and file descriptor after containerd restarts. containerd/ttrpc#37
> - Fix a bug that a container can't be deleted if first deletion attempt is canceled or timeout. containerd/containerd#3264
> - Fix a bug that containerd leaks file descriptor when using v2 containerd shims, e.g. containerd-shim-runc-v1. containerd/containerd#3273
> - Fix a bug that a container with lingering processes can't terminate when it shares pid namespace with another container. moby/moby#38978
> - Fix a bug that containerd can't read shim logs after restart. containerd/containerd#3282
> - Fix a bug that shim_debug option is not honored for existing containerd shims after containerd restarts. containerd/containerd#3283
> - cri: Fix a bug that a container can't be stopped when the exit event is not successfully published by the containerd shim. containerd/containerd#3125, containerd/containerd#3177
> - cri: Fix a bug that exec process is not cleaned up if grpc context is canceled or timeout. contaienrd/cri#1159
> - Fix a selinux keyring labeling issue by updating runc to v1.0.0-rc.8 and selinux library to v1.2.2. opencontainers/selinux#50
> - Update ttrpc to f82148331ad2181edea8f3f649a1f7add6c3f9c2. containerd/containerd#3316
> - Update cri to 49ca74043390bc2eeea7a45a46005fbec58a3f88. containerd/containerd#3330
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit d5669ec1c6)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Trying to link to a non-existing container is not valid, and should return an
"invalid parameter" (400) error. Returning a "not found" error in this situation
would make the client report the container's image could not be found.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 422067ba7b)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Ref: #39426
This is a common flaky test that I have seen on multiple PRs. It is not
consistent and should be skipped until it is fixed to be robust. A
simple fix for the swarm tests is not easy as they all poll and have 1
billion timeouts in all the tests so a skip is valid here.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
(cherry picked from commit b94218560e)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
When mounting overlays which have children, enforce that
the mount is always performed as read only. Newer versions
of the kernel return a device busy error when a lower directory
is in use as an upper directory in another overlay mount.
Adds committed file to indicate when an overlay is being used
as a parent, ensuring it will no longer be mounted with an
upper directory.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
(cherry picked from commit 477bf1e413)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Improve the readability of the connection error displayed to the user on
Windows when running docker commands fails by checking if the client is
privileged. If so then display the actual error wrapped in a generic
error "This error may indicate that the docker daemon is not running."
If not that display the actual error wrapped in a more specific error:
"In the default daemon configuration on Windows, the docker client must
be run with elevated privileges to connect."
Signed-off-by: Nick Adcock <nick.adcock@docker.com>
(cherry picked from commit 1a5dafb31e)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The `ioctl` interface for the `LOOP_CTL_GET_FREE` request on
`/dev/loop-control` is a little different from what `unix.IoctlGetInt`
expects: the first index is the returned status in `r1`, not an `int`
pointer as the first parameter.
Unfortunately we have to go a little lower level to get the appropriate
loop device index out, using `unix.Syscall` directly to read from
`r1`. Internally, the index is returned as a signed integer to match the
internal `ioctl` expectations of interpreting a negative signed integer
as an error at the userspace ABI boundary, so the direct interface of
`ioctlLoopCtlGetFree` can remain as-is.
[@kolyshkin: it still worked before this fix because of
/dev scan fallback in ioctlLoopCtlGetFree()]
Signed-off-by: Daniel Sweet <danieljsweet@icloud.com>
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit db2bc43017)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Windows RS1 has problems with leaking NdisAdapters during the integration
tests; the windows.ps1 script has a cleanup stesp to remove those
leaked adapters.
For internal testing at Microsoft on internal builds, this cleanup step
was skipped, and only ran on the CI machines in our Jenkins.
Due to the move to our new Jenkins, the names of Windows machines changed,
and because of that, the cleanup step was never executed, resulting in the
leaked adapters not being cleaned up:
```
20:32:23 WARNING: There are 608 NdisAdapters leaked under Psched\Parameters
20:32:23 WARNING: Not cleaning as not a production RS1 server
20:32:24 WARNING: There are 608 NdisAdapters leaked under WFPLWFS\Parameters
20:32:24 WARNING: Not cleaning as not a production RS1 server
```
```
22:01:31 WARNING: There are 1209 NdisAdapters leaked under Psched\Parameters
22:01:31 WARNING: Not cleaning as not a production RS1 server
22:01:31 WARNING: There are 1209 NdisAdapters leaked under WFPLWFS\Parameters
22:01:31 WARNING: Not cleaning as not a production RS1 server
```
This patch removes the check for non-production builds, and unconditionally
cleans up leaked adapters if they are found.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 156ad54fb7)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The "new test" detector in test-integration-flaky was a bit flaky since
it would detect function signatures that are not new tests.
In addition, the test calls `return` outside of a function which is not
allowed.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit e2b24490e4)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This was introduced in a76ff632a4:
+ find bundles -path */root/*overlay2 -prune -o -type f ( -o -name *.log -o -name *.prof ) -print
find: invalid expression; you have used a binary operator '-o' with nothing before it.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit ca1e7a3b4a)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Failing to archive the bundles should not mark the build as failed.
This can happen if a build is terminated early, or if (to be implemented)
an optional build-stage is skipped / failed;
```
2019-08-24T10:53:09.354Z] + bundleName=janky
[2019-08-24T10:53:09.354Z] + echo Creating janky-bundles.tar.gz
[2019-08-24T10:53:09.354Z] Creating janky-bundles.tar.gz
[2019-08-24T10:53:09.354Z] + xargs tar -czf janky-bundles.tar.gz
[2019-08-24T10:53:09.354Z] + find bundles -path */root/*overlay2 -prune -o -type f ( -name *-report.json -o -name *.log -o -name *.prof -o -name *-report.xml ) -print
[2019-08-24T10:53:09.354Z] find: bundles: No such file or directory
[2019-08-24T10:53:09.354Z] tar: Cowardly refusing to create an empty archive
[2019-08-24T10:53:09.354Z] Try 'tar --help' or 'tar --usage' for more information.
Error when executing always post condition:
hudson.AbortException: script returned exit code 123
at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.handleExit(DurableTaskStep.java:569)
at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.check(DurableTaskStep.java:515)
at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.run(DurableTaskStep.java:461)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit a76ff632a4)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The `libprotobuf-c0-dev` virtual package is no longer available
in Debian Buster, but is provided by `libprotobuf-c-dev`, which
is available.
https://packages.debian.org/stretch/libprotobuf-c0-dev
> Virtual Package: libprotobuf-c0-dev
>
> This is a virtual package. See the Debian policy for a definition of virtual packages.
>
> Packages providing libprotobuf-c0-dev
> libprotobuf-c-dev
> Protocol Buffers C static library and headers (protobuf-c)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit d185ca78ec)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This symlink was added in d42753485b,
to allow finding the path to the latest built binary, because at the time,
those paths were prefixed with the version or commit (e.g. `bundles/1.5.0-dev`).
Commit bac2447964 removed the version-prefix in
paths, but kept the old symlink for backward compatiblity. However, many
things were moved since then (e.g. paths were renamed to `binary-daemon`,
and various other changes). With the symlink pointing to the symlink's parent
directory, following the symlink may result into an infinite recursion,
which can happen if scripts using wildcards / globbing to find files.
With this symlink no longer serving a real purpose, we can probably safely
remove this symlink now.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit dde1fd78c7)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This will run the DCO check in a lightweight alpine container, before
running other stages, and before building the development image/container
(which can take a long time).
A Jenkins parameter was added to optionally skip the DCO check (skip_dco)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit d6f7909c76)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Using a build-arg so that we don't have to specify it for each
`apt-get install`, and to preserve that the `DEBIAN_FRONTEND` is
preserved in the image itself (which changes the default behavior,
and can be surprising if the image is run interactively).`
With this patch, some (harmless, but possibly confusing) errors
are no longer printed during build, for example:
```patch
Unpacking libgcc1:armhf (1:6.3.0-18+deb9u1) ...
Selecting previously unselected package libc6:armhf.
Preparing to unpack .../04-libc6_2.24-11+deb9u4_armhf.deb ...
-debconf: unable to initialize frontend: Dialog
-debconf: (TERM is not set, so the dialog frontend is not usable.)
-debconf: falling back to frontend: Readline
Unpacking libc6:armhf (2.24-11+deb9u4) ...
Selecting previously unselected package libgcc1:arm64.
Preparing to unpack .../05-libgcc1_1%3a6.3.0-18+deb9u1_arm64.deb ...
Unpacking libgcc1:arm64 (1:6.3.0-18+deb9u1) ...
Selecting previously unselected package libc6:arm64.
Preparing to unpack .../06-libc6_2.24-11+deb9u4_arm64.deb ...
-debconf: unable to initialize frontend: Dialog
-debconf: (TERM is not set, so the dialog frontend is not usable.)
-debconf: falling back to frontend: Readline
```
Looks like some output is now also printed on stdout instead of stderr
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 2ff9ac4de5)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This is basically taking some stuff that make a custom shell function
for.
This takes a test filter, builds the appropriate TESTFLAGS, and sets the
integration API test dirs that match the given filter to avoid building
all test dirs.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit 13064b155e)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Adds `DOCKER_BINDDIR_MOUNT_OPTS` to easily tweak the BINDDIR mount
options... primarily adding so I can control the caching mode for
osxfs because compiling takes > 1min for me with the default and < 30s
with both `cached` and `delegated`.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit b1e6536ceb)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Attempting to fix;
```
21:16:00 Traceback (most recent call last):
21:16:00 File "/usr/local/bin/yamllint", line 11, in <module>
21:16:00 sys.exit(run())
21:16:00 File "/usr/local/lib/python3.5/dist-packages/yamllint/cli.py", line 170, in run
21:16:00 problems = linter.run(f, conf, filepath)
21:16:00 File "/usr/local/lib/python3.5/dist-packages/yamllint/linter.py", line 233, in run
21:16:00 content = input.read()
21:16:00 File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
21:16:00 return codecs.ascii_decode(input, self.errors)[0]
21:16:00 UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 123522: ordinal not in range(128)
21:16:00 Build step 'Execute shell' marked build as failure
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit b5e5cac0f5)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This also makes sure that we can test all functionality of the
daemon, because some features are not available on static binaries.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 4ddb40ee8a)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
- SC2006: use $(...) notation instead of legacy backticked `...`
- SC2086: double quote to prevent globbing and word splitting
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 0b3d201892)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Jenkins groups them per stage, so collecting them for all architectures
is possible (without them conflicting or becoming ambiguous)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit e2f5b78e78)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This will send the results directly after the tests complete,
and make the stage more atomic.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 7f9328ad2e)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
In case a job fails before even generating a report file.
Signed-off-by: Andrew Hsu <andrewhsu@docker.com>
(cherry picked from commit 0cfc1ec2bd)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The z-master stage will just run the integration-cli tests. The
existing z stage will run the unit tests and the integration
tests. In this way, PR check jobs will be shorter, but all
integration tests will run after PR is merged to master.
Signed-off-by: Andrew Hsu <andrewhsu@docker.com>
(cherry picked from commit bdc1c1a02a)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The powerpc-master stage will just run the integration-cli tests. The
existing powerpc stage will run the unit tests and the integration
tests. In this way, PR check jobs will be shorter, but all integration
tests will run after PR is merged to master.
Signed-off-by: Andrew Hsu <andrewhsu@docker.com>
(cherry picked from commit c2f9d58375)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
There are many errors like this one:
> 01:39:28.750 find: ‘bundles/test-integration/dbc77018d39a5/root/overlay2/f49953a883daceee60a481dd8e1e37b0f806d309258197d6ba0f6871236d3d47/work/work’: Permission denied
(probably caused by bad permissions)
These directories are not to be looked at when we search for logs, so
let's exclude them. It's not super easy to do in find, here is some
kind of an explanation for find arguments
```
PATTERN ACTION OR PATTERN ACTION
-path X -prune -o -type f [AND] (-name A -o name B) -print
```
(here -o means OR, while AND is implicit)
While at it,
- let the find know we're only looking for files, not directories
- remove a subshell and || true
- remove `-name integration.test` (there are no such files)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit b283dff3ff)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This was inadvertedly removed in 7bfe48cc00,
because it was documented as a dependency for docker-py, but
actually used to validate the swagger file.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit b1723b3721)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
and remove `PullImageTest::test_build_invalid_platform` from the list,
which was a copy/paste error in f8cde0b32d
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 6f5c377ddc)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
All tests that require experimental either spin up a separate daemon,
or use the main daemon if experimental is enabled.
This patch
- allows enabling "experimental" for stages through an environment variable
- enables experimental by default on all stages, so that some of these tests
don't have to start a new daemon.
- removes the seaprate "experimental" stage, because it was running exactly
the same tests as the "janky" stage.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit e856b46cfb)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
These tests are fixed upstream, but those fixes are not yet in a
released version.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit f8cde0b32d)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The .git mount is only needed for the DCO check, and for building
the binaries if `DOCKER_GITCOMMIT` is not set.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 47ac8a97de)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Having this information can help debugging issues in CI (which could
be caused by missing/incorrect configuration of the machines).
We ping to a fixed version of the script, because this script is ran
directly on the host, and we don't want pull-requests modifying this
script to have direct access to the machines.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit a2ad56dfad)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Both of these tests are fairly short, and shouldn't interfer with
eachother, so we can combine them and re-use the same dev-image
(so that it'll only be built once).
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit f51c139792)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This patch removes the manual steps to resolve the Git commit, and
instead, uses the `GIT_COMMIT` that's set by Jenkins's Git plugin.
Behavior changes slightly, because `GIT_PLUGIN` contains the full
commit-sha, not the short one.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit be0e6e9d34)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Container and image names are already unique because they have
the git-sha or build-number, and a single machine won't be running
tests for multiple architectures.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 337d03a5f0)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The main Dockerfile is multi-arch now.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 61fd8b7384)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
1. Use `go list` to get list of integration dirs to build. This means we
do not need to have a valid `.go` in every subdirectory and also
filters out other dirs like "bundles" which may have been created.
2. Add option to specify custom flags for integration and
integration-cli. This is needed so both suites can be run AND set
custom flags... since the cli suite does not support standard go
flags.
3. Add options to skip an entire integration suite.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit abece9b562)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Codecov has shown to be flaky, and calculate the wrong diff, in
addition, it doesn't show coverage for integration tests, which
makes the coverage report not useful.
Removing it for now, while we look at alternatives.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit bd5c5373f1)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
When connecting with the daemon using a UNIX socket, the HTTP hostname was set, based
on the socket location, which was generating some noise in the test-logs.
Given that the actual hostname is not important (the URL just has to be well-formed),
the hostname/address can be cleaned up to reduce the noise.
This patch strips the path from the `addr`, and keeps `<random-id>.sock` as address.
Before:
daemon.go:329: [d15d31ba75501] error pinging daemon on start: Get http://%2Ftmp%2Fdocker-integration%2Fd15d31ba75501.sock/_ping: dial unix /tmp/docker-integration/d15d31ba75501.sock: connect: no such file or directory
After:
daemon.go:329: [d15d31ba75501] error pinging daemon on start: Get http://d15d31ba75501.sock/_ping: dial unix /tmp/docker-integration/d15d31ba75501.sock: connect: no such file or directory
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 92e6e7dd5f)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The test-integration/test=integration-cli directory contains
a directory for each daemon that was created during the integration
tests, which makes it a long list to browse through. In addition,
some tests spin up multiple daemons, and when debugging test-failures,
the daemon-logs often have to be looked at together.
This patch organizes the bundl directory to group daemon storage
locationos per test, making it easier to find information about
all the daemons that were used in a specific test.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 9b5e78888d)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
See if networking works if we run it first
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 6aafe0fd9e)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
When building this image docker-in-docker, the DNS in the environment
may not be usable for the build-container, causing resolution to fail:
```
02:35:31 W: Failed to fetch http://deb.debian.org/debian/dists/jessie/Release.gpg Temporary failure resolving 'deb.debian.org'
```
This patch detects if we're building from within a container, and if
so, skips creating a networking namespace for the build by using
`--network=host`.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 3c15cea650)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This removes all the installation steps for docker-py from the
Dockerfile, and instead builds the upstream Dockerfile, and runs
docker-py tests in a container.
To test;
```
make test-docker-py
...
Removing bundles/
---> Making bundle: dynbinary (in bundles/dynbinary)
Building: bundles/dynbinary-daemon/dockerd-dev
Created binary: bundles/dynbinary-daemon/dockerd-dev
---> Making bundle: test-docker-py (in bundles/test-docker-py)
---> Making bundle: .integration-daemon-start (in bundles/test-docker-py)
Using test binary docker
Starting dockerd
INFO: Waiting for daemon to start...
.
INFO: Building docker-sdk-python3:3.7.0...
sha256:686428ae28479e9b5c8fdad1cadc9b7a39b462e66bd13a7e35bd79c6a152a402
INFO: Starting docker-py tests...
============================= test session starts ==============================
platform linux -- Python 3.6.8, pytest-4.1.0, py-1.8.0, pluggy-0.9.0
rootdir: /src, inifile: pytest.ini
plugins: timeout-1.3.3, cov-2.6.1
collected 359 items
tests/integration/api_build_test.py .......s....
....
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 7bfe48cc00)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This test is failing on Windows currently:
```
11:59:47 --- FAIL: TestHealthKillContainer (8.12s)
11:59:47 health_test.go:57: assertion failed: error is not nil: Error response from daemon: Invalid signal: SIGUSR1
``
That test was added recently in https://github.com/moby/moby/pull/39454, but
rewritten in a commit in the same PR:
f8aef6a92f
In that rewrite, there were some changes:
- originally it was skipped on Windows, but the rewritten test doesn't have that skip:
```go
testRequires(c, DaemonIsLinux) // busybox doesn't work on Windows
```
- the original test used `SIGINT`, but the new one uses `SIGUSR1`
Analysis:
- The Error bubbles up from: 8e610b2b55/pkg/signal/signal.go (L29-L44)
- Interestingly; `ContainerKill` should validate if a signal is valid for the given platform, but somehow we don't hit that part; f1b5612f20/daemon/kill.go (L40-L48)
- Windows only looks to support 2 signals currently 8e610b2b55/pkg/signal/signal_windows.go (L17-L26)
- Upstream Golang looks to define `SIGINT` as well; 77f9b2728e/src/runtime/defs_windows.go (L44)
- This looks like the current list of Signals upstream in Go; 3b58ed4ad3/windows/types_windows.go (L52-L67)
```go
const (
// More invented values for signals
SIGHUP = Signal(0x1)
SIGINT = Signal(0x2)
SIGQUIT = Signal(0x3)
SIGILL = Signal(0x4)
SIGTRAP = Signal(0x5)
SIGABRT = Signal(0x6)
SIGBUS = Signal(0x7)
SIGFPE = Signal(0x8)
SIGKILL = Signal(0x9)
SIGSEGV = Signal(0xb)
SIGPIPE = Signal(0xd)
SIGALRM = Signal(0xe)
SIGTERM = Signal(0xf)
)
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit eeaa0b30d4)
Signed-off-by: Dani Louca <dani.louca@docker.com>
Docker daemon always stops healthcheck before sending signal to a
container now. However, when we use "docker kill" to send signals
other than SIGTERM or SIGKILL to a container, such as SIGINT,
daemon still stops container health check though container process
handles the signal normally and continues to work.
Signed-off-by: Ruilin Li <liruilin4@huawei.com>
(cherry picked from commit da574f9343)
Signed-off-by: Dani Louca <dani.louca@docker.com>
go1.12.8 (released 2019/08/13) includes security fixes to the net/http and net/url packages.
See the Go 1.12.8 milestone on our issue tracker for details:
https://github.com/golang/go/issues?q=milestone%3AGo1.12.8
- net/http: Denial of Service vulnerabilities in the HTTP/2 implementation
net/http and golang.org/x/net/http2 servers that accept direct connections from untrusted
clients could be remotely made to allocate an unlimited amount of memory, until the program
crashes. Servers will now close connections if the send queue accumulates too many control
messages.
The issues are CVE-2019-9512 and CVE-2019-9514, and Go issue golang.org/issue/33606.
Thanks to Jonathan Looney from Netflix for discovering and reporting these issues.
This is also fixed in version v0.0.0-20190813141303-74dc4d7220e7 of golang.org/x/net/http2.
net/url: parsing validation issue
- url.Parse would accept URLs with malformed hosts, such that the Host field could have arbitrary
suffixes that would appear in neither Hostname() nor Port(), allowing authorization bypasses
in certain applications. Note that URLs with invalid, not numeric ports will now return an error
from url.Parse.
The issue is CVE-2019-14809 and Go issue golang.org/issue/29098.
Thanks to Julian Hector and Nikolai Krein from Cure53, and Adi Cohen (adico.me) for discovering
and reporting this issue.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 73b0e4c589)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
1. Use "in-place" variables for if statements to limit their scope to
the respectful `if` block.
2. Report the error returned from sd_journal_* by using CErr().
3. Use errors.New() instead of fmt.Errorf().
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 20a0e58a79)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
From the first glance, `docker logs --tail 0` does not make sense,
as it is supposed to produce no output, but `tail -n 0` from GNU
coreutils is working like that, plus there is even a test case
(`TestLogsTail` in integration-cli/docker_cli_logs_test.go).
Now, something like `docker logs --follow --tail 0` makes total
sense, so let's make it work.
(NOTE if --tail is not used, config.Tail is set to -1)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit dd4bfe30a8)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
If we take a long time to process log messages, and during that time
journal file rotation occurs, the journald client library will keep
those rotated files open until sd_journal_process() is called.
By periodically calling sd_journal_process() during the processing
loop we shrink the window of time a client instance has open file
descriptors for rotated (deleted) journal files.
This code is modelled after that of journalctl [1]; the above explanation
as well as the value of 1024 is taken from there.
[v2: fix CErr() argument]
[1] https://github.com/systemd/systemd/blob/dc16327c48d/src/journal/journalctl.c#L2676
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit b73fb8fd5d)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
TL;DR: simplify the code, fix --follow hanging indefinitely
Do the following to simplify the followJournal() code:
1. Use Go-native select instead of C-native polling.
2. Use Watch{Producer,Consumer}Gone(), eliminating the need
to have journald.closed variable, and an extra goroutine.
3. Use sd_journal_wait(). In the words of its own man page:
> A synchronous alternative for using sd_journal_get_fd(),
> sd_journal_get_events(), sd_journal_get_timeout() and
> sd_journal_process() is sd_journal_wait().
Unfortunately, the logic is still not as simple as it
could be; the reason being, once the container has exited,
journald might still be writing some logs from its internal
buffers onto journal file(s), and there is no way to
figure out whether it's done so we are guaranteed to
read all of it back. This bug can be reproduced with
something like
> $ ID=$(docker run -d busybox seq 1 150000); docker logs --follow $ID
> ...
> 128123
> $
(The last expected output line should be `150000`).
To avoid exiting from followJournal() early, add the
following logic: once the container is gone, keep trying
to drain the journal until there's no new data for at
least `waitTimeout` time period.
Should fix https://github.com/docker/for-linux/issues/575
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit f091febc94)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
1. The journald client library initializes inotify watch(es)
during the first call to sd_journal_get_fd(), and it make sense
to open it earlier in order to not lose any journal file rotation
events.
2. It only makes sense to call this if we're going to use it
later on -- so add a check for config.Follow.
3. Remove the redundant call to sd_journal_get_fd().
NOTE that any subsequent calls to sd_journal_get_fd() return
the same file descriptor, so there's no real need to save it
for later use in wait_for_data_cancelable().
Based on earlier patch by Nalin Dahyabhai <nalin@redhat.com>.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 981c01665b)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
In case the LogConsumer is gone, the code that sends the message can
stuck forever. Wrap the code in select case, as all other loggers do.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 79039720c8)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
In case Tail=N parameter is requested, we need to show N lines.
It does not make sense to walk backwards one by one if we can
do it at once. Now, if Since=T is also provided, make sure we
haven't jumped too far (before T), and if we did, move forward.
The primary motivation for this was to make the code simpler.
This also fixes a tiny bug in the "since" implementation.
Before this commit:
> $ docker logs -t --tail=6000 --since="2019-03-10T03:54:25.00" $ID | head
> 2019-03-10T03:54:24.999821000Z 95981
After:
> $ docker logs -t --tail=6000 --since="2019-03-10T03:54:25.00" $ID | head
> 2019-03-10T03:54:25.000013000Z 95982
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit ff3cd167ea)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Clean up a deferred function call in the journal reading logic.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
(cherry picked from commit 1ada3e85bf)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
There are a few more places, apparently, that List operations against
Swarm exist, besides just in the List methods. This increases the max
received message size in those places.
Signed-off-by: Drew Erny <drew.erny@docker.com>
(cherry picked from commit a84a78e976)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
3029e765e2 broke compilation on
non-Linux/Windows systems.
This change fixes that.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit 34d5b8867f)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This is needed to provide fixes for ETW on ARM. The updated ETW package will
no-op on ARM, rather than crashing. Further changes are needed to Go itself to
allow ETW on ARM to work properly.
Signed-off-by: Kevin Parsons <kevpar@microsoft.com>
(cherry picked from commit e1f0f77bf4)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This fixes a regression introduced in 6d87f19142,
causing `COPY --from` to fail if the target directory does not exist:
```
FROM mcr.microsoft.com/windows/servercore:ltsc2019 as s1
RUN echo "Hello World" > /hello
FROM mcr.microsoft.com/windows/servercore:ltsc2019
COPY --from=s1 /hello /hello/another/world
```
Would produce an error:
```
Step 4/4 : COPY --from=s1 /hello /hello/another/world
failed to copy files: mkdir \\?: The filename, directory name, or volume label syntax is incorrect.
```
The cause for this was that Go's `os.MkdirAll()` does not support/detect volume GUID paths
(`\\?\Volume{dae8d3ac-b9a1-11e9-88eb-e8554b2ba1db}\hello\another}`), and as a result
attempted to create the volume as a directory (`\\?`), causing it to fail.
This patch replaces `os.MkdirAll()` with our own `system.MkdirAll()` function, which
is capable of detecting GUID volumes.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 5858a99267)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This function changed to the correct working directory before starting the tests
(which is the same as on Linux), however the `ProcessStartInfo` process does
not inherit this working directory, which caused Windows tests to be running
with a different working directory as Linux (causing files used in tests to not
be found).
From the documentation; https://docs.microsoft.com/en-us/dotnet/api/system.diagnostics.processstartinfo.workingdirectory?view=netframework-4.8
> When `UseShellExecute` is `true`, the fully qualified name of the directory that contains
> the process to be started. When the `UseShellExecute` property is `false`, the working
> directory for the process to be started. The default is an empty string (`""`).
This patch sets the `ProcessStartInfo.WorkingDirectory` to the correct working
directory before starting the process.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 6ae46aeabf)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The Golang base images switch to buster, which causes some breakage
in networking and packages that are no longer available; (`btrfs-tools`
is now an empty package, and `libprotobuf-c0-dev` is gone).
Some of out tests also start faiilng on stretch, and will have to be
investigated further;
```
15:13:06 --- FAIL: TestRenameAnonymousContainer (3.37s)
15:13:06 rename_test.go:168: assertion failed: 0 (int) != 1 (inspect.State.ExitCode int): container a7fe866d588d65f353f42ffc5ea5288e52700384e1d90850e9c3d4dce8657666 exited with the wrong exitcode:
15:13:38 --- FAIL: TestHostnameDnsResolution (2.23s)
15:13:38 run_linux_test.go:128: assertion failed:
15:13:38 --- ←
15:13:38 +++ →
15:13:38 @@ -1 +1,2 @@
15:13:38 +ping: bad address 'foobar'
15:13:38
15:13:38
15:13:38 run_linux_test.go:129: assertion failed: 0 (int) != 1 (res.ExitCode int)
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit ed672bb523)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This check was used to make sure we don't bump Go versions independently
(Linux/Windows). The Dockerfile switched to using a build-arg to allow
overriding the Go version, which rendered this check non-functional.
It also fails if Linux versions use a specific variant of the image;
08:41:31 ERROR: Failed 'ERROR: Mismatched GO versions between Dockerfile and Dockerfile.windows. Update your PR to ensure that both files are updated and in sync. ${GO_VERSION}-stretch ${GO_VERSION}' at 07/20/2019 08:41:31
08:41:31 At C:\gopath\src\github.com\docker\docker\hack\ci\windows.ps1:448 char:9
08:41:31 + Throw "ERROR: Mismatched GO versions between Dockerfile and D ...
08:41:31 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This patch fixes the check by looking for the value of `GO_VERSION` instead
of looking at the `FROM` line (which is harder to parse).
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 4fa57a8191)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This allows overriding the version of Go without making modifications in the
source code, which can be useful to test against multiple versions.
For example:
make GO_VERSION=1.13beta1 shell
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit c6281bc438)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The fastly cdn mirror we're using also mirrors the debian security
repository;
```
Welcome to deb.debian.org (fastly instance)!
This is deb.debian.org. This service provides mirrors for the following Debian archive repositories:
/debian/
/debian-debug/
/debian-ports/
/debian-security/
The server deb.debian.org does not have packages itself, but the name has SRV records in DNS that let apt in stretch and later find places.
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit c8f43b5f6f)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Protect access to q.quotas map, and lock around changing nextProjectID.
Techinically, the lock in findNextProjectID() is not needed as it is
only called during initialization, but one can never be too careful.
Fixes: 52897d1c09 ("projectquota: utility class for project quota controls")
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 1ac0a66a64)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Path-specific rules were removed, so this is no longer used.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 530e63c1a61b105a6f7fc143c5acb9b5cd87f958)
Signed-off-by: Tibor Vass <tibor@docker.com>
(cherry picked from commit f8a0f26843)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Commit 77b8465d7e added a secret update
endpoint to allow updating labels on existing secrets. However, when
implementing the endpoint, the DebugRequestMiddleware was not updated
to scrub the Data field (as is being done when creating a secret).
When updating a secret (to set labels), the Data field should be either
`nil` (not set), or contain the same value as the existing secret. In
situations where the Data field is set, and the `dockerd` daemon is
running with debugging enabled / log-level debug, the base64-encoded
value of the secret is printed to the daemon logs.
The docker cli does not have a `docker secret update` command, but
when using `docker stack deploy`, the docker cli sends the secret
data both when _creating_ a stack, and when _updating_ a stack, thus
leaking the secret data if the daemon runs with debug enabled:
1. Start the daemon in debug-mode
dockerd --debug
2. Initialize swarm
docker swarm init
3. Create a file containing a secret
echo secret > my_secret.txt
4. Create a docker-compose file using that secret
cat > docker-compose.yml <<'EOF'
version: "3.3"
services:
web:
image: nginx:alpine
secrets:
- my_secret
secrets:
my_secret:
file: ./my_secret.txt
EOF
5. Deploy the stack
docker stack deploy -c docker-compose.yml test
6. Verify that the secret is scrubbed in the daemon logs
DEBU[2019-07-01T22:36:08.170617400Z] Calling POST /v1.30/secrets/create
DEBU[2019-07-01T22:36:08.171364900Z] form data: {"Data":"*****","Labels":{"com.docker.stack.namespace":"test"},"Name":"test_my_secret"}
7. Re-deploy the stack to trigger an "update"
docker stack deploy -c docker-compose.yml test
8. Notice that this time, the Data field is not scrubbed, and the base64-encoded secret is logged
DEBU[2019-07-01T22:37:35.828819400Z] Calling POST /v1.30/secrets/w3hgvwpzl8yooq5ctnyp71v52/update?version=34
DEBU[2019-07-01T22:37:35.829993700Z] form data: {"Data":"c2VjcmV0Cg==","Labels":{"com.docker.stack.namespace":"test"},"Name":"test_my_secret"}
This patch modifies `maskSecretKeys` to unconditionally scrub `Data` fields.
Currently, only the `secrets` and `configs` endpoints use a field with this
name, and no other POST API endpoints use a data field, so scrubbing this
field unconditionally will only scrub requests for those endpoints.
If a new endpoint is added in future where this field should not be scrubbed,
we can re-introduce more fine-grained (path-specific) handling.
This patch introduces some change in behavior:
- In addition to secrets, requests to create or update _configs_ will
now have their `Data` field scrubbed. Generally, the actual data should
not be interesting for debugging, so likely will not be problematic.
In addition, scrubbing this data for configs may actually be desirable,
because (even though they are not explicitely designed for this purpose)
configs may contain sensitive data (credentials inside a configuration
file, e.g.).
- Requests that send key/value pairs as a "map" and that contain a
key named "data", will see the value of that field scrubbed. This
means that (e.g.) setting a `label` named `data` on a config, will
scrub/mask the value of that label.
- Note that this is already the case for any label named `jointoken`,
`password`, `secret`, `signingcakey`, or `unlockkey`.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit c7ce4be93ae8edd2da62a588e01c67313a4aba0c)
Signed-off-by: Tibor Vass <tibor@docker.com>
(cherry picked from commit 73db8c77bf)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 32d70c7e21631224674cd60021d3ec908c2d888c)
Signed-off-by: Tibor Vass <tibor@docker.com>
(cherry picked from commit ebb542b3f8)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Add tests for
- case-insensitive matching of fields
- recursive masking
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit db5f811216e70bcb4a10e477c1558d6c68f618c5)
Signed-off-by: Tibor Vass <tibor@docker.com>
(cherry picked from commit 18dac2cf32)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This will add a warning log in the daemon, and will send the message
to be displayed by the CLI.
Signed-off-by: Tibor Vass <tibor@docker.com>
(cherry picked from commit d35f8f4329)
Signed-off-by: Tibor Vass <tibor@docker.com>
This reverts commit 98fc09128b in order to
keep registry v2 schema1 handling and libtrust-key-based engine ID.
Because registry v2 schema1 was not officially deprecated and
registries are still relying on it, this patch puts its logic back.
However, registry v1 relics are not added back since v1 logic has been
removed a while ago.
This also fixes an engine upgrade issue in a swarm cluster. It was relying
on the Engine ID to be the same upon upgrade, but the mentioned commit
modified the logic to use UUID and from a different file.
Since the libtrust key is always needed to support v2 schema1 pushes,
that the old engine ID is based on the libtrust key, and that the engine ID
needs to be conserved across upgrades, adding a UUID-based engine ID logic
seems to add more complexity than it solves the problems.
Hence reverting the engine ID changes as well.
Signed-off-by: Tibor Vass <tibor@docker.com>
(cherry picked from commit f695e98cb7)
Signed-off-by: Tibor Vass <tibor@docker.com>
Previously, getWalkRoot("/", "foo") would return "//foo"
Now it returns "/foo"
Signed-off-by: Tibor Vass <tibor@docker.com>
(cherry picked from commit 7410f1a859)
Signed-off-by: Tibor Vass <tibor@docker.com>
Before 7a7357da, archive.TarResourceRebase was being used to copy files
and folders from the container. That function splits the source path
into a dirname + basename pair to support copying a file:
if you wanted to tar `dir/file` it would tar from `dir` the file `file`
(as part of the IncludedFiles option).
However, that path splitting logic was kept for folders as well, which
resulted in weird inputs to archive.TarWithOptions:
if you wanted to tar `dir1/dir2` it would tar from `dir1` the directory
`dir2` (as part of IncludedFiles option).
Although it was weird, it worked fine until we started chrooting into
the container rootfs when doing a `docker cp` with container source set
to `/` (cf 3029e765).
The fix is to only do the path splitting logic if the source is a file.
Unfortunately, 7a7357da added support for LCOW by duplicating some of
this subtle logic. Ideally we would need to do more refactoring of the
archive codebase to properly encapsulate these behaviors behind well-
documented APIs.
This fix does not do that. Instead, it fixes the issue inline.
Signed-off-by: Tibor Vass <tibor@docker.com>
(cherry picked from commit 171538c190)
Signed-off-by: Tibor Vass <tibor@docker.com>
This test runs on a daemon also used by other tests
so make sure we don't get failures if another test
doesn't cleanup or is running in parallel.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 915acffdb4)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This adds some logs, handles timers better, and sets a request timeout
for the ping request.
I'm not sure the ticker in that loop is what we really want since the
ticker keeps ticking while we are (attempting) to make a request... but
I opted to not change that for now.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit 20ea8942b8)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Moby currently sorts uid and gid ranges in id maps. This causes subuid
and subgid files to be interpreted wrongly.
The subuid file
```
> cat /etc/subuid
jonas:100000:1000
jonas:1000:1
```
configures that the container uids 0-999 are mapped to the host uids
100000-100999 and uid 1000 in the container is mapped to uid 1000 on the
host. The expected uid_map is:
```
> docker run ubuntu cat /proc/self/uid_map
0 100000 1000
1000 1000 1
```
Moby currently sorts the ranges by the first id in the range. Therefore
with the subuid file above the uid 0 in the container is mapped to uid
100000 on host and the uids 1-1000 in container are mapped to the uids
1-1000 on the host. The resulting uid_map is:
```
> docker run ubuntu cat /proc/self/uid_map
0 1000 1
1 100000 1000
```
The ordering was implemented to work around a limitation in Linux 3.8.
This is fixed since Linux 3.9 as stated on the user namespaces manpage
[1]:
> In the initial implementation (Linux 3.8), this requirement was
> satisfied by a simplistic implementation that imposed the further
> requirement that the values in both field 1 and field 2 of successive
> lines must be in ascending numerical order, which prevented some
> otherwise valid maps from being created. Linux 3.9 and later fix this
> limitation, allowing any valid set of nonoverlapping maps.
This fix changes the interpretation of subuid and subgid files which do
not have the ids of in the numerical order for each individual user.
This breaks users that rely on the current behaviour.
The desired mapping above - map low user ids in the container to high
user ids on the host and some higher user ids in the container to lower
user on host - can unfortunately not archived with the current
behaviour.
[1] http://man7.org/linux/man-pages/man7/user_namespaces.7.html
Signed-off-by: Jonas Dohse <jonas@dohse.ch>
(cherry picked from commit c4628d79d2)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Running a bundled aufs benchmark sometimes results in this warning:
> WARN[0001] Couldn't run auplink before unmount /tmp/aufs-tests/aufs/mnt/XXXXX error="exit status 22" storage-driver=aufs
If we take a look at what aulink utility produces on stderr, we'll see:
> auplink:proc_mnt.c:96: /tmp/aufs-tests/aufs/mnt/XXXXX: Invalid argument
and auplink exits with exit code of 22 (EINVAL).
Looking into auplink source code, what happens is it tries to find a
record in /proc/self/mounts corresponding to the mount point (by using
setmntent()/getmntent_r() glibc functions), and it fails.
Some manual testing, as well as runtime testing with lots of printf
added on mount/unmount, as well as calls to check the superblock fs
magic on mount point (as in graphdriver.Mounted(graphdriver.FsMagicAufs, target)
confirmed that this record is in fact there, but sometimes auplink
can't find it. I was also able to reproduce the same error (inability
to find a mount in /proc/self/mounts that should definitely be there)
using a small C program, mocking what `auplink` does:
```c
#include <stdio.h>
#include <err.h>
#include <mntent.h>
#include <string.h>
#include <stdlib.h>
int main(int argc, char **argv)
{
FILE *fp;
struct mntent m, *p;
char a[4096];
char buf[4096 + 1024];
int found =0, lines = 0;
if (argc != 2) {
fprintf(stderr, "Usage: %s <mountpoint>\n", argv[0]);
exit(1);
}
fp = setmntent("/proc/self/mounts", "r");
if (!fp) {
err(1, "setmntent");
}
setvbuf(fp, a, _IOLBF, sizeof(a));
while ((p = getmntent_r(fp, &m, buf, sizeof(buf)))) {
lines++;
if (!strcmp(p->mnt_dir, argv[1])) {
found++;
}
}
printf("found %d entries for %s (%d lines seen)\n", found, argv[1], lines);
return !found;
}
```
I have also wrote a few other C proggies -- one that reads
/proc/self/mounts directly, one that reads /proc/self/mountinfo instead.
They are also prone to the same occasional error.
It is not perfectly clear why this happens, but so far my best theory
is when a lot of mounts/unmounts happen in parallel with reading
contents of /proc/self/mounts, sometimes the kernel fails to provide
continuity (i.e. it skips some part of file or mixes it up in some
other way). In other words, this is a kernel bug (which is probably
hard to fix unless some other interface to get a mount entry is added).
Now, there is no real fix, and a workaround I was able to come up
with is to retry when we got EINVAL. It usually works on the second
attempt, although I've once seen it took two attempts to go through.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit ae431b10a9)
Do not use filepath.Walk() as there's no requirement to recursively
go into every directory under mnt -- a (non-recursive) list of
directories in mnt is sufficient.
With filepath.Walk(), in case some container will fail to unmount,
it'll go through the whole container filesystem which is both
excessive and useless.
This is similar to commit f1a4592297 ("devmapper.shutdown:
optimize")
While at it, raise the priority of "unmount error" message from debug
to a warning. Note we don't have to explicitly add `m` as unmount error (from
pkg/mount) will have it.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 8fda12c607)
In case there are a big number of layers, so that mount data won't fit
into a single memory page (4096 bytes on most platforms, which is good
enough for about 40 layers, depending on how long graphdriver root path
is), we supply additional layers with O_REMOUNT, as described in aufs
documentation.
Problem is, the current implementation does that one layer at a time
(i.e. there is one mount syscall per each additional layer).
Optimize the code to supply as many layers as we can fit in one page
(basically reusing the same code as for the original mount).
Note, per aufs docs, "[a]t remount-time, the options are interpreted
in the given order, e.g. left to right" so we should be good.
Tested on an image with ~100 layers.
Before (35 syscalls):
> [pid 22756] 1556919088.686955 mount("none", "/mnt/volume_sfo2_09/docker-aufs/aufs/mnt/a86f8c9dd0ec2486293119c20b0ec026e19bbc4d51332c554f7cf05d777c9866", "aufs", 0, "br:/mnt/volume_sfo2_09/docker-au"...) = 0 <0.000504>
> [pid 22756] 1556919088.687643 mount("none", "/mnt/volume_sfo2_09/docker-aufs/aufs/mnt/a86f8c9dd0ec2486293119c20b0ec026e19bbc4d51332c554f7cf05d777c9866", 0xc000c451b0, MS_REMOUNT, "append:/mnt/volume_sfo2_09/docke"...) = 0 <0.000105>
> [pid 22756] 1556919088.687851 mount("none", "/mnt/volume_sfo2_09/docker-aufs/aufs/mnt/a86f8c9dd0ec2486293119c20b0ec026e19bbc4d51332c554f7cf05d777c9866", 0xc000c451ba, MS_REMOUNT, "append:/mnt/volume_sfo2_09/docke"...) = 0 <0.000098>
> ..... (~30 lines skipped for clarity)
> [pid 22756] 1556919088.696182 mount("none", "/mnt/volume_sfo2_09/docker-aufs/aufs/mnt/a86f8c9dd0ec2486293119c20b0ec026e19bbc4d51332c554f7cf05d777c9866", 0xc000c45310, MS_REMOUNT, "append:/mnt/volume_sfo2_09/docke"...) = 0 <0.000266>
After (2 syscalls):
> [pid 24352] 1556919361.799889 mount("none", "/mnt/volume_sfo2_09/docker-aufs/aufs/mnt/8e7ba189e347a834e99eea4ed568f95b86cec809c227516afdc7c70286ff9a20", "aufs", 0, "br:/mnt/volume_sfo2_09/docker-au"...) = 0 <0.001717>
> [pid 24352] 1556919361.801761 mount("none", "/mnt/volume_sfo2_09/docker-aufs/aufs/mnt/8e7ba189e347a834e99eea4ed568f95b86cec809c227516afdc7c70286ff9a20", 0xc000dbecb0, MS_REMOUNT, "append:/mnt/volume_sfo2_09/docke"...) = 0 <0.001358>
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit d58c434bff)
Apparently there is some kind of race in aufs kernel module code,
which leads to the errors like:
[98221.158606] aufs au_xino_create2:186:dockerd[25801]: aufs.xino create err -17
[98221.162128] aufs au_xino_set:1229:dockerd[25801]: I/O Error, failed creating xino(-17).
[98362.239085] aufs au_xino_create2:186:dockerd[6348]: aufs.xino create err -17
[98362.243860] aufs au_xino_set:1229:dockerd[6348]: I/O Error, failed creating xino(-17).
[98373.775380] aufs au_xino_create:767:dockerd[27435]: open /dev/shm/aufs.xino(-17)
[98389.015640] aufs au_xino_create2:186:dockerd[26753]: aufs.xino create err -17
[98389.018776] aufs au_xino_set:1229:dockerd[26753]: I/O Error, failed creating xino(-17).
[98424.117584] aufs au_xino_create:767:dockerd[27105]: open /dev/shm/aufs.xino(-17)
So, we have to have a lock around mount syscall.
While at it, don't call the whole Unmount() on an error path, as
it leads to bogus error from auplink flush.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 5cd62852fa)
1. Use mount.Unmount() which ignores EINVAL ("not mounted") error,
and provides better error diagnostics (so we don't have to explicitly
add target to error messages).
2. Since we're ignoring "not mounted" error, we can call
multiple unmounts without any locking -- but since "auplink flush"
is still involved and can produce an error in logs, let's keep
the check for fs being mounted (it's just a statfs so should be fast).
2. While at it, improve the "can't unmount" error message in Put().
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 4beee98026)
Both mount and unmount calls are already protected by fine-grained
(per id) locks in Get()/Put() introduced in commit fc1cf1911b
("Add more locking to storage drivers"), so there's no point in
having a global lock in mount/unmount.
The only place from which unmount is called without any locking
is Cleanup() -- this is to be addressed in the next patch.
This reverts commit 824c24e680.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit f93750b2c4)
Increases the max recieved gRPC message size for Node and Secret list
operations. This has already been done for the other swarm types, but
was not done for these.
Signed-off-by: Drew Erny <drew.erny@docker.com>
(cherry picked from commit a0903e1fa3)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Previously `docker info` had reported "cgroupfs" as the cgroup driver
but the driver wasn't actually used at all.
This PR reports "none" as the cgroup driver so as to avoid confusion.
e.g. kubeadm/kubelet will detect cgroupless-ness by checking this docker
info field. https://github.com/rootless-containers/usernetes/pull/97
Note that user still cannot specify `native.cgroupdriver=none` manually.
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
(cherry picked from commit 153466ba0a)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Previously only unpack operations were supported with chroot.
This adds chroot support for packing operations.
This prevents potential breakouts when copying data from a container.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit 3029e765e2)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This is useful for preventing CVE-2018-15664 where a malicious container
process can take advantage of a race on symlink resolution/sanitization.
Before this change chrootarchive would chroot to the destination
directory which is attacker controlled. With this patch we always chroot
to the container's root which is not attacker controlled.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit d089b63937)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Reverts the change to swarmkit that made all updates set UpdateStatus to
Completed
Signed-off-by: Drew Erny <drew.erny@docker.com>
(cherry picked from commit c7d9599e3d)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
As pointed out by Tonis, there's a race between ReleaseRWLayer()
and GetRWLayer():
```
----- goroutine 1 ----- ----- goroutine 2 -----
ReleaseRWLayer()
m := ls.mounts[l.Name()]
...
m.deleteReference(l)
m.hasReferences()
... GetRWLayer()
... mount := ls.mounts[id]
ls.driver.Remove(m.mountID)
ls.store.RemoveMount(m.name) return mount.getReference()
delete(ls.mounts, m.Name())
----------------------- -----------------------
```
When something like this happens, GetRWLayer will return
an RWLayer without a storage. Oops.
There might be more races like this, and it seems the best
solution is to lock by layer id/name by using pkg/locker.
With this in place, name collision could not happen, so remove
the part of previous commit that protected against it in
CreateRWLayer (temporary nil assigmment and associated rollback).
So, now we have
* layerStore.mountL sync.Mutex to protect layerStore.mount map[]
(against concurrent access);
* mountedLayer's embedded `sync.Mutex` to protect its references map[];
* layerStore.layerL (which I haven't touched);
* per-id locker, to avoid name conflicts and concurrent operations
on the same rw layer.
The whole rig seems to look more readable now (mutexes use is
straightforward, no nested locks).
Reported-by: Tonis Tiigi <tonistiigi@gmail.com>
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit af433dd200)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This is an additon to commit 1fea38856a ("Remove v1.10 migrator")
aka PR #38265. Since that one, CreateRWLayerByGraphID() is not
used anywhere, so let's drop it.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit b4e9b50765)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Goroutine stack analisys shown some lock contention
while doing massively (100 instances of `docker rm`)
parallel image removal, with many goroutines waiting
for the mountL mutex. Optimize it.
With this commit, the above operation is about 3x
faster, with no noticeable change to container
creation times (tested on aufs and overlay2).
kolyshkin@:
- squashed commits
- added description
- protected CreateRWLayer against name collisions by
temporary assiging nil to ls.mounts[name], and treating
nil as "non-existent" in all the other functions.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 05250a4f00)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Add a mutex to protect concurrent access to mountedLayer.references map.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit f73b5cb4e8)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Currently the API spec would allow `"443/tcp": [null]`, but what should
be allowed is `"443/tcp": null`
Signed-off-by: Dominic Tubach <dominic.tubach@to.com>
(cherry picked from commit 32b5d296ea)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Make sure adapter.removeNetworks executes during task Remove
adapter.removeNetworks was being skipped for cases when
isUnknownContainer(err) was true after adapter.remove was executed
This fix eliminates the nil return case forcing the function
to continue executing unless there is a true error
Fixes https://github.com/moby/moby/issues/39225
Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>
(cherry picked from commit 70fa7b6a3f)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This test is dependent on the search results returned by Docker Hub, which
can change at any moment, and causes this test to be unpredictable.
Removing this test instead of trying to catch up with Docker Hub any time
the results change, because it's effectively testing Docker Hub, and not
the daemon.
Unit tests are already in place to test the core functionality of the daemon,
so it should be safe to remove this test.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 21e662c774)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Errors were being ignored and always telling the user that the path
doesn't exist even if it was some other problem, such as a permission
error.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit ebcef28834)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
`chmod` is a legacy syscall, and not present on arm64, which
caused this test to fail.
Add `fchmodat` to the profile so that this test can run both
on x64 and arm64.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 4bd8964b23)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Sometimes this test fails (allegedly due to problems with Docker Hub),
but it fails later than it should, for example:
> 01:20:34.845 assertion failed: expression is false: strings.Count(outSearchCmdStars, "[OK]") <= strings.Count(outSearchCmd, "[OK]"): The quantity of images with stars should be less than that of all images: <...>
This, with non-empty list of images following, means that the initial
`docker search busybox` command returned not enough results. So, add
a check that `docker search busybox` returns something.
While at it,
* raise the number of stars to 10;
* simplify check for number of lines (no need to count [OK]'s);
* improve error message.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 4f80a1953d)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Today `$ docker service create --limit-cpu` configures a containers
`CpuPeriod` and `CpuQuota` variables, this commit switches this to
configure a containers `NanoCpu` variable instead.
Signed-off-by: Olly Pomeroy <olly@docker.com>
(cherry picked from commit 8a60a1e14a)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Killing the shell script process does not kill the forked process.
This commit switches to `exec` so that the executed process can be
easily killed.
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
(cherry picked from commit 34cc5c24d0)
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
no local changes, just syncing with containerd
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit d6d2b30fd2)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
As of Go 1.12, the `testing` package panics if a goroutine logs to a
`testing.T` after the relevant test has completed. This was not
documented as a change at all; see the commit
95d06ab6c982f58b127b14a52c3325acf0bd3926 in the Go repository for the
relevant change.
At any point in the integration tests, tests could panic with the
message "Log in goroutine after TEST_FUNCTION has completed". This was
exacerbated by less direct logging I/O, e.g. running `make test` with
its output piped instead of attached to a TTY.
The most common cause of panics was that there was a race condition
between an exit logging goroutine and the `StopWithError` method:
`StopWithError` could return, causing the calling test method to return,
causing the `testing.T` to be marked as finished, before the goroutine
could log that the test daemon had exited. The fix is simple: capture
the result of `cmd.Wait()`, _then_ log, _then_ send the captured
result over the `Wait` channel. This ensures that the message is
logged before `StopWithError` can return, blocking the test method
so that the target `testing.T` is not marked as finished.
Signed-off-by: Daniel Sweet <danieljsweet@icloud.com>
(cherry picked from commit 7546322e99)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
make sure the LB sandbox is removed when a service is updated
with a --network-rm option
Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>
(cherry picked from commit 680d0ba4ab)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The `--rootless` flag had a couple of issues:
* #38702: euid=0, $USER="root" but no access to cgroup ("rootful" Docker in rootless Docker)
* #39009: euid=0 but $USER="docker" (rootful boot2docker)
To fix#38702, XDG dirs are ignored as in rootful Docker, unless the
dockerd is directly running under RootlessKit namespaces.
RootlessKit detection is implemented by checking whether `$ROOTLESSKIT_STATE_DIR` is set.
To fix#39009, the non-robust `$USER` check is now completely removed.
The entire logic can be illustrated as follows:
```
withRootlessKit := getenv("ROOTLESSKIT_STATE_DIR")
rootlessMode := withRootlessKit || cliFlag("--rootless")
honorXDG := withRootlessKit
useRootlessKitDockerProxy := withRootlessKit
removeCgroupSpec := rootlessMode
adjustOOMScoreAdj := rootlessMode
```
Close#39024Fix#38702#39009
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
(cherry picked from commit 3518383ed9)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
lxc-user-nic can eliminate slirp overhead but needs /etc/lxc/lxc-usernet to be configured for the current user.
To use lxc-user-nic, $DOCKERD_ROOTLESS_ROOTLESSKIT_NET=lxc-user-nic also needs to be set.
This commit also bumps up RootlessKit from v0.3.0 to v0.4.0:
70e0502f32...e92d5e772e
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
(cherry picked from commit 63a66b0eb0)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
1. There is no need to persist DOCKER_GITCOMMIT,
as it's not needed for runtime, only for build.
So, remove ENV.
2. In case $GITCOMMIT is not defined during build time
(and it happens if .git directory is not present),
we still need to have some value set, so set it to
`undefined`. Otherwise we'll have something like
> => ERROR [builder 2/3] RUN hack/make.sh build-integration-test-binary
> ------
> > [builder 2/3] RUN hack/make.sh build-integration-test-binary:
> #32 0.488
> #32 0.505 error: .git directory missing and DOCKER_GITCOMMIT not specified
> #32 0.505 Please either build with the .git directory accessible, or specify the
> #32 0.505 exact (--short) commit hash you are building using DOCKER_GITCOMMIT for
> #32 0.505 future accountability in diagnosing build issues. Thanks!
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit c3b24944ca)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Package "gotest.tools/assert" uses source introspection to
print more info in case of assertion failure. When source code
is not available, it prints an error instead.
In other words, before this commit:
> --- SKIP: TestCgroupDriverSystemdMemoryLimit (0.00s)
> cgroupdriver_systemd_test.go:32: failed to parse source file: /go/src/github.com/docker/docker/integration/system/cgroupdriver_systemd_test.go: open /go/src/github.com/docker/docker/integration/system/cgroupdriver_systemd_test.go: no such file or directory
> cgroupdriver_systemd_test.go:32:
and after:
> --- SKIP: TestCgroupDriverSystemdMemoryLimit (0.09s)
> cgroupdriver_systemd_test.go:32: !hasSystemd()
This increases the resulting image size by about 2 MB
on my system (from 758 to 760 MB).
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 0deb18ab42)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This test case requires not just daemon >= 1.40, but also
client API >= 1.40. In case older client is used, we'll
get failure from the very first check:
> ipcmode_linux_test.go:313: assertion failed: shareable (string) != private (string)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 1ada1c8391)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Older versions did not use an UUID as ID
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 05bd9958f2)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The daemon may already have other volumes, so filter out those
when running the test.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 566eea13e6)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
TestNISDomainname in the integration suite covers this
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 2b5880c2eb)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Older versions of the daemon would concatenate hostname and
domainname, so hostname "foobar" and domainname "baz.cyphar.com"
would produce `foobar.baz.cyphar.com` as hostname.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit c91c3776ea)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The Dockerfile missed some fixtures, which caused this test
fail when running from this image.
I also noticed some other fixtures missing in integration-cli,
where the image had symlinks to some certificates, but the
original files were not included;
```
|-- integration-cli
|-- fixtures
| |-- auth
| | `-- docker-credential-shell-test
| |-- credentialspecs
| | `-- valid.json
| |-- https
| | |-- ca.pem -> ../../../integration/testdata/https/ca.pem
| | |-- client-cert.pem -> ../../../integration/testdata/https/client-cert.pem
| | |-- client-key.pem -> ../../../integration/testdata/https/client-key.pem
| | |-- client-rogue-cert.pem
| | |-- client-rogue-key.pem
| | |-- server-cert.pem -> ../../../integration/testdata/https/server-cert.pem
| | |-- server-key.pem -> ../../../integration/testdata/https/server-key.pem
| | |-- server-rogue-cert.pem
| | `-- server-rogue-key.pem
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 48fd0e921c)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
When we see an `ECONNREFUSED` (or equivalent) from an attempted `HEAD` on the
`/_ping` endpoint there is no point in trying again with `GET` since the server
is not responding/available at all.
Once vendored into the cli this will partially mitigate https://github.com/docker/cli/issues/1739
("Docker commands take 1 minute to timeout if context endpoint is unreachable")
by cutting the effective timeout in half.
Signed-off-by: Ian Campbell <ijc@docker.com>
(cherry picked from commit 8c8457b0f2)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Checks for environment variable VERSION if it exists then it sets dockerVersion to VERSION
Signed-off-by: corbin-coleman <corbin.coleman@docker.com>
Signed-off-by: Eli Uriegas <eli.uriegas@docker.com>
This cross-compiling other binaries simpler.
It would be nice if the cross Makefile target built all the required
bins, but at least this is a first step.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
No affiliation change (NTT).
The former email address will continue to be available for the time
being.
For daily communication, I still prefer to use my gmail.com address.
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
Not exactly sure why, but this line;
build: DOCKER_CROSS ?= false
Always overwrote `DOCKER_CROSS` when running `make cross`.
Perhaps because it is set in `cross: DOCKER_CROSS := true`,
and in a different scope? May also be dependent on the
version of `make` in use.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
full diff: 18e7e58ea1...59163bf75d
- Add missing return when configuring VXLAN port
- Prevent possible panic in cnmallocator.IsAttachmentAllocated()
- update github.com/pivotal-golang/clock
- new name for package: code.cloudfoundry.org/clock
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
full diff: 2c00daeb6c...d98352740c
relevant changes:
- syndtr/gocapability#11 Add support for ambient capabilities
- syndtr/gocapability#13 Fix issue #12: break too early
- syndtr/gocapability#16 Fix capHeader.pid type
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Katherine Louise Bouman is an imaging scientist and Assistant Professor
of Computer Science at the California Institute of Technology. She
researches computational methods for imaging, and developed an algorithm
that made possible the picture first visualization of a black hole
using the Event Horizon Telescope. - https://en.wikipedia.org/wiki/Katie_Bouman
Thank you for being amazing!
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
WithAPIVersionNegotiation enables automatic API version negotiation for the client.
With this option enabled, the client automatically negotiates the API version
to use when making requests. API version negotiation is performed on the first
request; subsequent requests will not re-negotiate.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Pretty much cross-compile doesn't work because of this:
> profiles/seccomp/seccomp.go:13:2: build constraints exclude all Go files in /go/src/github.com/docker/docker/vendor/github.com/seccomp/libseccomp-golang
This changes adds a new Dockerfile target for cross compilation with the
neccesary arch specific libseccomp packages and CC toolchains.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
commit 3d72963ab8 fixed
situations where a version negotiation could override
the version, even though a client was initialized with a
fixed version.
In situations where the "fixed" version is empty, we
should ignore the option, and treat the client as
"not having a fixed version", so that API version
negotiation can still be performed.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Minor improvement, but makes defining a list of options
a bit cleaner, and more descriptive;
Before:
opts := make([]func(*client.Client) error, 0)
After:
opts := make([]client.Opt, 0)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Add Linux build support;
Increase compatibility as the module will appear empty and cause an error on Linux.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
`defer r.WaitTimeout(10s)` was in a wrong place and had caused the
daemon to hang for 10 seconds.
Fix#39025
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
These tests have bad output and its hard to figure out what went wrong
when one of them fails. Move them to the end to atleast get the real
test output first and better debug things.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Full diff: https://github.com/sirupsen/logrus/compare/v1.3.0...v1.4.1
Fixes:
- Remove dependency on golang.org/x/crypto
- Fix wrong method calls Logger.Print and Logger.Warningln
- Update Entry.Logf to not do string formatting unless the log level is enabled
- Fix infinite recursion on unknown Level.String()
- Fix race condition in getCaller
- Fix Entry.WithContext method to return a copy of the initial entry
New:
- Add DeferExitHandler, similar to RegisterExitHandler but prepending the handler to the list of handlers (semantically like defer)
- Add CallerPrettyfier to JSONFormatter and `TextFormatter`
- Add Entry.WithContext() and Entry.Context, to set a context on entries to be used e.g. in hooks
- Enhance TextFormatter to not print caller information when they are empty
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
* Use rc_ulimit for ulimit constraints
* Synchronize ulimit settings to systemd's
* Add support for reload command
* Add support for retry settings for docker stop/restart
Signed-off-by: Manuel Rüger <manuel@rueg.eu>
`/proc/self/mountinfo` uses `\040` for spaces, however, `parseInfoFile()`
did not decode those spaces in paths, therefore attempting to use `\040`
as a literal part of the path.
This patch un-quotes the `root` and `mount point` fields to fix
situations where paths contain spaces.
Note that the `mount source` field is not modified, given that
this field is documented (man `PROC(5)`) as:
filesystem-specific information or "none"
Which I interpreted as "the format in this field is undefined".
Reported-by: Daniil Yaroslavtsev <daniilyar@users.noreply.github.com>
Reported-by: Nathan Ringo <remexre@gmail.com>
Based-on-patch-by: Diego Becciolini <itizir@users.noreply.github.com>
Based-on-patch-by: Sergei Utinski <sergei-utinski@users.noreply.github.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This PR is a small gofmt fix of https://goreportcard.com/report/github.com/moby/moby#gofmt
```
gofmt99%
Gofmt formats Go programs. We run gofmt -s on your code, where -s is for the "simplify" command
moby/integration-cli/docker_cli_run_test.go
Line 1: warning: file is not gofmted with -s (gofmt)
```
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
The `ErrAlreadyExists` error is used for 304 statuses, which
is not an error-condition, so should probably not be defined
as part of the errdefs package.
This patch removes the `ErrAlreadyExists` interface, and related
helpers, as it was currently not used.
Note that a 304 status can fulfil certain use-cases, but (refering
to https://www.codetinkerer.com/2015/12/04/choosing-an-http-status-code.html)
could probably be handled by a 200 OK, unless we want to perform
caching in the client.
If we do want to use 304 statuses, perhaps we need a separate class
of "errors" for this (?).
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
* Add new adjectives to the names generator
Signed-off-by: sh7dm <d3dx12.xx@gmail.com>
* Add some more adjectives to the names generator
Signed-off-by: sh7dm <d3dx12.xx@gmail.com>
Signed-off-by: John Howard <jhoward@microsoft.com>
Some permissions corrections here. Also needs re-vendor of go-winio.
- Create the layer folder directory as standard, not with SDDL. It will inherit permissions from the data-root correctly.
- Apply the VM Group SID access to layer.vhd
Permissions after this changes
Data root:
```
PS C:\> icacls test
test BUILTIN\Administrators:(OI)(CI)(F)
NT AUTHORITY\SYSTEM:(OI)(CI)(F)
```
lcow subdirectory under dataroot
```
PS C:\> icacls test\lcow
test\lcow BUILTIN\Administrators:(I)(OI)(CI)(F)
NT AUTHORITY\SYSTEM:(I)(OI)(CI)(F)
```
layer.vhd in a layer folder for LCOW
```
.\test\lcow\c33923d21c9621fea2f990a8778f469ecdbdc57fd9ca682565d1fa86fadd5d95\layer.vhd NT VIRTUAL MACHINE\Virtual Machines:(R)
BUILTIN\Administrators:(I)(F)
NT AUTHORITY\SYSTEM:(I)(F)
```
And showing working
```
PS C:\> docker-ci-zap -folder=c:\test
INFO: Zapped successfully
PS C:\> docker run --rm alpine echo hello
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
8e402f1a9c57: Pull complete
Digest: sha256:644fcb1a676b5165371437feaa922943aaf7afcfa8bfee4472f6860aad1ef2a0
Status: Downloaded newer image for alpine:latest
hello
```
Now `docker run -p` ports can be exposed to the host namespace automatically when `dockerd-rootless.sh` is launched with
`--userland-proxy --userland-proxy-path $(which rootlesskit-docker-proxy)`.
This is akin to how Docker for Mac/Win works with `--userland-proxy-path=/path/to/vpnkit-expose-port`.
The port number on the host namespace needs to be set to >= 1024.
SCTP ports are currently unsupported.
RootlessKit changes: 7bbbc48a6f...ed26714429
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
In this case the message to stop the agent is never actually sent
because the swarm node is nil
Signed-off-by: Kyle Wuolle <kyle.wuolle@gmail.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
GCR does not currently support 401 response from blob endpoints.
This detects the case where no manifest requests have been
performed for the current resolver and does a dummy request
to enable authorization.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
io_pgetevents() is a new Linux system call. It is similar to io_getevents()
that is already whitelisted, and adds no special abilities over that system call.
Allow that system call to enable applications that use it.
Fixes#38894.
Signed-off-by: Avi Kivity <avi@scylladb.com>
This patch hard-codes support for NVIDIA GPUs.
In a future patch it should move out into its own Device Plugin.
Signed-off-by: Tibor Vass <tibor@docker.com>
Unlike a plain `net/http/client.Do()`, requests made through client/request
use the `sendRequest` function, which parses the server response, and may
convert non-transport errors into errors (through `cli.checkResponseErr()`).
This means that we cannot assume that no reader was opened if an error is
returned.
This patch changes various locations where `ensureReaderClosed` was only
called in the non-error situation, and uses a `defer` to make sure it's
always called.
`ensureReaderClosed` itself already checks if the response's body was set,
so in situations where the error was due to a transport error, calling
`ensureReaderClosed` should be a no-op.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Running a cluster in a two-manager configuration effectively *doubles*
the chance of loosing control over the cluster (compared to running
in a single-manager setup). Users may have the assumption that having
two managers provides fault tolerance, so it's best to warn them if
they're using this configuration.
This patch adds a warning to the `info` response if Swarm is configured
with two managers:
WARNING: Running Swarm in a two-manager configuration. This configuration provides
no fault tolerance, and poses a high risk to loose control over the cluster.
Refer to https://docs.docker.com/engine/swarm/admin_guide/ to configure the
Swarm for fault-tolerance.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
append(newRoot.DiffIDs) without element does nothing,
so it's probably not what was intended. Changed code
to perform a slice copying instead.
Fixes#38834.
Signed-off-by: Iskander Sharipov <quasilyte@gmail.com>
Instead of having to go through files or registry values as is currently the
case.
While adding GMSA support to Kubernetes (https://github.com/kubernetes/kubernetes/pull/73726)
I stumbled upon the fact that Docker currently only allows passing Windows
credential specs through files or registry values, forcing the Kubelet
to perform a rather awkward dance of writing-then-deleting to either the
disk or the registry to be able to create a Windows container with cred
specs.
This patch solves this problem by making it possible to directly pass
whole base64-encoded cred specs to the engine's API. I took the opportunity
to slightly refactor the method responsible for Windows cred spec as it
seemed hard to read to me.
Added some unit tests on Windows credential specs handling, as there were
previously none.
Added/amended the relevant integration tests.
I have also tested it manually: given a Windows container using a cred spec
that you would normally start with e.g.
```powershell
docker run --rm --security-opt "credentialspec=file://win.json" mcr.microsoft.com/windows/servercore:ltsc2019 nltest /parentdomain
# output:
# my.ad.domain.com. (1)
# The command completed successfully
```
can now equivalently be started with
```powershell
$rawCredSpec = & cat 'C:\ProgramData\docker\credentialspecs\win.json'
$escaped = $rawCredSpec.Replace('"', '\"')
docker run --rm --security-opt "credentialspec=raw://$escaped" mcr.microsoft.com/windows/servercore:ltsc2019 nltest /parentdomain
# same output!
```
I'll do another PR on Swarmkit after this is merged to allow services to use
the same option.
(It's worth noting that @dperny faced the same problem adding GMSA support
to Swarmkit, to which he came up with an interesting solution - see
https://github.com/moby/moby/pull/38632 - but alas these tricks are not
available to the Kubelet.)
Signed-off-by: Jean Rouge <rougej+github@gmail.com>
the containerd errdefs functions have the same name as the
docker errdefs, but their types use a different signature;
use an alias to prevent them from being mistaken for the
docker errdefs equivalents.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
looks like we don't need this handling
Before this patch:
Error: No such image: nosuchimage
After this patch:
Error response from daemon: No such image: nosuchimage:latest
"
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This utility allows a client to convert an API response
back to a typed error; allowing the client to perform
different actions based on the type of error, without
having to resort to string-matching the error.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Also removed the `-stretch` suffix, because Debian Stretch
is the default base-image now, so there should be no need
to keep the suffix
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Notable Updates
- Fix an issue that non-existent parent directory in image layers is created with permission 0700. containerd#3017
- Fix an issue that snapshots of the base image can be deleted by mistake, when images built on top of it are deleted. containerd#3087
- Support for GC references to content from snapshot and container objects. containerd#3080
- cgroups updated to dbea6f2bd41658b84b00417ceefa416b97 to fix issues for systemd 420 and non-existent cgroups. containerd#3079
- runc updated to 2b18fe1d885ee5083ef9f0838fee39b62d653e30 to include the improved fix for CVE-2019-5736. containerd#3082
- cri: Fix a bug that pod can't get started when the same volume is defined differently in the image and the pod spec. cri#1059
- cri: Fix a bug that causes container start failure after in-place upgrade containerd to 1.2.4+ or 1.1.6+. cri#1082
- cri updated to a92c40017473cbe0239ce180125f12669757e44f. containerd#3084
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
When initializing graphdrivers without root a permission warning
log is given due to lack of permission to create a device. This
error should be treated the same as quota not supported.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
pborman/uuid and google/uuid used to be different versions of
the same package, but now pborman/uuid is a compatibility wrapper
around google/uuid, maintained by the same person.
Clean up some of the usage as the functions differ slightly.
Not yet removed some uses of pborman/uuid in vendored code but
I have PRs in process for these.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
Signed-off-by: John Howard <jhoward@microsoft.com>
LCOWv1 will be deprecated soon anyway (and LCOW is experimental regardless).
Removing lcow.initrd and lcow.kernel options which will not be supported
in LCOWv2 (via containerd).
Signed-off-by: John Howard <jhoward@microsoft.com>
Fixes#38719
Fixes some subtle bugs on Windows
- Fixes https://github.com/moby/moby/issues/38719. This one is the most important
as failure to start the init process in a Windows container will cause leaked
handles. (ie where the `ctr.hcsContainer.CreateProcess(...)` call fails).
The solution to the leak is to split out the `reapContainer` part of `reapProcess`
into a separate function. This ensures HCS resources are cleaned up correctly and
not leaked.
- Ensuring the reapProcess goroutine is started immediately the process
is actually started, so we don't leak in the case of failures such as
from `newIOFromProcess` or `attachStdio`
- libcontainerd on Windows (local, not containerd) was not sending the EventCreate
back to the monitor on Windows. Just LCOW. This was just an oversight from
refactoring a couple of years ago by Mikael as far as I can tell. Technically
not needed for functionality except for the logging being missing, but is correct.
Signed-off-by: John Howard <jhoward@microsoft.com>
Also fixes https://github.com/moby/moby/issues/22874
This commit is a pre-requisite to moving moby/moby on Windows to using
Containerd for its runtime.
The reason for this is that the interface between moby and containerd
for the runtime is an OCI spec which must be unambigious.
It is the responsibility of the runtime (runhcs in the case of
containerd on Windows) to ensure that arguments are escaped prior
to calling into HCS and onwards to the Win32 CreateProcess call.
Previously, the builder was always escaping arguments which has
led to several bugs in moby. Because the local runtime in
libcontainerd had context of whether or not arguments were escaped,
it was possible to hack around in daemon/oci_windows.go with
knowledge of the context of the call (from builder or not).
With a remote runtime, this is not possible as there's rightly
no context of the caller passed across in the OCI spec. Put another
way, as I put above, the OCI spec must be unambigious.
The other previous limitation (which leads to various subtle bugs)
is that moby is coded entirely from a Linux-centric point of view.
Unfortunately, Windows != Linux. Windows CreateProcess uses a
command line, not an array of arguments. And it has very specific
rules about how to escape a command line. Some interesting reading
links about this are:
https://blogs.msdn.microsoft.com/twistylittlepassagesallalike/2011/04/23/everyone-quotes-command-line-arguments-the-wrong-way/https://stackoverflow.com/questions/31838469/how-do-i-convert-argv-to-lpcommandline-parameter-of-createprocesshttps://docs.microsoft.com/en-us/cpp/cpp/parsing-cpp-command-line-arguments?view=vs-2017
For this reason, the OCI spec has recently been updated to cater
for more natural syntax by including a CommandLine option in
Process.
What does this commit do?
Primary objective is to ensure that the built OCI spec is unambigious.
It changes the builder so that `ArgsEscaped` as commited in a
layer is only controlled by the use of CMD or ENTRYPOINT.
Subsequently, when calling in to create a container from the builder,
if follows a different path to both `docker run` and `docker create`
using the added `ContainerCreateIgnoreImagesArgsEscaped`. This allows
a RUN from the builder to control how to escape in the OCI spec.
It changes the builder so that when shell form is used for RUN,
CMD or ENTRYPOINT, it builds (for WCOW) a more natural command line
using the original as put by the user in the dockerfile, not
the parsed version as a set of args which loses fidelity.
This command line is put into args[0] and `ArgsEscaped` is set
to true for CMD or ENTRYPOINT. A RUN statement does not commit
`ArgsEscaped` to the commited layer regardless or whether shell
or exec form were used.
Signed-off-by: John Howard <jhoward@microsoft.com>
This is the first step in refactoring moby (dockerd) to use containerd on Windows.
Similar to the current model in Linux, this adds the option to enable it for runtime.
It does not switch the graphdriver to containerd snapshotters.
- Refactors libcontainerd to a series of subpackages so that either a
"local" containerd (1) or a "remote" (2) containerd can be loaded as opposed
to conditional compile as "local" for Windows and "remote" for Linux.
- Updates libcontainerd such that Windows has an option to allow the use of a
"remote" containerd. Here, it communicates over a named pipe using GRPC.
This is currently guarded behind the experimental flag, an environment variable,
and the providing of a pipename to connect to containerd.
- Infrastructure pieces such as under pkg/system to have helper functions for
determining whether containerd is being used.
(1) "local" containerd is what the daemon on Windows has used since inception.
It's not really containerd at all - it's simply local invocation of HCS APIs
directly in-process from the daemon through the Microsoft/hcsshim library.
(2) "remote" containerd is what docker on Linux uses for it's runtime. It means
that there is a separate containerd service running, and docker communicates over
GRPC to it.
To try this out, you will need to start with something like the following:
Window 1:
containerd --log-level debug
Window 2:
$env:DOCKER_WINDOWS_CONTAINERD=1
dockerd --experimental -D --containerd \\.\pipe\containerd-containerd
You will need the following binary from github.com/containerd/containerd in your path:
- containerd.exe
You will need the following binaries from github.com/Microsoft/hcsshim in your path:
- runhcs.exe
- containerd-shim-runhcs-v1.exe
For LCOW, it will require and initrd.img and kernel in `C:\Program Files\Linux Containers`.
This is no different to the current requirements. However, you may need updated binaries,
particularly initrd.img built from Microsoft/opengcs as (at the time of writing), Linuxkit
binaries are somewhat out of date.
Note that containerd and hcsshim for HCS v2 APIs do not yet support all the required
functionality needed for docker. This will come in time - this is a baby (although large)
step to migrating Docker on Windows to containerd.
Note that the HCS v2 APIs are only called on RS5+ builds. RS1..RS4 will still use
HCS v1 APIs as the v2 APIs were not fully developed enough on these builds to be usable.
This abstraction is done in HCSShim. (Referring specifically to runtime)
Note the LCOW graphdriver still uses HCS v1 APIs regardless.
Note also that this does not migrate docker to use containerd snapshotters
rather than graphdrivers. This needs to be done in conjunction with Linux also
doing the same switch.
There's no need to try to re-build the test images if they already
exist. This change makes basically no difference to the upstream
integration test-suite running, but for users who want to run the
integration-cli suite on a host machine (such as distributions doing
tests) this change allows images to be pre-loaded such that compilers
aren't needed on the test machine.
However, this does remove the accidental re-compilation of nnp-test, as
well as handling errors far more cleanly (previously if an error
occurred during a test build, further tests won't attempt to rebuild
it).
Signed-off-by: Aleksa Sarai <asarai@suse.de>
It has been declared deprecated by the author, and has a knack for
false-positives (as well as giving bad advice when it comes to APIs --
which is quite clear when looking at "nolint: interfacer" comments).
Signed-off-by: Aleksa Sarai <asarai@suse.de>
When copying between stages, or copying from an image,
ownership of the copied files should not be changed, unless
the `--chown` option is set (in which case ownership of copied
files should be updated to the specified user/group).
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Create a new container for each subtest, so that individual
subtests are self-contained, and there's no need to execute
them in the exact order, or resetting the container in between.
This makes the test slower (6.54s vs 3.43s), but reduced the
difference by using `network=host`, which made a substantial
difference (without `network=host`, the test took more than
twice as long: 13.96s).
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
- Don't set `PidsLimit` when creating a container and
no limit was set (or the limit was set to "unlimited")
- Don't set `PidsLimit` if the host does not have pids-limit
support (previously "unlimited" was set).
- Do not generate a warning if the host does not have pids-limit
support, but pids-limit was set to unlimited (having no
limit set, or the limit set to "unlimited" is equivalent,
so no warning is nescessary in that case).
- When updating a container, convert `0`, and `-1` to
"unlimited" (`0`).
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This changes the default ipc mode of daemon/engine to be private,
meaning the containers will not have their /dev/shm bind-mounted
from the host by default. The benefits of doing this are:
1. No leaked mounts. Eliminate a possibility to leak mounts into
other namespaces (and therefore unfortunate errors like "Unable to
remove filesystem for <ID>: remove /var/lib/docker/containers/<ID>/shm:
device or resource busy").
2. Working checkpoint/restore. Make `docker checkpoint`
not lose the contents of `/dev/shm`, but save it to
the dump, and be restored back upon `docker start --checkpoint`
(currently it is lost -- while CRIU handles tmpfs mounts,
the "shareable" mount is seen as external to container,
and thus rightfully ignored).
3. Better security. Currently any container is opened to share
its /dev/shm with any other container.
Obviously, this change will break the following usage scenario:
$ docker run -d --name donor busybox top
$ docker run --rm -it --ipc container:donor busybox sh
Error response from daemon: linux spec namespaces: can't join IPC
of container <ID>: non-shareable IPC (hint: use IpcMode:shareable
for the donor container)
The soution, as hinted by the (amended) error message, is to
explicitly enable donor sharing by using --ipc shareable:
$ docker run -d --name donor --ipc shareable busybox top
Compatibility notes:
1. This only applies to containers created _after_ this change.
Existing containers are not affected and will work fine
as their ipc mode is stored in HostConfig.
2. Old backward compatible behavior ("shareable" containers
by default) can be enabled by either using
`--default-ipc-mode shareable` daemon command line option,
or by adding a `"default-ipc-mode": "shareable"`
line in `/etc/docker/daemon.json` configuration file.
3. If an older client (API < 1.40) is used, a "shareable" container
is created. A test to check that is added.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
There are two if statements checking for exactly same conditions:
> if hostConfig != nil && versions.LessThan(version, "1.40")
Merge these.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Move the test case from integration-cli to integration.
The test logic itself has not changed, except these
two things:
* the new test sets default-ipc-mode via command line
rather than via daemon.json (less code);
* the new test uses current API version.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Since container.Create() already initializes HostConfig
to be non-nil, there is no need for this code. Remove it.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
relevant changes:
- swarmkit#2815 Extension and resource API declarations
- swarmkit#2816 Moving swap options into `ResourceRequirements` instead of `ContainerSpec`s
- relates to moby#37872
- swarmkit#2821 allocator: use a map for network-IDs to prevent O(n2)
- swarmkit#2832 [api] Add created object to return types for extension and resource create apis
- swarmkit#2831 [controlapi] Extension api implementation
- swarmkit#2835 Resource controlapi Implemetation
- swarmkit#2802 Use custom gRPC dialer to override default proxy dialer
- addresses moby#35395 Swarm worker cannot connect to master if proxy is configured
- addresses moby#issues/36951 Swarm nodes cannot join as masters if http proxy is set
- relates to swarmkit#2419 Provide custom gRPC dialer to override default proxy dialer
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Previously, tag information automatically is added from
/etc/{redhat,system}-release in image (target directory).
But I want to specify any tag informtion when using mkimage-yum.sh.
Because a Linux distribution based RHEL (It's Asianux Server) uses
SPn notation (e.g. SP3) instead of period notaion (e.g. 7.6).
Signed-off-by: Noriki Nakamura <noriki.nakamura@miraclelinux.com>
As people are using the UUID in `docker info` that was based on the v1 manifest signing key, replace
with a UUID instead.
Remove deprecated `--disable-legacy-registry` option that was scheduled to be removed in 18.03.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
It's already defined on the daemon. This allows us to not call
`SystemInfo` which is failry heavy and potentially can even error.
Takes care of todo item from Derek's containerd integration PR.
51c412f26e/daemon/cluster/services.go (L148-L149)
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
When manually stopping a container with a restart-policy, the container
would show as "restarting" in `docker ps` whereas its actual state
is "exited".
Stopping a container with a restart policy shows the container as "restarting"
docker run -d --name test --restart unless-stopped busybox false
docker stop test
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
7e07409fa1d3 busybox "false" 5 minutes ago Restarting (1) 4 minutes ago test
However, inspecting the same container shows that it's exited:
docker inspect test --format '{{ json .State }}'
{
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 1,
"Error": "",
"StartedAt": "2019-02-14T13:26:27.6091648Z",
"FinishedAt": "2019-02-14T13:26:27.689427Z"
}
And killing the container confirms this;
docker kill test
Error response from daemon: Cannot kill container: test: Container 7e07409fa1d36dc8d8cb8f25cf12ee1168ad9040183b85fafa73ee2c1fcf9361 is not running
docker run -d --name test --restart unless-stopped busybox false
docker stop test
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d0595237054a busybox "false" 5 minutes ago Restarting (1) 4 minutes ago exit
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Like its counterpart in images and volumes, introduce the dangling
filter while listing networks. When the filter value is set to true,
only networks which aren't attached to containers and aren't builtin
networks are shown. When set to false, all builtin networks and
networks which are attached to containers are shown.
Signed-off-by: Karthik Nayak <Karthik.188@gmail.com>
This commit uses the exitings jenkins freestyle job configuration to
create a jenkinsfile. This allows the project to use multi-branch
pipelines in Jenkins and to allow others to contribute to the CI setup.
Signed-off-by: Dave Tucker <dt@docker.com>
Older API clients did not use a pointer for `PidsLimit`, so
API requests would always send `0`, resulting in any previous
value to be reset after an update:
Before this patch:
(using a 17.06 Docker CLI):
```bash
docker run -dit --name test --pids-limit=16 busybox
docker container inspect --format '{{json .HostConfig.PidsLimit}}' test
16
docker container update --memory=100M --memory-swap=200M test
docker container inspect --format '{{json .HostConfig.PidsLimit}}' test
0
docker container exec test cat /sys/fs/cgroup/pids/pids.max
max
```
With this patch applied:
(using a 17.06 Docker CLI):
```bash
docker run -dit --name test --pids-limit=16 busybox
docker container inspect --format '{{json .HostConfig.PidsLimit}}' test
16
docker container update --memory=100M --memory-swap=200M test
docker container inspect --format '{{json .HostConfig.PidsLimit}}' test
16
docker container exec test cat /sys/fs/cgroup/pids/pids.max
16
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Some tests were skipped if the local daemon did not have
experimental features enabled; at the same time, some tests
unconditionally created a new (experimental) daemon, even if
the local daemon already had experimental enabled.
This patch;
- Checks if the "testEnv" is an experimental Linux daemon
- If not, and the daemon is running locally; spin up a new
experimental daemon to be used during the test.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This function was previously used on the client to validate
tmpfs options, but is no longer used since
b9b8d8b364, as this validation
is platform-specific, so should be handled by the daemon.
Removing this function as it's no longer used anywhere.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Due to a bug in Golang (github.com/golang#27640), the "character device"
bit was omitted when checking file-modes with `os.ModeType`.
This bug was resolved in Go 1.12, but as a result, graphdrivers
would no longer recognize "device" files, causing pulling of
images that have a file with this filemode to fail;
failed to register layer:
unknown file type for /var/lib/docker/vfs/dir/.../dev/console
The current code checked for an exact match of Modes to be set. The
`os.ModeCharDevice` and `os.ModeDevice` bits will always be set in
tandem, however, because the code was only looking for an exact
match, this detection broke now that `os.ModeCharDevice` was added.
This patch changes the code to be more defensive, and instead
check if the `os.ModeDevice` bit is set (either with, or without
the `os.ModeCharDevice` bit).
In addition, some information was added to the error-message if
no type was matched, to assist debugging in case additional types
are added in future.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The Swarmkit api specifies a target for configs called called "Runtime"
which indicates that the config is not mounted into the container but
has some other use. This commit updates the Docker api to reflect this.
Signed-off-by: Drew Erny <drew.erny@docker.com>
Here's what happens:
1. One runs `make binary` once
2. Days go by...
3. One makes changes to dockerd sources
4. One runs `make test-integration` to test the changes
5. One spends a long time figuring out why on Earth
those changes in step 3 are ignored by step 4.
6. One writes this patch
7. ...
8. PROFIT!!
OK, so `make test-integration` builds a dockerd binary
in bundles/dynbinary-daemon/, when starts a daemon instance
for testing. The problem is, the script that starts the
daemon sets PATH to try `bundles/binary-daemon/` first,
and `bundles/dynbinary-daemon/` second.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
containerd has two objects with regard to containers.
There is a "container" object which is metadata and a "task" which is
manging the actual runtime state.
When docker starts a container, it creartes both the container metadata
and the task at the same time. So when a container exits, docker deletes
both of these objects as well.
This ensures that if, on start, when we go to create the container metadata object
in containerd, if there is an error due to a name conflict that we go
ahead and clean that up and try again.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
- Fixes a vulnerability in runc that allows a container escape (CVE-2019-5736)
6635b4f0c6,
- Includes security fix for `runc run --no-pivot` (`DOCKER_RAMDISK=1`):
28a697cce3
(NOTE: the vuln is attackable only when `DOCKER_RAMDISK=1` is set && seccomp is disabled)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This release drops support for Go < 1.7, and removes the gorilla/context
dependency (which was needed for older Go versions).
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This assists to address a regression where distribution errors were not properly
handled, resulting in a generic 500 (internal server error) to be returned for
`/distribution/name/json` if you weren't authenticated, whereas it should return
a 40x (401).
This patch attempts to extract the HTTP status-code that was returned by the
distribution code, and falls back to returning a 500 status if unable to match.
Before this change:
curl -v --unix-socket /var/run/docker.sock http://localhost/distribution/name/json
* Trying /var/run/docker.sock...
* Connected to localhost (/var/run/docker.sock) port 80 (#0)
> GET /distribution/name/json HTTP/1.1
> Host: localhost
> User-Agent: curl/7.52.1
> Accept: */*
>
< HTTP/1.1 500 Internal Server Error
< Api-Version: 1.37
< Content-Type: application/json
< Docker-Experimental: false
< Ostype: linux
< Server: Docker/dev (linux)
< Date: Tue, 03 Jul 2018 15:52:53 GMT
< Content-Length: 115
<
{"message":"errors:\ndenied: requested access to the resource is denied\nunauthorized: authentication required\n"}
* Curl_http_done: called premature == 0
* Connection #0 to host localhost left intact
daemon logs:
DEBU[2018-07-03T15:52:51.424950601Z] Calling GET /distribution/name/json
DEBU[2018-07-03T15:52:53.179895572Z] FIXME: Got an API for which error does not match any expected type!!!: errors:
denied: requested access to the resource is denied
unauthorized: authentication required
error_type=errcode.Errors module=api
ERRO[2018-07-03T15:52:53.179942783Z] Handler for GET /distribution/name/json returned error: errors:
denied: requested access to the resource is denied
unauthorized: authentication required
With this patch applied:
curl -v --unix-socket /var/run/docker.sock http://localhost/distribution/name/json
* Trying /var/run/docker.sock...
* Connected to localhost (/var/run/docker.sock) port 80 (#0)
> GET /distribution/name/json HTTP/1.1
> Host: localhost
> User-Agent: curl/7.52.1
> Accept: */*
>
< HTTP/1.1 403 Forbidden
< Api-Version: 1.38
< Content-Type: application/json
< Docker-Experimental: false
< Ostype: linux
< Server: Docker/dev (linux)
< Date: Fri, 03 Aug 2018 14:58:09 GMT
< Content-Length: 115
<
{"message":"errors:\ndenied: requested access to the resource is denied\nunauthorized: authentication required\n"}
* Curl_http_done: called premature == 0
* Connection #0 to host localhost left intact
daemon logs:
DEBU[2018-08-03T14:58:08.018726228Z] Calling GET /distribution/name/json
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This fix fixes the following issue with `go vet`:
```
$ go tool vet cmd/dockerd/daemon.go
cmd/dockerd/daemon.go:163: the cancel function is not used on all paths (possible context leak)
cmd/dockerd/daemon.go:167: this return statement may be reached without using the cancel var defined on line 163
```
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
CIs are assumed to do a git fetch and git merge before running tests.
Therefore, no need for a git fetch inside our validate scripts in CI.
If VALIDATE_ORIGIN_BRANCH is set, then git fetch is skipped and
VALIDATE_ORIGIN_BRANCH is used in validate scripts.
Otherwise, behavior is unchanged.
Signed-off-by: Tibor Vass <tibor@docker.com>
Signed-off-by: John Howard <jhoward@microsoft.com>
Reported internally at Microsoft through VSO#19696554.
Using the solution from https://groups.google.com/forum/#!topic/Golang-Nuts/DpldsmrhPio
to quote file name and escape single quotes (https://play.golang.org/p/ntk8EEGjfk)
Simple repro steps are something like:
On an ubuntu box run something like
```
docker run -d --rm -p 5000:5000 registry:latest
hostname-I to get the ip address
```
On Windows start the daemon adding `--insecure-registry 10.124.186.18:5000`
(or whatever the IP address from above was)
```
docker run -it alpine sh
/ # echo bar > "with space"
/ # echo foo > 'single quote space'
/ # exit
docker ps -a
docker commit <containerid>
(note the first few of the image id)
docker tag <first few> 10.124.186.18:5000/test
docker push 10.124.186.18:5000/test
```
Resulting error when pushing the image:
```
PS E:\docker\build\19696554> docker push 10.124.186.18:5000/simpletest2
The push refers to repository [10.124.186.18:5000/simpletest2]
d328d7f5f277: Pushing [==================================================>] 74.24kB/74.24kB
503e53e365f3: Layer already exists
svm.runProcess: command cat /tmp/d59/single quote space failed with exit code 1
PS E:\docker\build\19696554>
```
After this change pushing the image:
```
PS E:\docker\build\19696554> docker push 10.124.186.18:5000/simpletest2
The push refers to repository [10.124.186.18:5000/simpletest2]
d328d7f5f277: Pushing [==================================================>] 74.24kB/74.24kB
503e53e365f3: Layer already exists
latest: digest: sha256:b9828a2d2a3d2421a4c342f48b7936714b3d8409dc32c103da5f3fb13b54bdbf size: 735
PS E:\docker\build\19696554>
```
Please refer to `docs/rootless.md`.
TLDR:
* Make sure `/etc/subuid` and `/etc/subgid` contain the entry for you
* `dockerd-rootless.sh --experimental`
* `docker -H unix://$XDG_RUNTIME_DIR/docker.sock run ...`
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
This brings in a single fix; swarmkit#2813 where a field inadvertedly
used incorrect capitalization.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Full diff: 8af8c420f4...1a0ebd43b2
relevant changes:
- swarmkit#2771 Allow using Configs as CredentialSpecs
- swarmkit#2804 Make Service.UpdateStatus non-ambiguous
- swarmkit#2805 Refactor condition in restart supervisor
- swarmkit#2780 api: add BindOptions.NonRecursive
- related to moby#38003
- swarmkit#2790 Fix possible panic if NetworkConfig is nil
- swarmkit#2797 Include old error-message for backward compatibility
- related to swarmkit#2779 / moby#38140 / moby#38142
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Monitoring systems and load balancers are usually configured to use HEAD
requests for health monitoring. The /_ping endpoint currently does not
support this type of request, which means that those systems have fallback
to GET requests.
This patch adds support for HEAD requests on the /_ping endpoint.
Although optional, this patch also returns `Content-Type` and `Content-Length`
headers in case of a HEAD request; Refering to RFC 7231, section 4.3.2:
The HEAD method is identical to GET except that the server MUST NOT
send a message body in the response (i.e., the response terminates at
the end of the header section). The server SHOULD send the same
header fields in response to a HEAD request as it would have sent if
the request had been a GET, except that the payload header fields
(Section 3.3) MAY be omitted. This method can be used for obtaining
metadata about the selected representation without transferring the
representation data and is often used for testing hypertext links for
validity, accessibility, and recent modification.
A payload within a HEAD request message has no defined semantics;
sending a payload body on a HEAD request might cause some existing
implementations to reject the request.
The response to a HEAD request is cacheable; a cache MAY use it to
satisfy subsequent HEAD requests unless otherwise indicated by the
Cache-Control header field (Section 5.2 of [RFC7234]). A HEAD
response might also have an effect on previously cached responses to
GET; see Section 4.3.5 of [RFC7234].
With this patch applied, either `GET` or `HEAD` requests work; the only
difference is that the body is empty in case of a `HEAD` request;
curl -i --unix-socket /var/run/docker.sock http://localhost/_ping
HTTP/1.1 200 OK
Api-Version: 1.40
Cache-Control: no-cache, no-store, must-revalidate
Docker-Experimental: false
Ostype: linux
Pragma: no-cache
Server: Docker/dev (linux)
Date: Mon, 14 Jan 2019 12:35:16 GMT
Content-Length: 2
Content-Type: text/plain; charset=utf-8
OK
curl --head -i --unix-socket /var/run/docker.sock http://localhost/_ping
HTTP/1.1 200 OK
Api-Version: 1.40
Cache-Control: no-cache, no-store, must-revalidate
Content-Length: 0
Content-Type: text/plain; charset=utf-8
Docker-Experimental: false
Ostype: linux
Pragma: no-cache
Server: Docker/dev (linux)
Date: Mon, 14 Jan 2019 12:34:15 GMT
The client is also updated to use `HEAD` by default, but fallback to `GET`
if the daemon does not support this method.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Update to the latest patch release of 17.06.2. This
keeps the same API requirements.
This also enables pre-built binaries for armhf instead
of compiling from source.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
When running CI in other repositories (e.g. Docker's downstream
docker/engine repository), or other branches, the validation
scripts were calculating the list of changes based on the wrong
information.
This lead to weird failures in CI in a branch where these values
were not updated ':-) (CI on a pull request failed because it detected
that new tests were added to the deprecated `integration-cli` test-suite,
but the pull request did not actually make changes in that area).
This patch allows overriding the target repository (and branch)
to compare to (without having to edit the scripts).
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
As reported in docker/for-linux/issues/484, since Docker 18.06
docker cp with a destination file name fails with the following error:
> archive/tar: cannot encode header: Format specifies USTAR; and USTAR cannot encode Name="a_very_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx_long_filename_that_is_101_characters"
The problem is caused by changes in Go 1.10 archive/tar, which
mis-guesses the tar stream format as USTAR (rather than PAX),
which, in turn, leads to inability to specify file names
longer than 100 characters.
This tar stream is sent by TarWithOptions() (which, since we switched to
Go 1.10, explicitly sets format=PAX for every file, see FileInfoHeader(),
and before Go 1.10 it was PAX by default). Unfortunately, the receiving
side, RebaseArchiveEntries(), which calls tar.Next(), mistakenly guesses
header format as USTAR, which leads to the above error.
The fix is easy: set the format to PAX in RebaseArchiveEntries()
where we read the tar stream and change the file name.
A unit test is added to prevent future regressions.
NOTE this code is not used by dockerd, but rather but docker cli
(also possibly other clients), so this needs to be re-vendored
to cli in order to take effect.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
for windows all networks are re-populated in the store during network controller initialization. In current version it also regenerate network Ids which may be referenced by other components and it may cause broken references to a networks. This commit avoids regeneration of network ids.
Signed-off-by: Andrey Kolomentsev <andrey.kolomentsev@docker.com>
The docs maintainers role was in the maintainers file
from way back when the documentation was still in this
repository.
Now that the documentation has moved to its own repository,
we should no longer need this section.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
- Add support for exact list of capabilities, support only OCI model
- Support OCI model on CapAdd and CapDrop but remain backward compatibility
- Create variable locally instead of declaring it at the top
- Use const for magic "ALL" value
- Rename `cap` variable as it overlaps with `cap()` built-in
- Normalize and validate capabilities before use
- Move validation for conflicting options to validateHostConfig()
- TweakCapabilities: simplify logic to calculate capabilities
Signed-off-by: Olli Janatuinen <olli.janatuinen@gmail.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This fix moves multiple places of serviceRunningTasksCount
to one location in integration/internal/swarm, so that
code duplication could be removed.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
This fix replaces waitAndAssert in config_test.go with poll.WaitOn
so that the testing is consistent with all other tests in integration.
Also, config_test.go uses to wait and sleep for 2 * (1 minutes) to get the task
info. This fix combined those two sleep and wait for 1 mins. Think 1 min
is enough for config test.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
This introduces `NoTasksForService` and `NoTasks` poller checks, that
can be used to check if no tasks are left in general, or for a specific
service.
Some redundant checks were also removed from some tests.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Clean up and refactor this test;
- make `serviceRunningTasksCount` to use a `desired-state` filter
- use subtests, and inline the `validNetworkVerbose` checks; also use
asserts for the individual checks, so that any failure will log exactly
what failed
- remove helper functions that are no longer needed
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Tests generally already have a client instance, so it
probably makes more sense to just pass it, and make
this utility a bit more flexible to use.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This test sometimes failed because the number of events received did not
match the expected number:
FAIL: docker_cli_events_test.go:316: DockerSuite.TestEventsFilterLabels
docker_cli_events_test.go:334:
c.Assert(len(events), checker.Equals, 3)
... obtained int = 2
... expected int = 3
This patch makes the test more stable, by:
- use a wider range between `--since` and `--until`. These options were set
so that the client detaches after events were received, but the actual
range should not matter. Changing the range will cause more events to be
returned, but we're specifically looking for the container ID's, so this
should not make a difference for the actual test.
- use `docker create` instead of `docker run` for the containers. the
containers don't have to be running to trigger an event; using `create`
speeds up the test.
- check the exit code of the `docker create` to verify the containers were
succesfully created.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
I noticed that this test failed, because the node was in status "pending".
The test checks for the node's status immediately after it was restarted, so
possibly it needs some time to unlock.
14:07:10 FAIL: docker_cli_swarm_test.go:1128: DockerSwarmSuite.TestSwarmLockUnlockCluster
...
14:07:10 docker_cli_swarm_test.go:1168:
14:07:10 checkSwarmLockedToUnlocked(c, d)
14:07:10 docker_cli_swarm_test.go:1017:
14:07:10 c.Assert(getNodeStatus(c, d), checker.Equals, swarm.LocalNodeStateActive)
14:07:10 ... obtained swarm.LocalNodeState = "pending"
14:07:10 ... expected swarm.LocalNodeState = "active"
This patch adds a `waitAndAssert` for the node's status, with a 1 second timeout.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
By default, exec uses the environment of the current process, however,
if `exec.Env` is not `nil`, the environment is discarded:
e73f489494/src/os/exec/exec.go (L57-L60)
> If Env is nil, the new process uses the current process's environment.
When adding a new environment variable, prepend the current environment,
to make sure it is not discarded.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
... so finally I'll be able to talk to a bot rather than a real person
in order to restart a CI job ;)
[v2: add to curators in MAINTAINERS]
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
comparing PR commit(s) to HEAD of moby/moby master branch and if founds
new (or renamed) integration tests will run stress tests for them.
Signed-off-by: Olli Janatuinen <olli.janatuinen@gmail.com>
When running a container in the host's network namespace, the container
gets a copy of the host's resolv.conf (copied to `/etc/resolv.conf` inside
the container).
The current code always used the default (`/etc/resolv.conf`) path on the
host, irregardless if `systemd-resolved` was used or not.
This patch uses the correct file if `systemd-resolved` was detected
to be running.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This commit fixes two possible crashes in the `*Daemon` bound method
`reloadMaxConcurrentDownloadsAndUploads()`.
The first fixed issue is when `daemon.imageService` is `nil`. The second
panic can occur if the provided `*config.Config` is incomplete and the
fields `conf.MaxConcurrentDownloads` or `conf.MaxConcurrentUploads` are
`nil`.
Signed-off-by: Sascha Grunert <sgrunert@suse.com>
This patch;
- adds support for multiple newlines
- removes the 1-space indentation of wrapped lines
- allows numerical characters in targets (0-9)
Given these targets:
```Makefile
.PHONY: foobar
foobar: ## runs the foobar lorum ipsum.\nand so pn\nand so on
echo foobar
.PHONY: e2e-tests
e2e-tests: ## runs the end-to-end tests
echo e2e-tests
```
Before this change, the output of `make help` was
```
foobar runs the foobar lorum ipsum.
and so pn\nand so on
```
After this change, the output is:
```
foobar runs the foobar lorum ipsum.
and so pn
and so on
e2e-tests runs the end-to-end tests
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
These tests should not require a local daemon; they may fail if
the local and remote system's clocks are out of sync with more
than a minute though, but that's something we should prevent from
happening :-)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Tests which will re-deploy containers uses function serviceIsUpdated() to
make sure that service update really reached state UpdateStateCompleted.
Tests which will not re-deploy container uses function
serviceSpecIsUpdated to make sure that service version is increased.
Signed-off-by: Olli Janatuinen <olli.janatuinen@gmail.com>
- Fix a bug that a container can't be stopped or inspected when its corresponding image is deleted
- Fix a bug that the cri plugin handles containerd events outside of k8s.io namespace
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This fix fixes go vet issue:
```
daemon/daemon.go:273: loop variable id captured by func literal
daemon/daemon.go:280: loop variable id captured by func literal
```
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
RHEL/CentOS 3.10 kernels report that kernel-memory accounting is supported,
but it actually does not work.
Runc (when compiled for those kernels) will be compiled without kernel-memory
support, so even though the daemon may be reporting that it's supported,
it actually is not.
This cause tests to fail when testing against a daemon that's using a runc
version without kmem support.
For now, skip these tests based on the kernel version reported by the daemon.
This should fix failures such as:
```
FAIL: /go/src/github.com/docker/docker/integration-cli/docker_cli_run_unix_test.go:499: DockerSuite.TestRunWithKernelMemory
assertion failed:
Command: /usr/bin/docker run --kernel-memory 50M --name test1 busybox cat /sys/fs/cgroup/memory/memory.kmem.limit_in_bytes
ExitCode: 0
Error: <nil>
Stdout: 9223372036854771712
Stderr: WARNING: You specified a kernel memory limit on a kernel older than 4.0. Kernel memory limits are experimental on older kernels, it won't work as expected and can cause your system to be unstable.
Failures:
Expected stdout to contain "52428800"
FAIL: /go/src/github.com/docker/docker/integration-cli/docker_cli_update_unix_test.go:125: DockerSuite.TestUpdateKernelMemory
/go/src/github.com/docker/docker/integration-cli/docker_cli_update_unix_test.go:136:
...open /go/src/github.com/docker/docker/integration-cli/docker_cli_update_unix_test.go: no such file or directory
... obtained string = "9223372036854771712"
... expected string = "104857600"
----------------------------------------------------------------------
FAIL: /go/src/github.com/docker/docker/integration-cli/docker_cli_update_unix_test.go:139: DockerSuite.TestUpdateKernelMemoryUninitialized
/go/src/github.com/docker/docker/integration-cli/docker_cli_update_unix_test.go:149:
...open /go/src/github.com/docker/docker/integration-cli/docker_cli_update_unix_test.go: no such file or directory
... value = nil
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This is a slow test, taking over two minutes to complete,
so spin up a new daemon for this test and run it in parallel
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
A recent CI run shows a (seemingly random) failure from this test:
> 00:14:37.289 --- FAIL: TestRenameAnonymousContainer (1.75s)
> 00:14:37.289 rename_test.go:169: assertion failed: 0 (int) != 1 (inspect.State.ExitCode int): container baac251d5a1cb2221ffedf6f10acbad166b90e3549601e96d908e76762675a81 exited with the wrong exitcode: {ContainerJSONBase:0xc0007a4840 Mounts:[] Config:0xc000714500 NetworkSettings:0xc000235b00}
Apparently, printing the whole `inspect` value does not make any sense.
Let's output `inspect.State.Error` instead, maybe it will help to
figure out what is going on here.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
A client is already created in testenv.New(), so we can just
as well use that one, instead of creating a new client.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
`testEnv` is a package-level variable, so protecting / restoring
`testEnv` in parallel will result in "concurrent map write" errors.
This patch removes `t.Parallel()` from tests that use this
functionality (through `defer setupTest(t)()`).
Note that _subtests_ can still be run in parallel, as the defer
will be called after all subtests have completed.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The `time.Sleep()` and `apiclient.ImageList()` were added in d3cc071bb9,
but appear to be debugging-code and not needed for the test.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
CRIU supports checkpoint and restore of tty devices since version 2.12
which was released on 8th of March 2017. Support for this functionality
was implemented with opencontainers/runc@1c43d09 (checkpoint: add
support for containers with terminals) and containerd/containerd@60daa41
(Allow to checkpoint and restore a container with console).
Therefore, we can enable the support in moby/docker.
Signed-off-by: Radostin Stoyanov <rstoyanov1@gmail.com>
This test starts a new daemon, which will fail when testing
against a remote daemon;
--- FAIL: TestInfoAPIWarnings (0.00s)
info_test.go:53: failed to start daemon with arguments [-H=0.0.0.0:23756 -H=unix:///tmp/docker-integration/d5153ebcf89ef.sock] : [d5153ebcf89ef] could not find docker binary in $PATH: exec: "dockerd": executable file not found in $PATH
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
build the final stage of the Dockerfile (including COPY ...) if no BIND_DIR
is used.
if BIND_DIR is used, build the dev stage, thus skipping the COPY.
Original author: @thaJeztah
Signed-off-by: Mohammad Nasirifar <farnasirim@gmail.com>
Adds the following validations to the syslog logger test:
1. Only supported options are valid
2. Log option syslog-address has to be a valid URI
3. Log option syslog-address if is file has to exist
4. Log option syslog-address if udp/tcp scheme, default to port 513
5. Log-option syslog-facility has to be a valid facility
6. Log-option syslog-format has to be a valid format
Signed-off-by: Joao Trindade <trindade.joao@gmail.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
I noticed this in the build output;
```
16:05:07 [100%] Built target tini-static
16:05:07 + mkdir -p /build/
16:05:07 + cp tini-static /build//docker-init
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
These tests were spinning up a new daemon, but after the daemon was spun up,
the default test-daemon was used by the client.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
- TestAPISwarmLeaderElection
- TestAPISwarmRaftQuorum
- TestSwarmClusterRotateUnlockKey
because they are known to be flaky.
Signed-off-by: Olli Janatuinen <olli.janatuinen@gmail.com>
These scripts explicitly use Bash, so we should be able to use
`[[` instead of `[` (which seems to be recommended).
Also added curly brackets to some bare variables, and quoted some paths.
This makes my IDE a bit more silent :-)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
* Replaces `cocks` with `cerf` as the former might be perceived as
offensive by some people (as pointed out by @jeking3
[here](https://github.com/moby/moby/pull/37157#commitcomment-31758059))
* Removes a duplicate entry for `burnell`
* Re-arranges the entry for `sutherland` to ensure that the names are in
sorted order
* Adds entries for `shamir` and `wilbur`
Signed-off-by: Debayan De <debayande@users.noreply.github.com>
I noticed this error in CI:
```
20:37:25 INFO: Non-base image count on control daemon to delete is 9
20:37:25 "docker rmi" requires at least 1 argument.
20:37:25 See 'docker rmi --help'.
20:37:25
20:37:25 Usage: docker rmi [OPTIONS] IMAGE [IMAGE...]
20:37:25
20:37:25 Remove one or more images
```
Which indicated that the PowerShell script managed to find images to delete, but
not actually passing the images to `docker rmi`.
The reason for this failing was that the script attempted to convert the
collection/array to a string, which produces;
```powershell
Write-Output $(docker images --format "{{.Repository}}:{{.ID}}" | `
>> select-string -NotMatch "windowsservercore" | `
>> select-string -NotMatch "nanoserver" | `
>> select-string -NotMatch "docker" `
>> ).ToString()
System.Object[]
```
Which, when trying to split by the chosen separator (`:`), will return the same;
```powershell
Write-Output "System.Object[]".Split(":")[0]
```
This patch:
- Adds an intermediate variable (`$allImages`) to make the code better readable
- Switches the separator to `#`, to prevent breaking on images pulled from a
repository with a port in its name (`myregistry:5000/my/image`)
- Switches to use a comma-separated list for `-NotMatch` (for readability)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Using `errors.Errorf()` passes the error with the stack trace for
debugging purposes.
Also using `errdefs.InvalidParameter` for Windows, so that the API
will return a 4xx status, instead of a 5xx, and added tests for
both validations.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Description:
When using local volume option such as size=10G, type=tmpfs, if we provide wrong options, we could create volume successfully.
But when we are ready to use it, it will fail to start container by failing to mount the local volume(invalid option).
We should check the options at when we create it.
Signed-off-by: Wentao Zhang <zhangwentao234@huawei.com>
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Add awslogs-force-flush-interval-seconds and awslogs-max-buffered-events configurable options to aswlogs driver to replace hardcoded values of repsectively 5 seconds and 4K.
Signed-off-by: Maximiliano Maccanti <maccanti@amazon.com>
Many startup tasks have to run for each container, and thus using a
WaitGroup (which doesn't have a limit to the number of parallel tasks)
can result in Docker exceeding the NOFILE limit quite trivially. A more
optimal solution is to have a parallelism limit by using a semaphore.
In addition, several startup tasks were not parallelised previously
which resulted in very long startup times. According to my testing, 20K
dead containers resulted in ~6 minute startup times (during which time
Docker is completely unusable).
This patch fixes both issues, and the parallelStartupTimes factor chosen
(128 * NumCPU) is based on my own significant testing of the 20K
container case. This patch (on my machines) reduces the startup time
from 6 minutes to less than a minute (ideally this could be further
reduced by removing the need to scan all dead containers on startup --
but that's beyond the scope of this patchset).
In order to avoid the NOFILE limit problem, we also detect this
on-startup and if NOFILE < 2*128*NumCPU we will reduce the parallelism
factor to avoid hitting NOFILE limits (but also emit a warning since
this is almost certainly a mis-configuration).
Signed-off-by: Aleksa Sarai <asarai@suse.de>
In applyTar, if the driver's ApplyDiff returns an error, the function
returns early without calling io.Copy.
As a consequence, the resources (a goroutine and some buffers holding
the uncompressed image, the digest, etc...) allocated or referenced by
NewInputTarStream above aren't released, as the worker goroutine only
finishes when it finds EOF or a closed pipe.
Signed-off-by: Sergio Lopez <slp@redhat.com>
A lack of check in the test code can lead to a panic due to
`len(ids)` being `0`.
Avoid the panic by adding appropriate checks. Note `Assert()` should be
used rather than `Check()` as if it fails we should not proceed with the
test.
Originally found in https://github.com/moby/moby/pull/38404.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
go1.11.4 (released 2018/12/14) includes fixes to cgo, the compiler, linker,
runtime, documentation, go command, and the net/http and go/types packages. It
includes a fix to a bug introduced in Go 1.11.3 that broke go get for import
path patterns containing "...".
See the Go 1.11.4 milestone for details:
https://github.com/golang/go/issues?q=milestone%3AGo1.11.4+label%3ACherryPickApproved
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Some messages are logged before the logrus format was set,
therefore resulting in inconsistent log-message formatting
during startup;
Before this patch;
```
dockerd --experimental
WARN[0000] Running experimental build
INFO[2018-11-24T11:24:05.615249610Z] libcontainerd: started new containerd process pid=132
INFO[2018-11-24T11:24:05.615348322Z] parsed scheme: "unix" module=grpc
...
```
With this patch applied;
```
dockerd --experimental
WARN[2018-11-24T13:41:51.199057259Z] Running experimental build
INFO[2018-11-24T13:41:51.200412645Z] libcontainerd: started new containerd process pid=293
INFO[2018-11-24T13:41:51.200523051Z] parsed scheme: "unix" module=grpc
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Disabling the oom-killer for a container without setting a memory limit
is dangerous, as it can result in the container consuming unlimited memory,
without the kernel being able to kill it. A check for this situation is curently
done in the CLI, but other consumers of the API won't receive this warning.
This patch adds a check for this situation to the daemon, so that all consumers
of the API will receive this warning.
This patch will have one side-effect; docker cli's that also perform this check
client-side will print the warning twice; this can be addressed by disabling
the cli-side check for newer API versions, but will generate a bit of extra
noise when using an older CLI.
With this patch applied (and a cli that does not take the new warning into account);
```
docker create --oom-kill-disable busybox
WARNING: OOM killer is disabled for the container, but no memory limit is set, this can result in the system running out of resources.
669933b9b237fa27da699483b5cf15355a9027050825146587a0e5be0d848adf
docker run --rm --oom-kill-disable busybox
WARNING: Disabling the OOM killer on containers without setting a '-m/--memory' limit may be dangerous.
WARNING: OOM killer is disabled for the container, but no memory limit is set, this can result in the system running out of resources.
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This import got lost after commit 56cc56b0fa
was merged, likely because the PR was built against an outdated
master.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This avoids the corner case where `via` is not nil, but has a length of 0,
so the updated code does not panic in that situation.
Signed-off-by: Iskander Sharipov <quasilyte@gmail.com>
The errors returned from Mount and Unmount functions are raw
syscall.Errno errors (like EPERM or EINVAL), which provides
no context about what has happened and why.
Similar to os.PathError type, introduce mount.Error type
with some context. The error messages will now look like this:
> mount /tmp/mount-tests/source:/tmp/mount-tests/target, flags: 0x1001: operation not permitted
or
> mount tmpfs:/tmp/mount-test-source-516297835: operation not permitted
Before this patch, it was just
> operation not permitted
[v2: add Cause()]
[v3: rename MountError to Error, document Cause()]
[v4: fixes; audited all users]
[v5: make Error type private; changes after @cpuguy83 reviews]
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
It has been pointed out that we're ignoring EINVAL from umount(2)
everywhere, so let's move it to a lower-level function. Also, its
implementation should be the same for any UNIX incarnation, so
let's consolidate it.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
As standard mount.Unmount does what we need, let's use it.
In addition, this adds ignoring "not mounted" condition, which
was previously implemented (see PR#33329, commit cfa2591d3f)
via a very expensive call to mount.Mounted().
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
syscall.Stat (and Lstat), unlike functions from os pkg,
return "raw" errors (like EPERM or EINVAL), and those are
propagated up the function call stack unchanged, and gets
logged and/or returned to the user as is.
Wrap those into os.PathError{} so the error message will
at least have function name and file name.
Note we use Capitalized function names to distinguish
between functions in os and ours.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Ubuntu kernel supports overlayfs in user namespaces.
However, Docker had previously crafting overlay opaques directly
using mknod(2) and setxattr(2), which are not supported in userns.
Tested with LXD, Ubuntu 18.04, kernel 4.15.0-36-generic #39-Ubuntu.
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
This small fix renames `TestfillLicense` to `TestFillLicense`
as otherwise go vet reports:
```
$ go tool vet daemon/licensing_test.go
daemon/licensing_test.go:11: TestfillLicense has malformed name: first letter after 'Test' must not be lowercase
```
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
This fix tries to address the issue raised in 38258
where current RFC5424 sys log format does not zero pad
the time (trailing zeros are removed)
This fix apply the patch to fix the issue. This fix fixes 38258.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
Add Location following since security redirects to security-cdn and caused the repository to be added on Debian unstable.
Signed-off-by: Mattias Jernberg <nostrad@gmail.com>
@sw-pschmied originally post this in #38285
While looking through the Moby source code was found /proc/asound to be
shared with containers as read-only (as defined in
https://github.com/moby/moby/blob/master/oci/defaults.go#L128).
This can lead to two information leaks.
---
**Leak of media playback status of the host**
Steps to reproduce the issue:
- Listen to music/Play a YouTube video/Do anything else that involves
sound output
- Execute docker run --rm ubuntu:latest bash -c "sleep 7; cat
/proc/asound/card*/pcm*p/sub*/status | grep state | cut -d ' ' -f2 |
grep RUNNING || echo 'not running'"
- See that the containerized process is able to check whether someone
on the host is playing music as it prints RUNNING
- Stop the music output
- Execute the command again (The sleep is delaying the output because
information regarding playback status isn't propagated instantly)
- See that it outputs not running
**Describe the results you received:**
A containerized process is able to gather information on the playback
status of an audio device governed by the host. Therefore a process of a
container is able to check whether and what kind of user activity is
present on the host system. Also, this may indicate whether a container
runs on a desktop system or a server as media playback rarely happens on
server systems.
The description above is in regard to media playback - when examining
`/proc/asound/card*/pcm*c/sub*/status` (`pcm*c` instead of `pcm*p`) this
can also leak information regarding capturing sound, as in recording
audio or making calls on the host system.
Signed-off-by: Jonathan A. Schweder <jonathanschweder@gmail.com>
The OCI doesn't have a specific field for an NIS domainname[1] (mainly
because FreeBSD and Solaris appear to have a similar concept but it is
configured entirely differently).
However, on Linux, the NIS domainname can be configured through both the
setdomainname(2) syscall but also through the "kernel.domainname"
sysctl. Since the OCI has a way of injecting sysctls this means we don't
need to have any OCI changes to support NIS domainnames (and we can
always switch if the OCI picks up such support in the future).
It should be noted that because we have to generate this each spec
creation we also have to make sure that it's not clobbered by the
HostConfig. I'm pretty sure making this change generic (so that
HostConfig will not clobber any pre-set sysctls) will not cause other
issues to crop up.
[1]: https://github.com/opencontainers/runtime-spec/issues/592
Signed-off-by: Aleksa Sarai <asarai@suse.de>
The cancellable handler is no longer needed as the context that is
passed with the http request will be cancelled just like the close
notifier was doing.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Signed-off-by: John Howard <jhoward@microsoft.com>
This is a follow-on from https://github.com/moby/moby/pull/38277
but had to be done in a couple of stages to ensure that CI didn't
break. v1.1 of the busybox image is now based on a CMD of "sh"
rather than using an entrypoint. And it also uses the bin directory
rather than `c:\busybox`. This makes it look a lot closer to the
Linux busybox image, and means that a couple of Windows-isms in
CI tests can be reverted back to be identical to their Linux
equivalents.
This fix tries to address the issue raised in 37038 where
there were no memory.kernelTCP support for linux.
This fix add MemoryKernelTCP to HostConfig, and pass
the config to runtime-spec.
Additional test case has been added.
This fix fixes 37038.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
The roadmap is one of the most important ways that a new contributor may
get started on the codebase, as such it is important for it to reflect
the real effort that is currently happening.
This update just brings it up to date. There may be some other efforts
going on and I would encourage people to update the roadmap accordingly
as a separate effort.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Signed-off-by: John Howard <jhoward@microsoft.com>
If fixes an error in sameFsTime which was using `==` to compare two times. The correct way is to use go's built-in timea.Equals(timeb).
In changes_windows, it uses sameFsTime to compare mTim of a `system.StatT` to allow TestChangesDirsMutated to operate correctly now.
Note there is slight different between the Linux and Windows implementations of detecting changes. Due to https://github.com/moby/moby/issues/9874,
and the fix at https://github.com/moby/moby/pull/11422, Linux does not consider a change to the directory time as a change. Windows on NTFS
does. See https://github.com/moby/moby/pull/37982 for more information. The result in `TestChangesDirsMutated`, `dir3` is NOT considered a change
in Linux, but IS considered a change on Windows. The test mutates dir3 to have a mtime of +1 second.
With a handful of tests still outstanding, this change ports most of the unit tests under pkg/archive to Windows.
It provides an implementation of `copyDir` in tests for Windows. To make a copy similar to Linux's `cp -a` while preserving timestamps
and links to both valid and invalid targets, xcopy isn't sufficient. So I used robocopy, but had to circumvent certain exit codes that
robocopy exits with which are warnings. Link to article describing this is in the code.
The v1.10 layout and the migrator was added in 2015 via #17924.
Although the migrator is not marked as "deprecated" explicitly in
cli/docs/deprecated.md, I suppose people should have already migrated
from pre-v1.10 and they no longer need the migrator, because pre-v1.10
version do not support schema2 images (and these versions no longer
receives security updates).
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
This commit contains changes to configure DataPathPort
option. By default we use 4789 port number. But this commit
will allow user to configure port number during swarm init.
DataPathPort can't be modified after swarm init.
Signed-off-by: selansen <elango.siva@docker.com>
Implements the --device forwarding for Windows daemons. This maps the physical
device into the container at runtime.
Ex:
docker run --device="class/<clsid>" <image> <cmd>
Signed-off-by: Justin Terry (VM) <juterry@microsoft.com>
The previous update used a commit from master. Now that
all the fixes are backported to the containerd 1.2 release
branch, we can switch back to that branch.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
release notes: https://github.com/containerd/containerd/releases/tag/v1.2.0
- New V2 Runtime with a stable gRPC interface for managing containers through
external shims.
- Updated CRI Plugin, validated against Kubernetes v1.11 and v1.12, but it is
also compatible with Kubernetes v1.10.
- Support for Kubernetes Runtime Class, introduced in Kubernetes 1.12
- A new proxy plugin configuration has been added to allow external
snapshotters be connected to containerd using gRPC.-
- A new Install method on the containerd client allows users to publish host
level binaries using standard container build tooling and container
distribution tooling to download containerd related binaries on their systems.
- Add support for cleaning up leases and content ingests to garbage collections.
- Improved multi-arch image support using more precise matching and ranking
- Added a runtime `options` field for shim v2 runtime. Use the `options` field to
config runtime specific options, e.g. `NoPivotRoot` and `SystemdCgroup` for
runtime type `io.containerd.runc.v1`.
- Some Minor API additions
- Add `ListStream` method to containers API. This allows listing a larger
number of containers without hitting message size limts.
- Add `Sync` flag to `Delete` in leases API. Setting this option will ensure
a garbage collection completes before the removal call is returned. This can
be used to guarantee unreferenced objects are removed from disk after a lease.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
updates golang.org/x/sys to 90868a75fefd03942536221d7c0e2f84ec62a668, which
adds support for building on windows/arm.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Containerd should be "leading" when specifying which version of runc to use.
From the RUNC.MD document in the containerd repository
(https://github.com/containerd/containerd/blob/b1e202c32724e82779544365528a1a082
b335553/RUNC.md);
> We depend on a specific runc version when dealing with advanced features. You
> should have a specific runc build for development. The current supported runc
> commit is described in vendor.conf. Please refer to the line that starts with
> github.com/opencontainers/runc.
This patch adds a note to vendor.conf and runc.installer to describe the order
in which runc should be updated.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
On kernels without this options set, publishing ports for swarm
services does not work, making the published port not accessible
("connection refused")
Thanks to Wenbo Wang for reporting, and Tianon for finding this.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
These options were added in API 1.39, so should be ignored
when using an older version of the API.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
No code changes, but this aligns it to a tagged version,
and updates some nested license files to MIT.
vndr doesn't vendor those nested files, so no code changes in
the vendor directory.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
In case we're running on RHEL7 kernel, which has non-working
and broken kernel memory controller, add 'nokmem' build tag
so that runc never enables kmem accounting.
For more info, see the following runc commit:
https://github.com/opencontainers/runc/commit/6a2c1559684
This behavior can be overriden by having `RUNC_NOKMEM` environment
variable set (e.g. to empty value to disable setting nokmem).
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
This allows non-recursive bind-mount, i.e. mount(2) with "bind" rather than "rbind".
Swarm-mode will be supported in a separate PR because of mutual vendoring.
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
Only the filters the user specified should be added as cache filters to buildkit.
Make an AND operation of the provided filters.
ID filter now does prefix-matching.
Signed-off-by: Tibor Vass <tibor@docker.com>
4.8+ kernels have fixed the ptrace security issues
so we can allow ptrace(2) on the default seccomp
profile if we do the kernel version check.
93e35efb8d
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Some improvements in this test;
- use the volume-information that's returned by VolumeCreate as "expected"
- don't use an explict name for the volume, as it was only used to reference
the volume for inspection
- improve the test-output on failure, so that "expected" and "actual" values
are printed
Without this patch applied;
=== RUN TestVolumesInspect
--- FAIL: TestVolumesInspect (0.02s)
volume_test.go:108: assertion failed: false (bool) != true (true bool): Time Volume is CreatedAt not equal to current time
FAIL
With this patch applied;
=== RUN TestVolumesInspect
--- FAIL: TestVolumesInspect (0.02s)
volume_test.go:95: assertion failed: expression is false: createdAt.Truncate(time.Minute).Equal(now.Truncate(time.Minute)): CreatedAt (2018-11-01 16:15:20 +0000 UTC) not equal to creation time (2018-11-01 16:15:20.2421166 +0000 UTC m=+13.733512701)
FAIL
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
When starting docker daemons for swarm testing, we disable iptables
and use lo for communication (in order to avoid network conflicts).
The problem is, these options are lost on restart, that can lead
to any sorts of network conflicts and thus connectivity issues
between swarm nodes.
Fix this. This does not fix issues with swarm test failures, but
it seems they appear are less often after this one.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
This is repeated 6 times in different tests, with slight
minor variations. Let's factor it out, for clarity.
While at it, simplify the code: instead of more complex
parsing of "docker swarm init|update --autolock" output (1)
and checking if the key is also present in
"docker swarm unlock-key" output (2), get the key
from (2) and check it is present in (1).
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
A timer is leaking on every daemon start and stop.
Probably nothing major, but given the amount of
daemon starts/stops during tests, it's better to
be accurate about it.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
1. Using MNT_FORCE flag does not make sense for nsfs. Using MNT_DETACH
though might help.
2. When -check.vv is added to TESTFLAGS, there are a lot of messages
like this one:
> unmount of /tmp/dxr/d847fd103a4ba/netns failed: invalid argument
and some like
> unmount of /tmp/dxr/dd245af642d94/netns failed: no such file or directory
The first one means directory is not a mount point, the second one
means it's gone. Do ignore both of these.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Since commit 17173efbe0 checkSwarmLockedToUnlocked() no longer
require its third argument, so remove it.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
`d.Sock()` already returns the socket-path including the
`unix://` scheme.
Also removed `--iptables=false`, as it didn't really seem
nescessary for this test.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
4MB client side limit was introduced in vendoring go-grpc#1165 (v1.4.0)
making these requests likely to produce errors
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
The `aufs` storage driver is deprecated in favor of `overlay2`, and will
be removed in a future release. Users of the `aufs` storage driver are
recommended to migrate to a different storage driver, such as `overlay2`, which
is now the default storage driver.
The `aufs` storage driver facilitates running Docker on distros that have no
support for OverlayFS, such as Ubuntu 14.04 LTS, which originally shipped with
a 3.14 kernel.
Now that Ubuntu 14.04 is no longer a supported distro for Docker, and `overlay2`
is available to all supported distros (as they are either on kernel 4.x, or have
support for multiple lowerdirs backported), there is no reason to continue
maintenance of the `aufs` storage driver.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This feature was added in 514ce73391,
and was merged after API v1.39 shipped as part of the Docker 18.09
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This feature was added in 14da20f5e7,
and was merged after API v1.39 shipped as part of the Docker 18.09
release candidates.
This commit moves the feature to the correct API version.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Currently, the behaviour for the version field in ServiceUpdate()
is vague. Without an correct version number, users are unable to
successfully run ServiceUpdate(), which is a pretty critical method
for scaling services (for example). I've just added an extra sentence
explaining what the version number is for, and where to find it.
Signed-off-by: Harrison Turton <harrisonturton@gmail.com>
For some reason, shared mount propagation between the host
and a container does not work for btrfs, unless container
root directory (i.e. graphdriver home) is a bind mount.
The above issue was reproduced on SLES 12sp3 + btrfs using
the following script:
#!/bin/bash
set -eux -o pipefail
# DIR should not be under a subvolume
DIR=${DIR:-/lib}
MNT=$DIR/my-mnt
FILE=$MNT/file
ID=$(docker run -d --privileged -v $DIR:$DIR:rshared ubuntu sleep 24h)
docker exec $ID mkdir -p $MNT
docker exec $ID mount -t tmpfs tmpfs $MNT
docker exec $ID touch $FILE
ls -l $FILE
umount $MNT
docker rm -f $ID
which fails this way:
+ ls -l /lib/my-mnt/file
ls: cannot access '/lib/my-mnt/file': No such file or directory
meaning the mount performed inside a priviledged container is not
propagated back to the host (even if all the mounts have "shared"
propagation mode).
The remedy to the above is to make graphdriver home a bind mount.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
This function ensures the argument is the mount point
(i.e. if it's not, it bind mounts it to itself).
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
1. There is no need to specify rw argument -- bind mounts are
read-write by default.
2. There is no point in parsing /proc/self/mountinfo after performing
a mount, especially if we don't check whether the fs is mounted or
not -- the only outcome from it could be an error from our mountinfo
parser, which makes no sense in this context.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Relevant changes;
- swarmkit#2681 Handle an edge case in CA rotation where we reclaim CA service from an external CA
- swarmkit#2750 Use gometalinter; switch from x/net/context -> context
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
As pointed out in https://github.com/moby/moby/issues/37970,
Docker overlay driver can't work with index=on feature of
the Linux kernel "overlay" filesystem. In case the global
default is set to "yes", Docker will fail with EBUSY when
trying to mount, like this:
> error creating overlay mount to ...../merged: device or resource busy
and the kernel log should contain something like:
> overlayfs: upperdir is in-use by another mount, mount with
> '-o index=off' to override exclusive upperdir protection.
A workaround is to set index=off in overlay kernel module
parameters, or even recompile the kernel with
CONFIG_OVERLAY_FS_INDEX=n in .config. Surely this is not
always practical or even possible.
The solution, as pointed out my Amir Goldstein (as well as
the above kernel message:) is to use 'index=off' option
when mounting.
NOTE since older (< 4.13rc1) kernels do not support "index="
overlayfs parameter, try to figure out whether the option
is supported. In case it's not possible to figure out,
assume it is not.
NOTE the default can be changed anytime (by writing to
/sys/module/overlay/parameters/index) so we need to always
use index=off.
[v2: move the detection code to Init()]
[v3: don't set index=off if stat() failed]
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Discourage users from using deprecated storage-drivers
by skipping them when automatically selecting a storage-
driver.
This change does not affect existing installations, because
existing state will take precedence.
Users can still use deprecated drivers by manually configuring
the daemon to use a specific driver.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The `overlay` storage driver is deprecated in favor of the `overlay2` storage
driver, which has all the benefits of `overlay`, without its limitations (excessive
inode consumption). The legacy `overlay` storage driver will be removed in a future
release. Users of the `overlay` storage driver should migrate to the `overlay2`
storage driver.
The legacy `overlay` storage driver allowed using overlayFS-backed filesystems
on pre 4.x kernels. Now that all supported distributions are able to run `overlay2`
(as they are either on kernel 4.x, or have support for multiple lowerdirs
backported), there is no reason to keep maintaining the `overlay` storage driver.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The `devicemapper` storage driver is deprecated in favor of `overlay2`, and will
be removed in a future release. Users of the `devicemapper` storage driver are
recommended to migrate to a different storage driver, such as `overlay2`, which
is now the default storage driver.
The `devicemapper` storage driver facilitates running Docker on older (3.x) kernels
that have no support for other storage drivers (such as overlay2, or AUFS).
Now that support for `overlay2` is added to all supported distros (as they are
either on kernel 4.x, or have support for multiple lowerdirs backported), there
is no reason to continue maintenance of the `devicemapper` storage driver.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This change is needed for 32 bit targets as the default type is int32
and the handle constants do not fit into it.
Signed-off-by: Jiri Appl <jiria@microsoft.com>
The CloudWatch Logs API defines its limits in terms of bytes, but its
inputs in terms of UTF-8 encoded strings. Byte-sequences which are not
valid UTF-8 encodings are normalized to the Unicode replacement
character U+FFFD, which is a 3-byte sequence in UTF-8. This replacement
can cause the input to grow, exceeding the API limit and causing failed
API calls.
This commit adds logic for counting the effective byte length after
normalization and splitting input without splitting valid UTF-8
byte-sequences into two invalid byte-sequences.
Fixes https://github.com/moby/moby/issues/37747
Signed-off-by: Samuel Karp <skarp@amazon.com>
client.checkResponseErr() was hanging and consuming infinite memory
when the serverResp.Body io.Reader returns infinite stream.
This commit prohibits reading more than 1MiB.
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
The stdin fifo of exec process is created in containerd side after
client calls Start. If the client calls CloseIO before Start call, the
stdin of exec process is still opened and wait for close.
For this case, client closes stdinCloseSync channel after Start.
Signed-off-by: Wei Fu <fuweid89@gmail.com>
This check was not important anymore; we're only interested if
the API returns a matching commit for each binary.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Through the env variable of the same name.
The idea here is pretty simple: I/O perf on native mounted disks
on non-Linux (notably Mac OS) is just terrible, thus making it
a real pain to develop: one has to choose between re-building
the image after every single change (eg to run a test) or just
work directly inside the same container (eg with vim, but even then
one would have to re-configure their dev container every time
it gets destroyed - containers, after all, are not supposed to
be long-lived).
Allowing to override `DOCKER_MOUNT` makes it easy for everyone
to decide what their volume/syncing strategy is; for example
one can choose to use [docker-sync](https://github.com/EugenMayer/docker-sync)
This patch won't change anything for anyone who doesn't
set the `DOCKER_MOUNT` env variable in their environment.
Signed-off-by: Jean Rouge <jer329@cornell.edu>
Using a value such as `--cpuset-mems=1-9223372036854775807` would cause
`dockerd` to run out of memory allocating a map of the values in the
validation code. Set limits to the normal limit of the number of CPUs,
and improve the error handling.
Reported by Huawei PSIRT.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
With containerd reaching 1.0, the runtime now
has a stable API, so there's no need to do a check
if the installed version matches the expected version.
Current versions of Docker now also package containerd
and runc separately, and can be _updated_ separately.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This updates the containerd dependencies to match
the versions used by the vendored containerd version
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
We check for max value for -default-addr-pool-mask-length param as 32.
But There won't be enough addresses on the overlay network. Hence we are
keeping it 29 so that we would be having atleast 8 addresses in /29 network.
Signed-off-by: selansen <elango.siva@docker.com>
As of Alpine Linux 3.3 (or 3.2?) there exists a new --no-cache
option for apk. It allows users to install packages with an index
that is updated and used on-the-fly and not cached locally.
This avoids the need to use --update and remove /var/cache/apk/*
when done installing packages.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
go1.11.1 (released 2018/10/01) includes fixes to the compiler,
documentation, go command, runtime, and the crypto/x509, encoding/json,
go/types, net, net/http, and reflect packages.
See the Go 1.11.1 milestone on our issue tracker for details:
https://github.com/golang/go/issues?q=milestone%3AGo1.11.1
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
These messages were enhanced to include the path that was
missing (in df6af282b9), but
also changed the first part of the message.
This change complicates running e2e tests with mixed versions
of the engine.
Looking at the full error message, "mount" is a bit redundant
as well, because the error message already indicates this is
about a "mount";
docker run --rm --mount type=bind,source=/no-such-thing,target=/foo busybox
docker: Error response from daemon: invalid mount config for type "bind": bind mount source path does not exist: /no-such-thing.
Removing the "mount" part from the error message, because
it was redundant, and makes cross-version testing easier :)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This call is what is used to implement `dmesg` to get kernel messages
about the host. This can leak substantial information about the host.
It is normally available to unprivileged users on the host, unless
the sysctl `kernel.dmesg_restrict = 1` is set, but this is not set
by standard on the majority of distributions. Blocking this to restrict
leaks about the configuration seems correct.
Fix#37897
See also https://googleprojectzero.blogspot.com/2018/09/a-cache-invalidation-bug-in-linux.html
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
The tests performed by integration tests TestConfigCreateWithFile and
TestSecretCreateWithFile are already covered by integration tests under
integration/config and integration/secret, respectively, except for the
use of an input file. The latter is also covered by unit tests for
config and secret commands under docker/cli, making the above
integration tests redundant.
Signed-off-by: Arash Deshmeh <adeshmeh@ca.ibm.com>
Looks like -i (together with DOCKER_INCREMENTAL_BINARY etc)
were used to get faster incremental builds.
Nowdays (since Go 1.10) this is no longer the case, as
go build cache is used [1]. Here's a quote:
> You do not have to use "go test -i" or "go build -i" or
> "go install" just to get fast incremental builds. We will
> not have to teach new users those workarounds anymore.
> Everything will just be fast.
To enable go cache between builds, add a volume for /root/.cache.
[1] https://groups.google.com/forum/#!msg/golang-dev/qfa3mHN4ZPA/X2UzjNV1BAAJ
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Adds support for sysctl options in docker services.
* Adds API plumbing for creating services with sysctl options set.
* Adds swagger.yaml documentation for new API field.
* Updates the API version history document.
* Changes executor package to make use of the Sysctls field on objects
* Includes integration test to verify that new behavior works.
Essentially, everything needed to support the equivalent of docker run's
`--sysctl` option except the CLI.
Includes a vendoring of swarmkit for proto changes to support the new
behavior.
Signed-off-by: Drew Erny <drew.erny@docker.com>
This test is the same as TestExportContainerAndImportImage, except for the output file option.
A unit test has been added to docker/cli to cover the output file option. Therefore this test can be removed.
Signed-off-by: Arash Deshmeh <adeshmeh@ca.ibm.com>
The `/etc/docker` directory is used both by the dockerd daemon
and the docker cli (if installed on the saem host as the daemon).
In situations where the `/etc/docker` directory does not exist,
and an initial `key.json` (legacy trust key) is generated (at the
default location), the `/etc/docker/` directory was created with
0700 permissions, making the directory only accessible by `root`.
Given that the `0600` permissions on the key itself already protect
it from being used by other users, the permissions of `/etc/docker`
can be less restrictive.
This patch changes the permissions for the directory to `0755`, so
that the CLI (if executed as non-root) can also access this directory.
> **NOTE**: "strictly", this patch is only needed for situations where no _custom_
> location for the trustkey is specified (not overridden with `--deprecated-key-path`),
> but setting the permissions only for the "default" case would make
> this more complicated.
```bash
make binary shell
make install
ls -la /etc/ | grep docker
dockerd
^C
ls -la /etc/ | grep docker
drwxr-xr-x 2 root root 4096 Sep 14 12:11 docker
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: John Howard <jhoward@microsoft.com>
This also adds go.etcd.io/bbolt as boltdb/bolt is no longer
maintained, and we need https://github.com/etcd-io/bbolt/pull/122 which
was merged in https://github.com/etcd-io/bbolt/releases/tag/v1.3.1-etcd.8
in order to fix https://github.com/docker/libnetwork/issues/1950.
Note that I can't entirely remove boltdb/bolt as it is still used by
other components. Still need to work my way through them.... These include
containerd/containerd (https://github.com/containerd/containerd/pull/2634),
docker/swarmkit; moby/buildkit. And probably more....
Saw this in moby ci:
> 00:22:07.582 === RUN TestServiceWithDefaultAddressPoolInit
> 00:22:08.887 --- FAIL: TestServiceWithDefaultAddressPoolInit (1.30s)
> 00:22:08.887 daemon.go:290: [d905878b35bb9] waiting for daemon to start
> 00:22:08.887 daemon.go:322: [d905878b35bb9] daemon started
> 00:22:08.888 panic: runtime error: index out of range [recovered]
> 00:22:08.889 panic: runtime error: index out of range
> 00:22:08.889
> 00:22:08.889 goroutine 360 [running]:
> 00:22:08.889 testing.tRunner.func1(0xc42069d770)
> 00:22:08.889 /usr/local/go/src/testing/testing.go:742 +0x29d
> 00:22:08.890 panic(0x85d680, 0xb615f0)
> 00:22:08.890 /usr/local/go/src/runtime/panic.go:502 +0x229
> 00:22:08.890 github.com/docker/docker/integration/network.TestServiceWithDefaultAddressPoolInit(0xc42069d770)
> 00:22:08.891 /go/src/github.com/docker/docker/integration/network/service_test.go:348 +0xb53
> .....
Apparently `out.IPAM.Config[0]` is not there, so to avoid panic, let's
check the size of `out.IPAM.Config` first.
Fixes: f7ad95cab9
[v2: add logging of data returned by NetworkInspect()]
[v3: use assert.Assert to fail immediately]
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
As per https://github.com/etcd-io/etcd/blob/fa57f7fbc787b4/Gopkg.lock
List of packages required by subset of etcd used is provided by:
go list -f '{{join .Deps "\n"}}' \
github.com/docker/docker/vendor/github.com/coreos/etcd/... \
| grep -F . | grep -v coreos/etcd | sort | uniq
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
In newer kernels, AppArmor will reject attempts to send signals to a
container because the signal originated from outside of that AppArmor
profile. Correct this by allowing all unconfined signals to be received.
Signed-off-by: Goldwyn Rodrigues <rgoldwyn@suse.com>
Signed-off-by: Aleksa Sarai <asarai@suse.de>
Addressing few review comments as part of code refactoring.
Also moved validation logic from CLI to Moby.
Signed-off-by: selansen <elango.siva@docker.com>
As in other similar drivers (jsonlog, local), use a set
(i.e. `map[whatever]struct{}`), making the code simpler.
While at it, make sure we remove the reader from the set
after calling `ProducerGone()` on it.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
We can do that now as we're no longer carrying archive/tar.
Note that latest vndr removes vendor/ subdir so we don't have to,
thus the change in hack/validate/vendor.
While at it, re-run a new vndr version to make sure everything
that should be there is.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
It's that time of year again! Go 1.11 is released, time to use it.
This commit also
* removes our archive/tar fork, since upstream archive/tar
is fixed for static builds, and osusergo build tag is set.
* removes ENV GO_VERSION from Dockerfile as it's not needed
anymore since PR #37592 is merged.
[v2: switch to beta2]
[v3: switch to beta3]
[v4: rc1]
[v5: remove ENV GO_VERSION as PR #37592 is now merged]
[v6: rc2]
[v7: final!]
[v8: use 1.11.0]
[v9: back to 1.11]
[v8: use 1.11.0]
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
We would like to use a version with .0 suffix (like 1.11.0) in
Dockerfile, so that once a .1 version is out (like 1.11.1) we
won't accidentally switch to it.
Unfortunately it's not possible to use .0 suffix currently
as it breaks the check in make.ps1. This patch fixes that.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Instead of installing golang from sources, it's easier to use
golang image which is based on Debian Stretch.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
This should eliminate a bunch of new (go-1.11 related) validation
errors telling that the code is not formatted with `gofmt -s`.
No functional change, just whitespace (i.e.
`git show --ignore-space-change` shows nothing).
Patch generated with:
> git ls-files | grep -v ^vendor/ | grep .go$ | xargs gofmt -s -w
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
reference.store.addReference fails when adding a digest reference
that already exists (regardless of the reference target). Both
callers (via reference.store.AddDigest) do check in advance, using
reference.store.Get, whether the digest reference exists before
calling AddDigest, but the reference store lock is released between
the two calls, so if another thread sets the reference in the meantime,
AddDigest may fail with
> Cannot overwrite digest ...
.
Handle this by checking that the pre-existing reference points at the
same image, i.e. that there is nothing to do, and succeeding immediately
in that case. This is even cheaper, avoids a reference.store.save() call.
(In principle, the same failure could have happened via
reference.store.AddTag, as
> Conflict: Tag %s is already set to image %s, if you want to replace it, please use -f option
but almost all callers (except for migrate/v1.Migrate, which is run
single-threaded anyway) set the "force" parameter of AddTag to true,
which makes the race invisible. This commit does not change the behavior
of that case, except for speeding it up by avoiding the
reference.store.save() call.)
The existing reference.store.Get checks are now, in a sense, redundant
as such, but their existence allows the callers to provide nice
context-dependent error messages, so this commit leaves them unchanged.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Signed-off-by: John Howard <jhoward@microsoft.com>
This is a fix for a few related scenarios where it's impossible to remove layers or containers
until the host is rebooted. Generally (or at least easiest to repro) through a forced daemon kill
while a container is running.
Possibly slightly worse than that, as following a host reboot, the scratch layer would possibly be leaked and
left on disk under the dataroot\windowsfilter directory after the container is removed.
One such example of a failure:
1. run a long running container with the --rm flag
docker run --rm -d --name test microsoft/windowsservercore powershell sleep 30
2. Force kill the daemon not allowing it to cleanup. Simulates a crash or a host power-cycle.
3. (re-)Start daemon
4. docker ps -a
PS C:\control> docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
7aff773d782b malloc "powershell start-sl…" 11 seconds ago Removal In Progress malloc
5. Try to remove
PS C:\control> docker rm 7aff
Error response from daemon: container 7aff773d782bbf35d95095369ffcb170b7b8f0e6f8f65d5aff42abf61234855d: driver "windowsfilter" failed to remove root filesystem: rename C:\control\windowsfilter\7aff773d782bbf35d95095369ffcb170b7b8f0e6f8f65d5aff42abf61234855d C:\control\windowsfilter\7aff773d782bbf35d95095369ffcb170b7b8f0e6f8f65d5aff42abf61234855d-removing: Access is denied.
PS C:\control>
Step 5 fails.
When daemon.ContainerLogs() is called with options.follow=true
(as in "docker logs --follow"), the "loggerutils.followLogs()"
function never returns (even then the logs consumer is gone).
As a result, all the resources associated with it (including
an opened file descriptor for the log file being read, two FDs
for a pipe, and two FDs for inotify watch) are never released.
If this is repeated (such as by running "docker logs --follow"
and pressing Ctrl-C a few times), this results in DoS caused by
either hitting the limit of inotify watches, or the limit of
opened files. The only cure is daemon restart.
Apparently, what happens is:
1. logs producer (a container) is gone, calling (*LogWatcher).Close()
for all its readers (daemon/logger/jsonfilelog/jsonfilelog.go:175).
2. WatchClose() is properly handled by a dedicated goroutine in
followLogs(), cancelling the context.
3. Upon receiving the ctx.Done(), the code in followLogs()
(daemon/logger/loggerutils/logfile.go#L626-L638) keeps to
send messages _synchronously_ (which is OK for now).
4. Logs consumer is gone (Ctrl-C is pressed on a terminal running
"docker logs --follow"). Method (*LogWatcher).Close() is properly
called (see daemon/logs.go:114). Since it was called before and
due to to once.Do(), nothing happens (which is kinda good, as
otherwise it will panic on closing a closed channel).
5. A goroutine (see item 3 above) keeps sending log messages
synchronously to the logWatcher.Msg channel. Since the
channel reader is gone, the channel send operation blocks forever,
and resource cleanup set up in defer statements at the beginning
of followLogs() never happens.
Alas, the fix is somewhat complicated:
1. Distinguish between close from logs producer and logs consumer.
To that effect,
- yet another channel is added to LogWatcher();
- {Watch,}Close() are renamed to {Watch,}ProducerGone();
- {Watch,}ConsumerGone() are added;
*NOTE* that ProducerGone()/WatchProducerGone() pair is ONLY needed
in order to stop ConsumerLogs(follow=true) when a container is stopped;
otherwise we're not interested in it. In other words, we're only
using it in followLogs().
2. Code that was doing (logWatcher*).Close() is modified to either call
ProducerGone() or ConsumerGone(), depending on the context.
3. Code that was waiting for WatchClose() is modified to wait for
either ConsumerGone() or ProducerGone(), or both, depending on the
context.
4. followLogs() are modified accordingly:
- context cancellation is happening on WatchProducerGone(),
and once it's received the FileWatcher is closed and waitRead()
returns errDone on EOF (i.e. log rotation handling logic is disabled);
- due to this, code that was writing synchronously to logWatcher.Msg
can be and is removed as the code above it handles this case;
- function returns once ConsumerGone is received, freeing all the
resources -- this is the bugfix itself.
While at it,
1. Let's also remove the ctx usage to simplify the code a bit.
It was introduced by commit a69a59ffc7 ("Decouple removing the
fileWatcher from reading") in order to fix a bug. The bug was actually
a deadlock in fsnotify, and the fix was just a workaround. Since then
the fsnofify bug has been fixed, and a new fsnotify was vendored in.
For more details, please see
https://github.com/moby/moby/pull/27782#issuecomment-416794490
2. Since `(*filePoller).Close()` is fixed to remove all the files
being watched, there is no need to explicitly call
fileWatcher.Remove(name) anymore, so get rid of the extra code.
Should fix https://github.com/moby/moby/issues/37391
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
This code has many return statements, for some of them the
"end logs" or "end stream" message was not printed, giving
the impression that this "for" loop never ended.
Make sure that "begin logs" is to be followed by "end logs".
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Similar to a related issue where previously, private Hyper-V networks
would each add 15 secs to the daemon startup, non-hns governed internal
networks are reported by hns as network type "internal" which is not
mapped to any network plugin (and thus we get the same plugin load retry
loop as before).
This issue hits Docker for Desktop because we setup such a network for
the Linux VM communication.
Signed-off-by: Simon Ferquel <simon.ferquel@docker.com>
Add a test case for creating a 8GB file inside a container.
Due to a bug in tar-split this was failing in Docker 18.06.
The file being created is sparse, so there's not much I/O
happening or disk space being used -- meaning the test is
fast and does not require a lot of disk space.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
In case a volume is specified via Mounts API, and SELinux is enabled,
the following error happens on container start:
> $ docker volume create testvol
> $ docker run --rm --mount source=testvol,target=/tmp busybox true
> docker: Error response from daemon: error setting label on mount
> source '': no such file or directory.
The functionality to relabel the source of a local mount specified via
Mounts API was introduced in commit 5bbf5cc and later broken by commit
e4b6adc, which removed setting mp.Source field.
With the current data structures, the host dir is already available in
v.Mountpoint, so let's just use it.
Fixes: e4b6adc
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
The code in Close() that removes the watches was not working,
because it first sets `w.closed = true` and then calls w.close(),
which starts with
```
if w.closed {
return errPollerClosed
}
```
Fix by setting w.closed only after calling w.remove() for all the
files being watched.
While at it, remove the duplicated `delete(w.watches, name)` code.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
There is no need to wait for up to 200ms in order to close
the file descriptor once the chClose is received.
This commit might reduce the chances for occasional "The process
cannot access the file because it is being used by another process"
error on Windows, where an opened file can't be removed.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Commit 5c8da2e967 updated the filtering behavior
to match container-names without having to specify the leading slash.
This change caused a regression in situations where a regex was provided as
filter, using an explicit leading slash (`--filter name=^/mycontainername`).
This fix changes the filters to match containers both with, and without the
leading slash, effectively making the leading slash optional when filtering.
With this fix, filters with and without a leading slash produce the same result:
$ docker ps --filter name=^a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
21afd6362b0c busybox "sh" 2 minutes ago Up 2 minutes a2
56e53770e316 busybox "sh" 2 minutes ago Up 2 minutes a1
$ docker ps --filter name=^/a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
21afd6362b0c busybox "sh" 2 minutes ago Up 2 minutes a2
56e53770e316 busybox "sh" 3 minutes ago Up 3 minutes a1
$ docker ps --filter name=^b
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b69003b6a6fe busybox "sh" About a minute ago Up About a minute b1
$ docker ps --filter name=^/b
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b69003b6a6fe busybox "sh" 56 seconds ago Up 54 seconds b1
$ docker ps --filter name=/a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
21afd6362b0c busybox "sh" 3 minutes ago Up 3 minutes a2
56e53770e316 busybox "sh" 4 minutes ago Up 4 minutes a1
$ docker ps --filter name=a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
21afd6362b0c busybox "sh" 3 minutes ago Up 3 minutes a2
56e53770e316 busybox "sh" 4 minutes ago Up 4 minutes a1
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
* Add cool, crazy, charming, magical and sweet as a adjectives (Aug 18)
* Add four male scientists to the list - faraday, maxwell, sutherland, and moore (Aug 21)
* Add four female scientists to the list - cannon, moser and rhodes (Aug 28)
Signed-off-by: Yadnyawalkya Tale <yadnyawalkyatale@gmail.com>
Includes fixes to the go command, linker, and the net/http, mime/multipart,
ld/macho, bytes, and strings packages. See the Go 1.10.4 milestone on the
issue tracker for details:
https://github.com/golang/go/issues?q=milestone%3AGo1.10.4
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
In case of an exception, it makes great sense to print out some
information telling where exactly it happened.
_.InvocationInfo.PositionMessage gives script name, line number,
character position and (depending on the PS version) highlights
the part where error has happened.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Since PR 11353 (commit 7804cd36ee "Filter out default mounts that
are override by user") there can be no duplicated mounts in the list,
so the check is redundant.
This should speed up container start by a nanosecond or two.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
In case a user wants to have a child reaper inside a container
(i.e. run "docker --init") AND a bind-mounted /dev, the following
error occurs:
> docker run -d -v /dev:/dev --init busybox top
> 088c96808c683077f04c4cc2711fddefe1f5970afc085d59e0baae779745a7cf
> docker: Error response from daemon: OCI runtime create failed: container_linux.go:296: starting container process caused "exec: "/dev/init": stat /dev/init: no such file or directory": unknown.
This happens because if a user-suppled /dev is provided, all the
built-in /dev/xxx mounts are filtered out.
To solve, let's move in-container init to /sbin, as the chance that
/sbin will be bind-mounted to a container is smaller than that for /dev.
While at it, let's give it more unique name (docker-init).
NOTE it still won't work for the case of bind-mounted /sbin.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
It looks like the logic of the test became wrong after commit
ae0883c ("Move TestAttachDetach to integration-cli").
The original logic was:
* (a few first steps skipped for clarity)
* send escape sequence to "attach";
* check "attach" is exiting (i.e. escape sequence works);
* check the container is still alive;
* kill the container.
Also, timeouts were big at that time, in the order of seconds.
The logic after the above mentioned commit and until now is:
* ...
* send escape sequence to "attach";
* check the container is running (why shouldn't it?);
* kill the container;
* checks that the "attach" has exited.
So, from the "let's check detach using escape sequence is working"
the test became something like "let's check that attach is gone
once we kill the container".
Let's fix the above test, also increasing the timeout waiting
for attach to exit (which fails from time to time on power CI).
Now, the second test, TestAttachDetachTruncatedID, does the exact
same thing, except it uses a truncated container ID. It does not
seem to be of much value, so let's remove it.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
* Add `--format` option to `docker node ls` [#30424](https://github.com/docker/docker/pull/30424)
* Add `--prune` option to `docker stack deploy` to remove services that are no longer defined in the docker-compose file [#31302](https://github.com/docker/docker/pull/31302)
* Add `PORTS` column for `docker service ls` when using `ingress` mode [#30813](https://github.com/docker/docker/pull/30813)
- Fix unnescessary re-deploying of tasks when environment-variables are used [#32364](https://github.com/docker/docker/pull/32364)
- Fix unnecessary re-deploying of tasks when environment-variables are used [#32364](https://github.com/docker/docker/pull/32364)
- Fix `docker stack deploy` not supporting `endpoint_mode` when deploying from a docker compose file [#32333](https://github.com/docker/docker/pull/32333)
- Proceed with startup if cluster component cannot be created to allow recovering from a broken swarm setup [#31631](https://github.com/docker/docker/pull/31631)
# This allows the test suite to be able to run without worrying about the underlying fs used by the container running the daemon (e.g. aufs-on-aufs), so long as the host running the container is running a supported fs.
# The volume will be cleaned up when the container is removed due to `--rm`.
# Note that `BIND_DIR` will already be set to `bundles` if `DOCKER_HOST` is not set (see above BIND_DIR line), in such case this will do nothing since `DOCKER_MOUNT` will already be set.
$(warning The docker client CLI has moved to github.com/docker/cli. For a dev-test cycle involving the CLI, run:${\n}DOCKER_CLI_PATH=/host/path/to/cli/binary make shell ${\n}then change the cli and compile into a binary at the same location.${\n})
the URL is not supported by the daemon, a HTTP `400 Bad Request` error message
is returned.
If you omit the version-prefix, the current version of the API (v1.39) is used.
For example, calling `/info` is the same as calling `/v1.39/info`. Using the
If you omit the version-prefix, the current version of the API (v1.40) is used.
For example, calling `/info` is the same as calling `/v1.40/info`. Using the
API without a version-prefix is deprecated and will be removed in a future release.
Engine releases in the near future should support this version of the API,
@@ -210,6 +210,43 @@ definitions:
PathInContainer:"/dev/deviceName"
CgroupPermissions:"mrw"
DeviceRequest:
type:"object"
description:"A request for devices to be sent to device drivers"
properties:
Driver:
type:"string"
example:"nvidia"
Count:
type:"integer"
example:-1
DeviceIDs:
type:"array"
items:
type:"string"
example:
- "0"
- "1"
- "GPU-fef8089b-4820-abfc-e83e-94318197576e"
Capabilities:
description:|
A list of capabilities; an OR list of AND lists of capabilities.
type:"array"
items:
type:"array"
items:
type:"string"
example:
# gpu AND nvidia AND compute
- ["gpu","nvidia","compute"]
Options:
description:|
Driver-specific options, specified as a key/value pairs. These options
are passed directly to the driver.
type:"object"
additionalProperties:
type:"string"
ThrottleDevice:
type:"object"
properties:
@@ -238,11 +275,13 @@ definitions:
- `bind` Mounts a file or directory from the host into the container. Must exist prior to creating the container.
- `volume` Creates a volume with the given name and options (or uses a pre-existing volume with the same name and options). These are **not** removed when the container is removed.
- `tmpfs` Create a tmpfs with the given options. The mount source cannot be specified for tmpfs.
- `npipe` Mounts a named pipe from the host into the container. Must exist prior to creating the container.
type:"string"
enum:
- "bind"
- "volume"
- "tmpfs"
- "npipe"
ReadOnly:
description:"Whether the mount should be read-only."
type:"boolean"
@@ -263,6 +302,10 @@ definitions:
- "rshared"
- "slave"
- "rslave"
NonRecursive:
description:"Disable recursive bind mount."
type:"boolean"
default:false
VolumeOptions:
description:"Optional configuration for the `volume` type."
type:"object"
@@ -415,14 +458,20 @@ definitions:
items:
type:"string"
example:"c 13:* rwm"
DiskQuota:
description:"Disk limit (in bytes)."
type:"integer"
format:"int64"
DeviceRequests:
description:"a list of requests for devices to be sent to device drivers"
type:"array"
items:
$ref:"#/definitions/DeviceRequest"
KernelMemory:
description:"Kernel memory limit in bytes."
type:"integer"
format:"int64"
example:209715200
KernelMemoryTCP:
description:"Hard limit for kernel TCP buffer memory (in bytes)."
type:"integer"
format:"int64"
MemoryReservation:
description:"Memory soft limit in bytes."
type:"integer"
@@ -449,9 +498,11 @@ definitions:
type:"boolean"
x-nullable:true
PidsLimit:
description:"Tune a container's pids limit. Set -1 for unlimited."
description:|
Tune a container's PIDs limit. Set `0` or `-1` for unlimited, or `null` to not change.
type:"integer"
format:"int64"
x-nullable:true
Ulimits:
description:|
A list of resource limits to set in the container. For example: `{"Name": "nofile", "Soft": 1024, "Hard": 2048}`"
@@ -634,14 +685,22 @@ definitions:
$ref:"#/definitions/Mount"
# Applicable to UNIX platforms
Capabilities:
type:"array"
description:|
A list of kernel capabilities to be available for container (this overrides the default set).
Conflicts with options 'CapAdd' and 'CapDrop'"
items:
type:"string"
CapAdd:
type:"array"
description:"A list of kernel capabilities to add to the container."
description:"A list of kernel capabilities to add to the container. Conflicts with option 'Capabilities'"
items:
type:"string"
CapDrop:
type:"array"
description:"A list of kernel capabilities to drop from the container."
description:"A list of kernel capabilities to drop from the container. Conflicts with option 'Capabilities'"
items:
type:"string"
Dns:
@@ -1082,6 +1141,7 @@ definitions:
type:"object"
additionalProperties:
type:"array"
x-nullable:true
items:
$ref:"#/definitions/PortBinding"
example:
@@ -1106,7 +1166,6 @@ definitions:
PortBinding represents a binding between a host IP address and a host
port.
type:"object"
x-nullable:true
properties:
HostIp:
description:"Host IP address that the container's port is mapped to."
@@ -1473,11 +1532,9 @@ definitions:
type:"string"
Options:
description:"Driver-specific options, specified as a map."
type:"array"
items:
type:"object"
additionalProperties:
type:"string"
type:"object"
additionalProperties:
type:"string"
NetworkContainer:
type:"object"
@@ -1513,6 +1570,31 @@ definitions:
aux:
$ref:"#/definitions/ImageID"
BuildCache:
type:"object"
properties:
ID:
type:"string"
Parent:
type:"string"
Type:
type:"string"
Description:
type:"string"
InUse:
type:"boolean"
Shared:
type:"boolean"
Size:
type:"integer"
CreatedAt:
type:"integer"
LastUsedAt:
type:"integer"
x-nullable:true
UsageCount:
type:"integer"
ImageID:
type:"object"
description:"Image ID or Digest"
@@ -2434,6 +2516,31 @@ definitions:
description:"Whether there is currently a root CA rotation in progress for the swarm"
type:"boolean"
example:false
DataPathPort:
description:|
DataPathPort specifies the data path port number for data traffic.
Acceptable port range is 1024 to 49151.
If no port is set or is set to 0, the default port (4789) is used.
type:"integer"
format:"uint32"
default:4789
example:4789
DefaultAddrPool:
description:|
Default Address Pool specifies default subnet pools for global scope networks.
type:"array"
items:
type:"string"
format:"CIDR"
example:["10.10.0.0/16","20.20.0.0/16"]
SubnetSize:
description:|
SubnetSize specifies the subnet size of the networks created from the default subnet pool
type:"integer"
format:"uint32"
maximum:29
default:24
example:24
JoinTokens:
description:|
@@ -2556,8 +2663,20 @@ definitions:
type:"object"
description:"CredentialSpec for managed service account (Windows only)"
properties:
Config:
type:"string"
example:"0bt9dmxjvjiqermk6xrop3ekq"
description:|
Load credential spec from a Swarm Config with the given ID.
The specified config must also be present in the Configs field with the Runtime property set.
<p><br /></p>
> **Note**: `CredentialSpec.File`, `CredentialSpec.Registry`, and `CredentialSpec.Config` are mutually exclusive.
File:
type:"string"
example:"spec.json"
description:|
Load credential spec from this file. The file is read by the daemon, and must be present in the
`CredentialSpecs` subdirectory in the docker data directory, which defaults to
@@ -2567,7 +2686,7 @@ definitions:
<p><br /></p>
> **Note**: `CredentialSpec.File` and `CredentialSpec.Registry` are mutually exclusive.
> **Note**: `CredentialSpec.File`, `CredentialSpec.Registry`, and `CredentialSpec.Config` are mutually exclusive.
Registry:
type:"string"
description:|
@@ -2579,7 +2698,7 @@ definitions:
<p><br /></p>
> **Note**: `CredentialSpec.File` and `CredentialSpec.Registry` are mutually exclusive.
> **Note**: `CredentialSpec.File`, `CredentialSpec.Registry`, and `CredentialSpec.Config` are mutually exclusive.
SELinuxContext:
type:"object"
description:"SELinux labels of the container"
@@ -2690,7 +2809,12 @@ definitions:
type:"object"
properties:
File:
description:"File represents a specific target that is backed by a file."
description:|
File represents a specific target that is backed by a file.
<p><br /><p>
> **Note**: `Configs.File` and `Configs.Runtime` are mutually exclusive
type:"object"
properties:
Name:
@@ -2706,6 +2830,14 @@ definitions:
description:"Mode represents the FileMode of the file."
type:"integer"
format:"uint32"
Runtime:
description:|
Runtime represents a target that is not mounted into the container but is used by the task
<p><br /><p>
> **Note**: `Configs.File` and `Configs.Runtime` are mutually exclusive
type:"object"
ConfigID:
description:"ConfigID represents the ID of the specific config that we're referencing."
type:"string"
@@ -2725,6 +2857,18 @@ definitions:
description:"Run an init inside the container that forwards signals and reaps processes. This field is omitted if empty, and the default (as configured on the daemon) is used."
type:"boolean"
x-nullable:true
Sysctls:
description:|
Set kernel namedspaced parameters (sysctls) in the container.
The Sysctls option on services accepts the same sysctls as the
are supported on containers. Note that while the same sysctls are
supported, no guarantees or checks are made about their
suitability for a clustered environment, and it's up to the user
to determine whether a given sysctl will work properly in a
Service.
type:"object"
additionalProperties:
type:"string"
NetworkAttachmentSpec:
description:|
Read-only spec type for non-swarm containers attached to swarm overlay
@@ -2805,6 +2949,11 @@ definitions:
SpreadDescriptor:"node.labels.datacenter"
- Spread:
SpreadDescriptor:"node.labels.rack"
MaxReplicas:
description:"Maximum number of replicas for per node (default value is 0, which is unlimited)"
type:"integer"
format:"int64"
default:0
Platforms:
description:|
Platforms stores all the platforms that the service's image can
@@ -3113,7 +3262,7 @@ definitions:
<p><br /></p>
- "ingress" makes the target port accessible on on every node,
- "ingress" makes the target port accessible on every node,
regardless of whether there is a task for the service running on
that node or not.
- "host" bypasses the routing mesh and publish the port directly on
@@ -3605,6 +3754,10 @@ definitions:
See [cpuset(7)](https://www.kernel.org/doc/Documentation/cgroup-v1/cpusets.txt)
type:"boolean"
example:true
PidsLimit:
description:"Indicates if the host kernel has PID limit support enabled."
type:"boolean"
example:true
OomKillDisable:
description:"Indicates if OOM killer disable is supported on the host."
type:"boolean"
@@ -3652,7 +3805,7 @@ definitions:
description:|
The driver to use for managing cgroups.
type:"string"
enum:["cgroupfs","systemd"]
enum:["cgroupfs","systemd","none"]
default:"cgroupfs"
example:"cgroupfs"
NEventsListener:
@@ -3722,18 +3875,22 @@ definitions:
description:|
HTTP-proxy configured for the daemon. This value is obtained from the
Comma-separated list of domain extensions for which no proxy should be
@@ -3823,10 +3980,10 @@ definitions:
$ref:"#/definitions/Runtime"
default:
runc:
path:"docker-runc"
path:"runc"
example:
runc:
path:"docker-runc"
path:"runc"
runc-master:
path:"/go/bin/runc"
custom:
@@ -3883,7 +4040,7 @@ definitions:
SecurityOptions:
description:|
List of security features that are enabled on the daemon, such as
apparmor, seccomp, SELinux, and user-namespaces (userns).
apparmor, seccomp, SELinux, user-namespaces (userns), and rootless.
Additional configuration options for each security feature may
be present, and are included as a comma-separated list of key/value
@@ -3896,6 +4053,7 @@ definitions:
- "name=seccomp,profile=default"
- "name=selinux"
- "name=userns"
- "name=rootless"
ProductLicense:
description:|
Reports a summary of the product license on the daemon.
@@ -4450,9 +4608,9 @@ paths:
parameters:
- name:"name"
in:"query"
description:"Assign the specified name to the container. Must match `/?[a-zA-Z0-9_-]+`."
description:"Assign the specified name to the container. Must match `/?[a-zA-Z0-9][a-zA-Z0-9_.-]+`."
type:"string"
pattern:"/?[a-zA-Z0-9_-]+"
pattern:"^/?[a-zA-Z0-9][a-zA-Z0-9_.-]+$"
- name:"body"
in:"body"
description:"Container to create"
@@ -4537,7 +4695,7 @@ paths:
OomKillDisable:false
OomScoreAdj:500
PidMode:""
PidsLimit:-1
PidsLimit:0
PortBindings:
22/tcp:
- HostPort:"11022"
@@ -5016,15 +5174,15 @@ paths:
Note: This endpoint works only for containers with the `json-file` or `journald` logging driver.
operationId:"ContainerLogs"
responses:
101:
description:"logs returned as a stream"
200:
description:|
logs returned as a stream in response body.
For the stream format, [see the documentation for the attach endpoint](#operation/ContainerAttach).
Note that unlike the attach endpoint, the logs endpoint does not upgrade the connection and does not
set Content-Type.
schema:
type:"string"
format:"binary"
200:
description:"logs returned as a string in response body"
schema:
type:"string"
404:
description:"no such container"
schema:
@@ -5044,10 +5202,7 @@ paths:
type:"string"
- name:"follow"
in:"query"
description:|
Return the logs as a stream.
This will return a `101` HTTP response with a `Connection: upgrade` header, then hijack the HTTP connection to send raw output. For more information about hijacking and the stream format, [see the documentation for the attach endpoint](#operation/ContainerAttach).
description:"Keep connection after returning logs."
type:"boolean"
default:false
- name:"stdout"
@@ -5308,7 +5463,7 @@ paths:
/containers/{id}/resize:
post:
summary:"Resize a container TTY"
description:"Resize the TTY for a container. You must restart the container for the resize to take effect."
description:"Resize the TTY for a container."
operationId:"ContainerResize"
consumes:
- "application/octet-stream"
@@ -6062,12 +6217,17 @@ paths:
in:"query"
description:"If “1”, “true”, or “True” then it will be an error if unpacking the given content would cause an existing directory to be replaced with a non-directory and vice versa."
type:"string"
- name:"copyUIDGID"
in:"query"
description:"If “1”, “true”, then it will copy UID/GID maps to the dest file or dir"
type:"string"
- name:"inputStream"
in:"body"
required:true
description:"The input stream must be a tar archive compressed with one of the following algorithms: identity (no compression), gzip, bzip2, xz."
schema:
type:"string"
format:"binary"
tags:["Container"]
/containers/prune:
post:
@@ -6340,6 +6500,11 @@ paths:
description:"Target build stage"
type:"string"
default:""
- name:"outputs"
in:"query"
description:"BuildKit output configuration"
type:"string"
default:""
responses:
200:
description:"no error"
@@ -6358,6 +6523,29 @@ paths:
produces:
- "application/json"
operationId:"BuildPrune"
parameters:
- name:"keep-storage"
in:"query"
description:"Amount of disk space in bytes to keep for cache"
type:"integer"
format:"int64"
- name:"all"
in:"query"
type:"boolean"
description:"Remove all types of build cache"
- name:"filters"
in:"query"
type:"string"
description:|
A JSON encoded value of the filters (a `map[string][]string`) to process on the list of build cache objects. Available filters:
- `until=<duration>`: duration relative to daemon's time, during which build cache was not used, in Go's duration format (e.g., '24h')
- `id=<id>`
- `parent=<id>`
- `type=<string>`
- `description=<string>`
- `inuse`
- `shared`
- `private`
responses:
200:
description:"No error"
@@ -6365,6 +6553,11 @@ paths:
type:"object"
title:"BuildPruneResponse"
properties:
CachesDeleted:
type:"array"
items:
description:"ID of build cache object"
type:"string"
SpaceReclaimed:
description:"Disk space reclaimed in bytes"
type:"integer"
@@ -6999,6 +7192,51 @@ paths:
Docker-Experimental:
type:"boolean"
description:"If the server is running with experimental mode enabled"
Cache-Control:
type:"string"
default:"no-cache, no-store, must-revalidate"
Pragma:
type:"string"
default:"no-cache"
500:
description:"server error"
schema:
$ref:"#/definitions/ErrorResponse"
headers:
Cache-Control:
type:"string"
default:"no-cache, no-store, must-revalidate"
Pragma:
type:"string"
default:"no-cache"
tags:["System"]
head:
summary:"Ping"
description:"This is a dummy endpoint you can use to test if the server is accessible."
operationId:"SystemPingHead"
produces:["text/plain"]
responses:
200:
description:"no error"
schema:
type:"string"
example:"(empty)"
headers:
API-Version:
type:"string"
description:"Max API Version the server supports"
BuildKit-Version:
type:"string"
description:"Default version of docker image builder"
Docker-Experimental:
type:"boolean"
description:"If the server is running with experimental mode enabled"
Cache-Control:
type:"string"
default:"no-cache, no-store, must-revalidate"
Pragma:
type:"string"
default:"no-cache"
500:
description:"server error"
schema:
@@ -7199,6 +7437,10 @@ paths:
type:"array"
items:
$ref:"#/definitions/Volume"
BuildCache:
type:"array"
items:
$ref:"#/definitions/BuildCache"
example:
LayersSize:1092588
Images:
@@ -7892,6 +8134,10 @@ paths:
description:|
JSON encoded value of the filters (a `map[string][]string`) to process on the networks list. Available filters:
- `dangling=<boolean>` When set to `true` (or `1`), returns all
networks that are not in use by a container. When set to `false`
(or `0`), only networks that are in use by one or more
containers are returned.
- `driver=<driver-name>` Matches a network's driver.
- `id=<network-id>` Matches all or part of a network ID.
- `label=<key>` or `label=<key>=<value>` of a network label.
@@ -8609,6 +8855,7 @@ paths:
- `label=<engine label>`
- `membership=`(`accepted`|`pending`)`
- `name=<node name>`
- `node.label=<node label>`
- `role=`(`manager`|`worker`)`
type:"string"
tags:["Node"]
@@ -8781,6 +9028,13 @@ paths:
nodes in order to reach the containers running on this node. Using this parameter it is possible to
separate the container data traffic from the management traffic of the cluster.
type:"string"
DataPathPort:
description:|
DataPathPort specifies the data path port number for data traffic.
Acceptable port range is 1024 to 49151.
if no port is set or is set to 0, default port 4789 will be used.
type:"integer"
format:"uint32"
DefaultAddrPool:
description:|
Default Address Pool specifies default subnet pools for global scope networks.
@@ -8801,6 +9055,7 @@ paths:
example:
ListenAddr:"0.0.0.0:2377"
AdvertiseAddr:"192.168.1.1:2377"
DataPathPort:4789
DefaultAddrPool:["10.10.0.0/8","20.20.0.0/8"]
SubnetSize:24
ForceNewCluster:false
@@ -8857,7 +9112,9 @@ paths:
type:"string"
RemoteAddrs:
description:"Addresses of manager nodes already participating in the swarm."
type:"string"
type:"array"
items:
type:"string"
JoinToken:
description:"Secret token for joining this swarm."
type:"string"
@@ -9284,7 +9541,10 @@ paths:
- name:"version"
in:"query"
description:"The version number of the service object being updated. This is required to avoid conflicting writes."
description:"The version number of the service object being updated.
This is required to avoid conflicting writes.
This version number should be the value as currently set on the service *before* the update.
You can find the current version by calling `GET /services/{id}`"
required:true
type:"integer"
- name:"registryAuthFrom"
@@ -9310,23 +9570,16 @@ paths:
get:
summary:"Get service logs"
description:|
Get `stdout` and `stderr` logs from a service.
Get `stdout` and `stderr` logs from a service. See also [`/containers/{id}/logs`](#operation/ContainerLogs).
**Note**: This endpoint works only for services with the `json-file` or `journald` logging drivers.
**Note**: This endpoint works only for services with the `local`, `json-file` or `journald` logging drivers.
operationId:"ServiceLogs"
produces:
- "application/vnd.docker.raw-stream"
- "application/json"
responses:
101:
description:"logs returned as a stream"
200:
description:"logs returned as a stream in response body"
schema:
type:"string"
format:"binary"
200:
description:"logs returned as a string in response body"
schema:
type:"string"
404:
description:"no such service"
schema:
@@ -9355,10 +9608,7 @@ paths:
default:false
- name:"follow"
in:"query"
description:|
Return the logs as a stream.
This will return a `101` HTTP response with a `Connection: upgrade` header, then hijack the HTTP connection to send raw output. For more information about hijacking and the stream format, [see the documentation for the attach endpoint](#operation/ContainerAttach).
description:"Keep connection after returning logs."
type:"boolean"
default:false
- name:"stdout"
@@ -9567,23 +9817,16 @@ paths:
get:
summary:"Get task logs"
description:|
Get `stdout` and `stderr` logs from a task.
Get `stdout` and `stderr` logs from a task. See also [`/containers/{id}/logs`](#operation/ContainerLogs).
**Note**: This endpoint works only for services with the `json-file` or `journald` logging drivers.
**Note**: This endpoint works only for services with the `local`, `json-file` or `journald` logging drivers.
operationId:"TaskLogs"
produces:
- "application/vnd.docker.raw-stream"
- "application/json"
responses:
101:
description:"logs returned as a stream"
200:
description:"logs returned as a stream in response body"
schema:
type:"string"
format:"binary"
200:
description:"logs returned as a string in response body"
schema:
type:"string"
404:
description:"no such task"
schema:
@@ -9612,10 +9855,7 @@ paths:
default:false
- name:"follow"
in:"query"
description:|
Return the logs as a stream.
This will return a `101` HTTP response with a `Connection: upgrade` header, then hijack the HTTP connection to send raw output. For more information about hijacking and the stream format, [see the documentation for the attach endpoint](#operation/ContainerAttach).
description:"Keep connection after returning logs."
type:"boolean"
default:false
- name:"stdout"
@@ -9643,6 +9883,7 @@ paths:
description:"Only return this number of log lines from the end of the logs. Specify as an integer or `all` to output all log lines."
type:"string"
default:"all"
tags:["Task"]
/secrets:
get:
summary:"List secrets"
@@ -10138,9 +10379,6 @@ paths:
description:|
Start a new interactive session with a server. Session allows server to call back to the client for advanced capabilities.
> **Note**: This endpoint is *experimental* and only available if the daemon is started with experimental
> features enabled. The specifications for this endpoint may still change in a future version of the API.
### Hijacking
This endpoint hijacks the HTTP connection to HTTP2 transport that allows the client to expose gPRC services on that connection.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
