diff --git a/integration/network/bridge/iptablesdoc/generated/new-daemon.md b/integration/network/bridge/iptablesdoc/generated/new-daemon.md index a51986ec78..ae2e8cbb75 100644 --- a/integration/network/bridge/iptablesdoc/generated/new-daemon.md +++ b/integration/network/bridge/iptablesdoc/generated/new-daemon.md @@ -20,11 +20,19 @@ Table `filter`: num pkts bytes target prot opt in out source destination 1 0 0 DROP 0 -- !docker0 docker0 0.0.0.0/0 0.0.0.0/0 + Chain DOCKER-BRIDGE (1 references) + num pkts bytes target prot opt in out source destination + 1 0 0 DOCKER 0 -- * docker0 0.0.0.0/0 0.0.0.0/0 + + Chain DOCKER-CT (1 references) + num pkts bytes target prot opt in out source destination + 1 0 0 ACCEPT 0 -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED + Chain DOCKER-FORWARD (1 references) num pkts bytes target prot opt in out source destination - 1 0 0 ACCEPT 0 -- * * 0.0.0.0/0 0.0.0.0/0 match-set docker-ext-bridges-v4 dst ctstate RELATED,ESTABLISHED + 1 0 0 DOCKER-CT 0 -- * * 0.0.0.0/0 0.0.0.0/0 2 0 0 DOCKER-ISOLATION-STAGE-1 0 -- * * 0.0.0.0/0 0.0.0.0/0 - 3 0 0 DOCKER 0 -- * * 0.0.0.0/0 0.0.0.0/0 match-set docker-ext-bridges-v4 dst + 3 0 0 DOCKER-BRIDGE 0 -- * * 0.0.0.0/0 0.0.0.0/0 4 0 0 ACCEPT 0 -- docker0 * 0.0.0.0/0 0.0.0.0/0 Chain DOCKER-ISOLATION-STAGE-1 (1 references) @@ -47,6 +55,8 @@ Table `filter`: -P FORWARD ACCEPT -P OUTPUT ACCEPT -N DOCKER + -N DOCKER-BRIDGE + -N DOCKER-CT -N DOCKER-FORWARD -N DOCKER-ISOLATION-STAGE-1 -N DOCKER-ISOLATION-STAGE-2 @@ -54,9 +64,11 @@ Table `filter`: -A FORWARD -j DOCKER-USER -A FORWARD -j DOCKER-FORWARD -A DOCKER ! -i docker0 -o docker0 -j DROP - -A DOCKER-FORWARD -m set --match-set docker-ext-bridges-v4 dst -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT + -A DOCKER-BRIDGE -o docker0 -j DOCKER + -A DOCKER-CT -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT + -A DOCKER-FORWARD -j DOCKER-CT -A DOCKER-FORWARD -j DOCKER-ISOLATION-STAGE-1 - -A DOCKER-FORWARD -m set --match-set docker-ext-bridges-v4 dst -j DOCKER + -A DOCKER-FORWARD -j DOCKER-BRIDGE -A DOCKER-FORWARD -i docker0 -j ACCEPT -A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2 -A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP @@ -75,7 +87,7 @@ The FORWARD chain's policy shown above is ACCEPT. However: [1]: https://github.com/moby/moby/blob/cff4f20c44a3a7c882ed73934dec6a77246c6323/libnetwork/drivers/bridge/setup_ip_forwarding.go#L44 -The FORWARD chain rules are numbered in the output above, they are: +The FORWARD chain rules, explained in the order they appear in the output above, are: 1. Unconditional jump to DOCKER-USER. This is set up by libnetwork, in [setupUserChain][10]. @@ -91,24 +103,27 @@ the DOCKER-USER chain, for rules that run before DOCKER's). The DOCKER-FORWARD chain contains the first stage of Docker's filter rules. Initial rules are inserted at the top of the table, then not touched. Per-network rules -are appended. +are appended. The DOCKER-FORWARD chain rules, explained in the order they appear in +the output above, are: - 1. Early ACCEPT for any RELATED,ESTABLISHED traffic to a docker bridge. This rule - matches against an `ipset` called `docker-ext-bridges-v4` (`v6` for IPv6). The - set contains the CIDR address of each docker network, and it is updated as networks - are created and deleted. This rule is created during driver initialisation, in - `setupIPChains`. + 1. Unconditional jump to DOCKER-CT. + Created during driver initialisation, in `setupIPChains`. 2. Unconditional jump to DOCKER-ISOLATION-STAGE-1. Also created during driver initialisation, in `setupIPChains`. - 3. Jump to DOCKER, for any packet destined for any bridge network, identified by - matching against the `docker-ext-bridge-v[46]` set. + 3. Unconditional jump to DOCKER-BRIDGE. Also created during driver initialisation, in `setupIPChains`. - The DOCKER chain implements per-port/protocol filtering for each container. 4. ACCEPT any packet leaving a network, set up when the network is created, in `setupIPTablesInternal`. Note that this accepts any packet leaving the network that's made it through the DOCKER and isolation chains, whether the destination is external or another network. +The DOCKER-CT chain is an early ACCEPT for any RELATED,ESTABLISHED traffic to a +docker bridge. It contains a conntrack ACCEPT rule for each bridge network. + +DOCKER-BRIDGE has a rule for each bridge network, to jump to the DOCKER chain. + +The DOCKER chain implements per-port/protocol filtering for each container. + [10]: https://github.com/moby/moby/blob/e05848c0025b67a16aaafa8cdff95d5e2c064105/libnetwork/firewall_linux.go#L50 [11]: https://github.com/robmry/moby/blob/52c89d467fc5326149e4bbb8903d23589b66ff0d/libnetwork/drivers/bridge/setup_ip_tables_linux.go#L230-L232 [12]: https://github.com/robmry/moby/blob/52c89d467fc5326149e4bbb8903d23589b66ff0d/libnetwork/drivers/bridge/setup_ip_tables_linux.go#L227-L229 diff --git a/integration/network/bridge/iptablesdoc/generated/swarm-portmap.md b/integration/network/bridge/iptablesdoc/generated/swarm-portmap.md index 978031a300..8eeeb4631b 100644 --- a/integration/network/bridge/iptablesdoc/generated/swarm-portmap.md +++ b/integration/network/bridge/iptablesdoc/generated/swarm-portmap.md @@ -18,16 +18,26 @@ The filter table is: Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes) num pkts bytes target prot opt in out source destination - Chain DOCKER (1 references) + Chain DOCKER (2 references) num pkts bytes target prot opt in out source destination 1 0 0 DROP 0 -- !docker0 docker0 0.0.0.0/0 0.0.0.0/0 2 0 0 DROP 0 -- !docker_gwbridge docker_gwbridge 0.0.0.0/0 0.0.0.0/0 + Chain DOCKER-BRIDGE (1 references) + num pkts bytes target prot opt in out source destination + 1 0 0 DOCKER 0 -- * docker0 0.0.0.0/0 0.0.0.0/0 + 2 0 0 DOCKER 0 -- * docker_gwbridge 0.0.0.0/0 0.0.0.0/0 + + Chain DOCKER-CT (1 references) + num pkts bytes target prot opt in out source destination + 1 0 0 ACCEPT 0 -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED + 2 0 0 ACCEPT 0 -- * docker_gwbridge 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED + Chain DOCKER-FORWARD (1 references) num pkts bytes target prot opt in out source destination - 1 0 0 ACCEPT 0 -- * * 0.0.0.0/0 0.0.0.0/0 match-set docker-ext-bridges-v4 dst ctstate RELATED,ESTABLISHED + 1 0 0 DOCKER-CT 0 -- * * 0.0.0.0/0 0.0.0.0/0 2 0 0 DOCKER-ISOLATION-STAGE-1 0 -- * * 0.0.0.0/0 0.0.0.0/0 - 3 0 0 DOCKER 0 -- * * 0.0.0.0/0 0.0.0.0/0 match-set docker-ext-bridges-v4 dst + 3 0 0 DOCKER-BRIDGE 0 -- * * 0.0.0.0/0 0.0.0.0/0 4 0 0 ACCEPT 0 -- docker0 * 0.0.0.0/0 0.0.0.0/0 5 0 0 DROP 0 -- docker_gwbridge docker_gwbridge 0.0.0.0/0 0.0.0.0/0 6 0 0 ACCEPT 0 -- docker_gwbridge !docker_gwbridge 0.0.0.0/0 0.0.0.0/0 @@ -60,6 +70,8 @@ The filter table is: -P FORWARD ACCEPT -P OUTPUT ACCEPT -N DOCKER + -N DOCKER-BRIDGE + -N DOCKER-CT -N DOCKER-FORWARD -N DOCKER-INGRESS -N DOCKER-ISOLATION-STAGE-1 @@ -70,9 +82,13 @@ The filter table is: -A FORWARD -j DOCKER-FORWARD -A DOCKER ! -i docker0 -o docker0 -j DROP -A DOCKER ! -i docker_gwbridge -o docker_gwbridge -j DROP - -A DOCKER-FORWARD -m set --match-set docker-ext-bridges-v4 dst -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT + -A DOCKER-BRIDGE -o docker0 -j DOCKER + -A DOCKER-BRIDGE -o docker_gwbridge -j DOCKER + -A DOCKER-CT -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT + -A DOCKER-CT -o docker_gwbridge -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT + -A DOCKER-FORWARD -j DOCKER-CT -A DOCKER-FORWARD -j DOCKER-ISOLATION-STAGE-1 - -A DOCKER-FORWARD -m set --match-set docker-ext-bridges-v4 dst -j DOCKER + -A DOCKER-FORWARD -j DOCKER-BRIDGE -A DOCKER-FORWARD -i docker0 -j ACCEPT -A DOCKER-FORWARD -i docker_gwbridge -o docker_gwbridge -j DROP -A DOCKER-FORWARD -i docker_gwbridge ! -o docker_gwbridge -j ACCEPT diff --git a/integration/network/bridge/iptablesdoc/generated/usernet-internal.md b/integration/network/bridge/iptablesdoc/generated/usernet-internal.md index ab6fda90a8..9cb3b41436 100644 --- a/integration/network/bridge/iptablesdoc/generated/usernet-internal.md +++ b/integration/network/bridge/iptablesdoc/generated/usernet-internal.md @@ -35,11 +35,19 @@ The filter table is updated as follows: num pkts bytes target prot opt in out source destination 1 0 0 DROP 0 -- !docker0 docker0 0.0.0.0/0 0.0.0.0/0 + Chain DOCKER-BRIDGE (1 references) + num pkts bytes target prot opt in out source destination + 1 0 0 DOCKER 0 -- * docker0 0.0.0.0/0 0.0.0.0/0 + + Chain DOCKER-CT (1 references) + num pkts bytes target prot opt in out source destination + 1 0 0 ACCEPT 0 -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED + Chain DOCKER-FORWARD (1 references) num pkts bytes target prot opt in out source destination - 1 0 0 ACCEPT 0 -- * * 0.0.0.0/0 0.0.0.0/0 match-set docker-ext-bridges-v4 dst ctstate RELATED,ESTABLISHED + 1 0 0 DOCKER-CT 0 -- * * 0.0.0.0/0 0.0.0.0/0 2 0 0 DOCKER-ISOLATION-STAGE-1 0 -- * * 0.0.0.0/0 0.0.0.0/0 - 3 0 0 DOCKER 0 -- * * 0.0.0.0/0 0.0.0.0/0 match-set docker-ext-bridges-v4 dst + 3 0 0 DOCKER-BRIDGE 0 -- * * 0.0.0.0/0 0.0.0.0/0 4 0 0 ACCEPT 0 -- docker0 * 0.0.0.0/0 0.0.0.0/0 5 0 0 ACCEPT 0 -- bridgeICC bridgeICC 0.0.0.0/0 0.0.0.0/0 6 0 0 DROP 0 -- bridgeNoICC bridgeNoICC 0.0.0.0/0 0.0.0.0/0 @@ -68,6 +76,8 @@ The filter table is updated as follows: -P FORWARD ACCEPT -P OUTPUT ACCEPT -N DOCKER + -N DOCKER-BRIDGE + -N DOCKER-CT -N DOCKER-FORWARD -N DOCKER-ISOLATION-STAGE-1 -N DOCKER-ISOLATION-STAGE-2 @@ -75,9 +85,11 @@ The filter table is updated as follows: -A FORWARD -j DOCKER-USER -A FORWARD -j DOCKER-FORWARD -A DOCKER ! -i docker0 -o docker0 -j DROP - -A DOCKER-FORWARD -m set --match-set docker-ext-bridges-v4 dst -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT + -A DOCKER-BRIDGE -o docker0 -j DOCKER + -A DOCKER-CT -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT + -A DOCKER-FORWARD -j DOCKER-CT -A DOCKER-FORWARD -j DOCKER-ISOLATION-STAGE-1 - -A DOCKER-FORWARD -m set --match-set docker-ext-bridges-v4 dst -j DOCKER + -A DOCKER-FORWARD -j DOCKER-BRIDGE -A DOCKER-FORWARD -i docker0 -j ACCEPT -A DOCKER-FORWARD -i bridgeICC -o bridgeICC -j ACCEPT -A DOCKER-FORWARD -i bridgeNoICC -o bridgeNoICC -j DROP diff --git a/integration/network/bridge/iptablesdoc/generated/usernet-portmap-lo.md b/integration/network/bridge/iptablesdoc/generated/usernet-portmap-lo.md index c0f24a253e..62b4607de3 100644 --- a/integration/network/bridge/iptablesdoc/generated/usernet-portmap-lo.md +++ b/integration/network/bridge/iptablesdoc/generated/usernet-portmap-lo.md @@ -23,17 +23,27 @@ The filter and nat tables are identical to [nat mode][0]: Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes) num pkts bytes target prot opt in out source destination - Chain DOCKER (1 references) + Chain DOCKER (2 references) num pkts bytes target prot opt in out source destination 1 0 0 ACCEPT 6 -- !bridge1 bridge1 0.0.0.0/0 192.0.2.2 tcp dpt:80 2 0 0 DROP 0 -- !docker0 docker0 0.0.0.0/0 0.0.0.0/0 3 0 0 DROP 0 -- !bridge1 bridge1 0.0.0.0/0 0.0.0.0/0 + Chain DOCKER-BRIDGE (1 references) + num pkts bytes target prot opt in out source destination + 1 0 0 DOCKER 0 -- * docker0 0.0.0.0/0 0.0.0.0/0 + 2 0 0 DOCKER 0 -- * bridge1 0.0.0.0/0 0.0.0.0/0 + + Chain DOCKER-CT (1 references) + num pkts bytes target prot opt in out source destination + 1 0 0 ACCEPT 0 -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED + 2 0 0 ACCEPT 0 -- * bridge1 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED + Chain DOCKER-FORWARD (1 references) num pkts bytes target prot opt in out source destination - 1 0 0 ACCEPT 0 -- * * 0.0.0.0/0 0.0.0.0/0 match-set docker-ext-bridges-v4 dst ctstate RELATED,ESTABLISHED + 1 0 0 DOCKER-CT 0 -- * * 0.0.0.0/0 0.0.0.0/0 2 0 0 DOCKER-ISOLATION-STAGE-1 0 -- * * 0.0.0.0/0 0.0.0.0/0 - 3 0 0 DOCKER 0 -- * * 0.0.0.0/0 0.0.0.0/0 match-set docker-ext-bridges-v4 dst + 3 0 0 DOCKER-BRIDGE 0 -- * * 0.0.0.0/0 0.0.0.0/0 4 0 0 ACCEPT 0 -- docker0 * 0.0.0.0/0 0.0.0.0/0 5 0 0 ACCEPT 0 -- bridge1 * 0.0.0.0/0 0.0.0.0/0 @@ -56,6 +66,8 @@ The filter and nat tables are identical to [nat mode][0]: -P FORWARD ACCEPT -P OUTPUT ACCEPT -N DOCKER + -N DOCKER-BRIDGE + -N DOCKER-CT -N DOCKER-FORWARD -N DOCKER-ISOLATION-STAGE-1 -N DOCKER-ISOLATION-STAGE-2 @@ -65,9 +77,13 @@ The filter and nat tables are identical to [nat mode][0]: -A DOCKER -d 192.0.2.2/32 ! -i bridge1 -o bridge1 -p tcp -m tcp --dport 80 -j ACCEPT -A DOCKER ! -i docker0 -o docker0 -j DROP -A DOCKER ! -i bridge1 -o bridge1 -j DROP - -A DOCKER-FORWARD -m set --match-set docker-ext-bridges-v4 dst -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT + -A DOCKER-BRIDGE -o docker0 -j DOCKER + -A DOCKER-BRIDGE -o bridge1 -j DOCKER + -A DOCKER-CT -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT + -A DOCKER-CT -o bridge1 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT + -A DOCKER-FORWARD -j DOCKER-CT -A DOCKER-FORWARD -j DOCKER-ISOLATION-STAGE-1 - -A DOCKER-FORWARD -m set --match-set docker-ext-bridges-v4 dst -j DOCKER + -A DOCKER-FORWARD -j DOCKER-BRIDGE -A DOCKER-FORWARD -i docker0 -j ACCEPT -A DOCKER-FORWARD -i bridge1 -j ACCEPT -A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2 diff --git a/integration/network/bridge/iptablesdoc/generated/usernet-portmap-natunprot.md b/integration/network/bridge/iptablesdoc/generated/usernet-portmap-natunprot.md index 3b7118e732..cf2eff06bc 100644 --- a/integration/network/bridge/iptablesdoc/generated/usernet-portmap-natunprot.md +++ b/integration/network/bridge/iptablesdoc/generated/usernet-portmap-natunprot.md @@ -21,16 +21,26 @@ The filter table is: Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes) num pkts bytes target prot opt in out source destination - Chain DOCKER (1 references) + Chain DOCKER (2 references) num pkts bytes target prot opt in out source destination 1 0 0 DROP 0 -- !docker0 docker0 0.0.0.0/0 0.0.0.0/0 2 0 0 ACCEPT 0 -- !bridge1 bridge1 0.0.0.0/0 0.0.0.0/0 + Chain DOCKER-BRIDGE (1 references) + num pkts bytes target prot opt in out source destination + 1 0 0 DOCKER 0 -- * docker0 0.0.0.0/0 0.0.0.0/0 + 2 0 0 DOCKER 0 -- * bridge1 0.0.0.0/0 0.0.0.0/0 + + Chain DOCKER-CT (1 references) + num pkts bytes target prot opt in out source destination + 1 0 0 ACCEPT 0 -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED + 2 0 0 ACCEPT 0 -- * bridge1 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED + Chain DOCKER-FORWARD (1 references) num pkts bytes target prot opt in out source destination - 1 0 0 ACCEPT 0 -- * * 0.0.0.0/0 0.0.0.0/0 match-set docker-ext-bridges-v4 dst ctstate RELATED,ESTABLISHED + 1 0 0 DOCKER-CT 0 -- * * 0.0.0.0/0 0.0.0.0/0 2 0 0 DOCKER-ISOLATION-STAGE-1 0 -- * * 0.0.0.0/0 0.0.0.0/0 - 3 0 0 DOCKER 0 -- * * 0.0.0.0/0 0.0.0.0/0 match-set docker-ext-bridges-v4 dst + 3 0 0 DOCKER-BRIDGE 0 -- * * 0.0.0.0/0 0.0.0.0/0 4 0 0 ACCEPT 0 -- docker0 * 0.0.0.0/0 0.0.0.0/0 5 0 0 ACCEPT 0 -- bridge1 * 0.0.0.0/0 0.0.0.0/0 @@ -56,6 +66,8 @@ The filter table is: -P FORWARD ACCEPT -P OUTPUT ACCEPT -N DOCKER + -N DOCKER-BRIDGE + -N DOCKER-CT -N DOCKER-FORWARD -N DOCKER-ISOLATION-STAGE-1 -N DOCKER-ISOLATION-STAGE-2 @@ -64,9 +76,13 @@ The filter table is: -A FORWARD -j DOCKER-FORWARD -A DOCKER ! -i docker0 -o docker0 -j DROP -A DOCKER ! -i bridge1 -o bridge1 -j ACCEPT - -A DOCKER-FORWARD -m set --match-set docker-ext-bridges-v4 dst -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT + -A DOCKER-BRIDGE -o docker0 -j DOCKER + -A DOCKER-BRIDGE -o bridge1 -j DOCKER + -A DOCKER-CT -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT + -A DOCKER-CT -o bridge1 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT + -A DOCKER-FORWARD -j DOCKER-CT -A DOCKER-FORWARD -j DOCKER-ISOLATION-STAGE-1 - -A DOCKER-FORWARD -m set --match-set docker-ext-bridges-v4 dst -j DOCKER + -A DOCKER-FORWARD -j DOCKER-BRIDGE -A DOCKER-FORWARD -i docker0 -j ACCEPT -A DOCKER-FORWARD -i bridge1 -j ACCEPT -A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2 diff --git a/integration/network/bridge/iptablesdoc/generated/usernet-portmap-noicc.md b/integration/network/bridge/iptablesdoc/generated/usernet-portmap-noicc.md index 7687748e3e..21b079ae12 100644 --- a/integration/network/bridge/iptablesdoc/generated/usernet-portmap-noicc.md +++ b/integration/network/bridge/iptablesdoc/generated/usernet-portmap-noicc.md @@ -21,17 +21,27 @@ The filter table is: Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes) num pkts bytes target prot opt in out source destination - Chain DOCKER (1 references) + Chain DOCKER (2 references) num pkts bytes target prot opt in out source destination 1 0 0 ACCEPT 6 -- !bridge1 bridge1 0.0.0.0/0 192.0.2.2 tcp dpt:80 2 0 0 DROP 0 -- !docker0 docker0 0.0.0.0/0 0.0.0.0/0 3 0 0 DROP 0 -- !bridge1 bridge1 0.0.0.0/0 0.0.0.0/0 + Chain DOCKER-BRIDGE (1 references) + num pkts bytes target prot opt in out source destination + 1 0 0 DOCKER 0 -- * docker0 0.0.0.0/0 0.0.0.0/0 + 2 0 0 DOCKER 0 -- * bridge1 0.0.0.0/0 0.0.0.0/0 + + Chain DOCKER-CT (1 references) + num pkts bytes target prot opt in out source destination + 1 0 0 ACCEPT 0 -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED + 2 0 0 ACCEPT 0 -- * bridge1 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED + Chain DOCKER-FORWARD (1 references) num pkts bytes target prot opt in out source destination - 1 0 0 ACCEPT 0 -- * * 0.0.0.0/0 0.0.0.0/0 match-set docker-ext-bridges-v4 dst ctstate RELATED,ESTABLISHED + 1 0 0 DOCKER-CT 0 -- * * 0.0.0.0/0 0.0.0.0/0 2 0 0 DOCKER-ISOLATION-STAGE-1 0 -- * * 0.0.0.0/0 0.0.0.0/0 - 3 0 0 DOCKER 0 -- * * 0.0.0.0/0 0.0.0.0/0 match-set docker-ext-bridges-v4 dst + 3 0 0 DOCKER-BRIDGE 0 -- * * 0.0.0.0/0 0.0.0.0/0 4 0 0 ACCEPT 0 -- docker0 * 0.0.0.0/0 0.0.0.0/0 5 0 0 DROP 0 -- bridge1 bridge1 0.0.0.0/0 0.0.0.0/0 6 0 0 ACCEPT 0 -- bridge1 !bridge1 0.0.0.0/0 0.0.0.0/0 @@ -58,6 +68,8 @@ The filter table is: -P FORWARD ACCEPT -P OUTPUT ACCEPT -N DOCKER + -N DOCKER-BRIDGE + -N DOCKER-CT -N DOCKER-FORWARD -N DOCKER-ISOLATION-STAGE-1 -N DOCKER-ISOLATION-STAGE-2 @@ -67,9 +79,13 @@ The filter table is: -A DOCKER -d 192.0.2.2/32 ! -i bridge1 -o bridge1 -p tcp -m tcp --dport 80 -j ACCEPT -A DOCKER ! -i docker0 -o docker0 -j DROP -A DOCKER ! -i bridge1 -o bridge1 -j DROP - -A DOCKER-FORWARD -m set --match-set docker-ext-bridges-v4 dst -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT + -A DOCKER-BRIDGE -o docker0 -j DOCKER + -A DOCKER-BRIDGE -o bridge1 -j DOCKER + -A DOCKER-CT -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT + -A DOCKER-CT -o bridge1 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT + -A DOCKER-FORWARD -j DOCKER-CT -A DOCKER-FORWARD -j DOCKER-ISOLATION-STAGE-1 - -A DOCKER-FORWARD -m set --match-set docker-ext-bridges-v4 dst -j DOCKER + -A DOCKER-FORWARD -j DOCKER-BRIDGE -A DOCKER-FORWARD -i docker0 -j ACCEPT -A DOCKER-FORWARD -i bridge1 -o bridge1 -j DROP -A DOCKER-FORWARD -i bridge1 ! -o bridge1 -j ACCEPT @@ -84,7 +100,7 @@ The filter table is: By comparison with [ICC=true][1]: - - Rules 6 and 7 replace the accept rule for outgoing packets. + - DOCKER-FORWARD rules 6 and 7 replace the accept rule for outgoing packets. - Rule 6, added by `setIcc`, drops any packet sent from the internal network to itself. - Rule 7, added by `setupIPTablesInternal` accepts any other outgoing packet. diff --git a/integration/network/bridge/iptablesdoc/generated/usernet-portmap-noproxy.md b/integration/network/bridge/iptablesdoc/generated/usernet-portmap-noproxy.md index 0f6a6b00b1..b87d77aeca 100644 --- a/integration/network/bridge/iptablesdoc/generated/usernet-portmap-noproxy.md +++ b/integration/network/bridge/iptablesdoc/generated/usernet-portmap-noproxy.md @@ -24,17 +24,27 @@ The filter table is the same as with the userland proxy enabled. Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes) num pkts bytes target prot opt in out source destination - Chain DOCKER (1 references) + Chain DOCKER (2 references) num pkts bytes target prot opt in out source destination 1 0 0 ACCEPT 6 -- !bridge1 bridge1 0.0.0.0/0 192.0.2.2 tcp dpt:80 2 0 0 DROP 0 -- !docker0 docker0 0.0.0.0/0 0.0.0.0/0 3 0 0 DROP 0 -- !bridge1 bridge1 0.0.0.0/0 0.0.0.0/0 + Chain DOCKER-BRIDGE (1 references) + num pkts bytes target prot opt in out source destination + 1 0 0 DOCKER 0 -- * docker0 0.0.0.0/0 0.0.0.0/0 + 2 0 0 DOCKER 0 -- * bridge1 0.0.0.0/0 0.0.0.0/0 + + Chain DOCKER-CT (1 references) + num pkts bytes target prot opt in out source destination + 1 0 0 ACCEPT 0 -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED + 2 0 0 ACCEPT 0 -- * bridge1 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED + Chain DOCKER-FORWARD (1 references) num pkts bytes target prot opt in out source destination - 1 0 0 ACCEPT 0 -- * * 0.0.0.0/0 0.0.0.0/0 match-set docker-ext-bridges-v4 dst ctstate RELATED,ESTABLISHED + 1 0 0 DOCKER-CT 0 -- * * 0.0.0.0/0 0.0.0.0/0 2 0 0 DOCKER-ISOLATION-STAGE-1 0 -- * * 0.0.0.0/0 0.0.0.0/0 - 3 0 0 DOCKER 0 -- * * 0.0.0.0/0 0.0.0.0/0 match-set docker-ext-bridges-v4 dst + 3 0 0 DOCKER-BRIDGE 0 -- * * 0.0.0.0/0 0.0.0.0/0 4 0 0 ACCEPT 0 -- docker0 * 0.0.0.0/0 0.0.0.0/0 5 0 0 ACCEPT 0 -- bridge1 * 0.0.0.0/0 0.0.0.0/0 @@ -57,6 +67,8 @@ The filter table is the same as with the userland proxy enabled. -P FORWARD ACCEPT -P OUTPUT ACCEPT -N DOCKER + -N DOCKER-BRIDGE + -N DOCKER-CT -N DOCKER-FORWARD -N DOCKER-ISOLATION-STAGE-1 -N DOCKER-ISOLATION-STAGE-2 @@ -66,9 +78,13 @@ The filter table is the same as with the userland proxy enabled. -A DOCKER -d 192.0.2.2/32 ! -i bridge1 -o bridge1 -p tcp -m tcp --dport 80 -j ACCEPT -A DOCKER ! -i docker0 -o docker0 -j DROP -A DOCKER ! -i bridge1 -o bridge1 -j DROP - -A DOCKER-FORWARD -m set --match-set docker-ext-bridges-v4 dst -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT + -A DOCKER-BRIDGE -o docker0 -j DOCKER + -A DOCKER-BRIDGE -o bridge1 -j DOCKER + -A DOCKER-CT -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT + -A DOCKER-CT -o bridge1 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT + -A DOCKER-FORWARD -j DOCKER-CT -A DOCKER-FORWARD -j DOCKER-ISOLATION-STAGE-1 - -A DOCKER-FORWARD -m set --match-set docker-ext-bridges-v4 dst -j DOCKER + -A DOCKER-FORWARD -j DOCKER-BRIDGE -A DOCKER-FORWARD -i docker0 -j ACCEPT -A DOCKER-FORWARD -i bridge1 -j ACCEPT -A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2 diff --git a/integration/network/bridge/iptablesdoc/generated/usernet-portmap-routed.md b/integration/network/bridge/iptablesdoc/generated/usernet-portmap-routed.md index e6ebdc7bb3..e42bb0a6ec 100644 --- a/integration/network/bridge/iptablesdoc/generated/usernet-portmap-routed.md +++ b/integration/network/bridge/iptablesdoc/generated/usernet-portmap-routed.md @@ -21,18 +21,28 @@ The filter table is: Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes) num pkts bytes target prot opt in out source destination - Chain DOCKER (1 references) + Chain DOCKER (2 references) num pkts bytes target prot opt in out source destination 1 0 0 ACCEPT 6 -- !bridge1 bridge1 0.0.0.0/0 192.0.2.2 tcp dpt:80 2 0 0 DROP 0 -- !docker0 docker0 0.0.0.0/0 0.0.0.0/0 3 0 0 ACCEPT 1 -- * bridge1 0.0.0.0/0 0.0.0.0/0 4 0 0 DROP 0 -- !bridge1 bridge1 0.0.0.0/0 0.0.0.0/0 + Chain DOCKER-BRIDGE (1 references) + num pkts bytes target prot opt in out source destination + 1 0 0 DOCKER 0 -- * docker0 0.0.0.0/0 0.0.0.0/0 + 2 0 0 DOCKER 0 -- * bridge1 0.0.0.0/0 0.0.0.0/0 + + Chain DOCKER-CT (1 references) + num pkts bytes target prot opt in out source destination + 1 0 0 ACCEPT 0 -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED + 2 0 0 ACCEPT 0 -- * bridge1 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED + Chain DOCKER-FORWARD (1 references) num pkts bytes target prot opt in out source destination - 1 0 0 ACCEPT 0 -- * * 0.0.0.0/0 0.0.0.0/0 match-set docker-ext-bridges-v4 dst ctstate RELATED,ESTABLISHED + 1 0 0 DOCKER-CT 0 -- * * 0.0.0.0/0 0.0.0.0/0 2 0 0 DOCKER-ISOLATION-STAGE-1 0 -- * * 0.0.0.0/0 0.0.0.0/0 - 3 0 0 DOCKER 0 -- * * 0.0.0.0/0 0.0.0.0/0 match-set docker-ext-bridges-v4 dst + 3 0 0 DOCKER-BRIDGE 0 -- * * 0.0.0.0/0 0.0.0.0/0 4 0 0 ACCEPT 0 -- docker0 * 0.0.0.0/0 0.0.0.0/0 5 0 0 ACCEPT 0 -- bridge1 * 0.0.0.0/0 0.0.0.0/0 @@ -60,6 +70,8 @@ The filter table is: -P FORWARD ACCEPT -P OUTPUT ACCEPT -N DOCKER + -N DOCKER-BRIDGE + -N DOCKER-CT -N DOCKER-FORWARD -N DOCKER-ISOLATION-STAGE-1 -N DOCKER-ISOLATION-STAGE-2 @@ -70,9 +82,13 @@ The filter table is: -A DOCKER ! -i docker0 -o docker0 -j DROP -A DOCKER -o bridge1 -p icmp -j ACCEPT -A DOCKER ! -i bridge1 -o bridge1 -j DROP - -A DOCKER-FORWARD -m set --match-set docker-ext-bridges-v4 dst -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT + -A DOCKER-BRIDGE -o docker0 -j DOCKER + -A DOCKER-BRIDGE -o bridge1 -j DOCKER + -A DOCKER-CT -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT + -A DOCKER-CT -o bridge1 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT + -A DOCKER-FORWARD -j DOCKER-CT -A DOCKER-FORWARD -j DOCKER-ISOLATION-STAGE-1 - -A DOCKER-FORWARD -m set --match-set docker-ext-bridges-v4 dst -j DOCKER + -A DOCKER-FORWARD -j DOCKER-BRIDGE -A DOCKER-FORWARD -i docker0 -j ACCEPT -A DOCKER-FORWARD -i bridge1 -j ACCEPT -A DOCKER-ISOLATION-STAGE-1 -i bridge1 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT @@ -115,7 +131,7 @@ needed._ The ICMP rule, as shown by `iptables -L`, looks alarming until you spot that it's for `prot 1`: - Chain DOCKER (1 references) + Chain DOCKER (2 references) num pkts bytes target prot opt in out source destination 1 0 0 ACCEPT 6 -- !bridge1 bridge1 0.0.0.0/0 192.0.2.2 tcp dpt:80 2 0 0 DROP 0 -- !docker0 docker0 0.0.0.0/0 0.0.0.0/0 diff --git a/integration/network/bridge/iptablesdoc/generated/usernet-portmap.md b/integration/network/bridge/iptablesdoc/generated/usernet-portmap.md index 8ca7833e04..8c39accaed 100644 --- a/integration/network/bridge/iptablesdoc/generated/usernet-portmap.md +++ b/integration/network/bridge/iptablesdoc/generated/usernet-portmap.md @@ -20,17 +20,27 @@ The filter table is updated as follows: Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes) num pkts bytes target prot opt in out source destination - Chain DOCKER (1 references) + Chain DOCKER (2 references) num pkts bytes target prot opt in out source destination 1 0 0 ACCEPT 6 -- !bridge1 bridge1 0.0.0.0/0 192.0.2.2 tcp dpt:80 2 0 0 DROP 0 -- !docker0 docker0 0.0.0.0/0 0.0.0.0/0 3 0 0 DROP 0 -- !bridge1 bridge1 0.0.0.0/0 0.0.0.0/0 + Chain DOCKER-BRIDGE (1 references) + num pkts bytes target prot opt in out source destination + 1 0 0 DOCKER 0 -- * docker0 0.0.0.0/0 0.0.0.0/0 + 2 0 0 DOCKER 0 -- * bridge1 0.0.0.0/0 0.0.0.0/0 + + Chain DOCKER-CT (1 references) + num pkts bytes target prot opt in out source destination + 1 0 0 ACCEPT 0 -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED + 2 0 0 ACCEPT 0 -- * bridge1 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED + Chain DOCKER-FORWARD (1 references) num pkts bytes target prot opt in out source destination - 1 0 0 ACCEPT 0 -- * * 0.0.0.0/0 0.0.0.0/0 match-set docker-ext-bridges-v4 dst ctstate RELATED,ESTABLISHED + 1 0 0 DOCKER-CT 0 -- * * 0.0.0.0/0 0.0.0.0/0 2 0 0 DOCKER-ISOLATION-STAGE-1 0 -- * * 0.0.0.0/0 0.0.0.0/0 - 3 0 0 DOCKER 0 -- * * 0.0.0.0/0 0.0.0.0/0 match-set docker-ext-bridges-v4 dst + 3 0 0 DOCKER-BRIDGE 0 -- * * 0.0.0.0/0 0.0.0.0/0 4 0 0 ACCEPT 0 -- docker0 * 0.0.0.0/0 0.0.0.0/0 5 0 0 ACCEPT 0 -- bridge1 * 0.0.0.0/0 0.0.0.0/0 @@ -56,6 +66,8 @@ The filter table is updated as follows: -P FORWARD ACCEPT -P OUTPUT ACCEPT -N DOCKER + -N DOCKER-BRIDGE + -N DOCKER-CT -N DOCKER-FORWARD -N DOCKER-ISOLATION-STAGE-1 -N DOCKER-ISOLATION-STAGE-2 @@ -65,9 +77,13 @@ The filter table is updated as follows: -A DOCKER -d 192.0.2.2/32 ! -i bridge1 -o bridge1 -p tcp -m tcp --dport 80 -j ACCEPT -A DOCKER ! -i docker0 -o docker0 -j DROP -A DOCKER ! -i bridge1 -o bridge1 -j DROP - -A DOCKER-FORWARD -m set --match-set docker-ext-bridges-v4 dst -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT + -A DOCKER-BRIDGE -o docker0 -j DOCKER + -A DOCKER-BRIDGE -o bridge1 -j DOCKER + -A DOCKER-CT -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT + -A DOCKER-CT -o bridge1 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT + -A DOCKER-FORWARD -j DOCKER-CT -A DOCKER-FORWARD -j DOCKER-ISOLATION-STAGE-1 - -A DOCKER-FORWARD -m set --match-set docker-ext-bridges-v4 dst -j DOCKER + -A DOCKER-FORWARD -j DOCKER-BRIDGE -A DOCKER-FORWARD -i docker0 -j ACCEPT -A DOCKER-FORWARD -i bridge1 -j ACCEPT -A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2 @@ -83,6 +99,7 @@ Note that: - In the DOCKER-FORWARD chain, rule 5 for outgoing traffic from the new network has been appended to the end of the chain. + - The DOCKER-CT and DOCKER-FORWARD chains each have a rule for the new network. - In the DOCKER-ISOLATION chains, rules equivalent to the docker0 rules have also been inserted for the new bridge. - In the DOCKER chain, there is an ACCEPT rule for TCP port 80 packets routed diff --git a/integration/network/bridge/iptablesdoc/templates/new-daemon.md b/integration/network/bridge/iptablesdoc/templates/new-daemon.md index 558e4abed8..2e0de2e310 100644 --- a/integration/network/bridge/iptablesdoc/templates/new-daemon.md +++ b/integration/network/bridge/iptablesdoc/templates/new-daemon.md @@ -24,7 +24,7 @@ The FORWARD chain's policy shown above is ACCEPT. However: [1]: https://github.com/moby/moby/blob/cff4f20c44a3a7c882ed73934dec6a77246c6323/libnetwork/drivers/bridge/setup_ip_forwarding.go#L44 -The FORWARD chain rules are numbered in the output above, they are: +The FORWARD chain rules, explained in the order they appear in the output above, are: 1. Unconditional jump to DOCKER-USER. This is set up by libnetwork, in [setupUserChain][10]. @@ -40,24 +40,27 @@ the DOCKER-USER chain, for rules that run before DOCKER's). The DOCKER-FORWARD chain contains the first stage of Docker's filter rules. Initial rules are inserted at the top of the table, then not touched. Per-network rules -are appended. +are appended. The DOCKER-FORWARD chain rules, explained in the order they appear in +the output above, are: - 1. Early ACCEPT for any RELATED,ESTABLISHED traffic to a docker bridge. This rule - matches against an `ipset` called `docker-ext-bridges-v4` (`v6` for IPv6). The - set contains the CIDR address of each docker network, and it is updated as networks - are created and deleted. This rule is created during driver initialisation, in - `setupIPChains`. + 1. Unconditional jump to DOCKER-CT. + Created during driver initialisation, in `setupIPChains`. 2. Unconditional jump to DOCKER-ISOLATION-STAGE-1. Also created during driver initialisation, in `setupIPChains`. - 3. Jump to DOCKER, for any packet destined for any bridge network, identified by - matching against the `docker-ext-bridge-v[46]` set. + 3. Unconditional jump to DOCKER-BRIDGE. Also created during driver initialisation, in `setupIPChains`. - The DOCKER chain implements per-port/protocol filtering for each container. 4. ACCEPT any packet leaving a network, set up when the network is created, in `setupIPTablesInternal`. Note that this accepts any packet leaving the network that's made it through the DOCKER and isolation chains, whether the destination is external or another network. +The DOCKER-CT chain is an early ACCEPT for any RELATED,ESTABLISHED traffic to a +docker bridge. It contains a conntrack ACCEPT rule for each bridge network. + +DOCKER-BRIDGE has a rule for each bridge network, to jump to the DOCKER chain. + +The DOCKER chain implements per-port/protocol filtering for each container. + [10]: https://github.com/moby/moby/blob/e05848c0025b67a16aaafa8cdff95d5e2c064105/libnetwork/firewall_linux.go#L50 [11]: https://github.com/robmry/moby/blob/52c89d467fc5326149e4bbb8903d23589b66ff0d/libnetwork/drivers/bridge/setup_ip_tables_linux.go#L230-L232 [12]: https://github.com/robmry/moby/blob/52c89d467fc5326149e4bbb8903d23589b66ff0d/libnetwork/drivers/bridge/setup_ip_tables_linux.go#L227-L229 diff --git a/integration/network/bridge/iptablesdoc/templates/usernet-portmap-noicc.md b/integration/network/bridge/iptablesdoc/templates/usernet-portmap-noicc.md index e47a8107a7..2a82baa205 100644 --- a/integration/network/bridge/iptablesdoc/templates/usernet-portmap-noicc.md +++ b/integration/network/bridge/iptablesdoc/templates/usernet-portmap-noicc.md @@ -21,7 +21,7 @@ The filter table is: By comparison with [ICC=true][1]: - - Rules 6 and 7 replace the accept rule for outgoing packets. + - DOCKER-FORWARD rules 6 and 7 replace the accept rule for outgoing packets. - Rule 6, added by `setIcc`, drops any packet sent from the internal network to itself. - Rule 7, added by `setupIPTablesInternal` accepts any other outgoing packet. diff --git a/integration/network/bridge/iptablesdoc/templates/usernet-portmap.md b/integration/network/bridge/iptablesdoc/templates/usernet-portmap.md index 50233169c9..fcefa1b9a3 100644 --- a/integration/network/bridge/iptablesdoc/templates/usernet-portmap.md +++ b/integration/network/bridge/iptablesdoc/templates/usernet-portmap.md @@ -22,6 +22,7 @@ Note that: - In the DOCKER-FORWARD chain, rule 5 for outgoing traffic from the new network has been appended to the end of the chain. + - The DOCKER-CT and DOCKER-FORWARD chains each have a rule for the new network. - In the DOCKER-ISOLATION chains, rules equivalent to the docker0 rules have also been inserted for the new bridge. - In the DOCKER chain, there is an ACCEPT rule for TCP port 80 packets routed diff --git a/libnetwork/drivers/bridge/bridge_linux.go b/libnetwork/drivers/bridge/bridge_linux.go index 3d5792eaa5..ee79b79548 100644 --- a/libnetwork/drivers/bridge/bridge_linux.go +++ b/libnetwork/drivers/bridge/bridge_linux.go @@ -31,7 +31,6 @@ import ( "go.opentelemetry.io/otel" "go.opentelemetry.io/otel/attribute" "go.opentelemetry.io/otel/trace" - "golang.org/x/sys/unix" ) const ( @@ -521,9 +520,6 @@ func (d *driver) configure(option map[string]interface{}) error { if config.EnableIPTables { removeIPChains(iptables.IPv4) - if err := setupHashNetIpset(ipsetExtBridges4, unix.AF_INET); err != nil { - return fmt.Errorf("%w (kernel modules ip_set, ip_set_hash_net and netfilter_xt_set are required)", err) - } if err := setupIPChains(config, iptables.IPv4); err != nil { return err } @@ -548,28 +544,21 @@ func (d *driver) configure(option map[string]interface{}) error { removeIPChains(iptables.IPv6) - if err := setupHashNetIpset(ipsetExtBridges6, unix.AF_INET6); err != nil { - // Continue, IPv4 will work (as below). - log.G(context.TODO()).WithError(err).Warn( - "ip6tables is enabled, but cannot set up IPv6 ipset (kernel modules ip_set, ip_set_hash_net and netfilter_xt_set are required)") + if err := setupIPChains(config, iptables.IPv6); err != nil { + // If the chains couldn't be set up, it's probably because the kernel has no IPv6 + // support, or it doesn't have module ip6_tables loaded. It won't be possible to + // create IPv6 networks without enabling ip6_tables in the kernel, or disabling + // ip6tables in the daemon config. But, allow the daemon to start because IPv4 + // will work. So, log the problem, and continue. + log.G(context.TODO()).WithError(err).Warn("ip6tables is enabled, but cannot set up ip6tables chains") } else { - err = setupIPChains(config, iptables.IPv6) - if err != nil { - // If the chains couldn't be set up, it's probably because the kernel has no IPv6 - // support, or it doesn't have module ip6_tables loaded. It won't be possible to - // create IPv6 networks without enabling ip6_tables in the kernel, or disabling - // ip6tables in the daemon config. But, allow the daemon to start because IPv4 - // will work. So, log the problem, and continue. - log.G(context.TODO()).WithError(err).Warn("ip6tables is enabled, but cannot set up ip6tables chains") - } else { - // Make sure on firewall reload, first thing being re-played is chains creation - iptables.OnReloaded(func() { - log.G(context.TODO()).Debugf("Recreating ip6tables chains on firewall reload") - if err := setupIPChains(config, iptables.IPv6); err != nil { - log.G(context.TODO()).WithError(err).Error("Error reloading ip6tables chains") - } - }) - } + // Make sure on firewall reload, first thing being re-played is chains creation + iptables.OnReloaded(func() { + log.G(context.TODO()).Debugf("Recreating ip6tables chains on firewall reload") + if err := setupIPChains(config, iptables.IPv6); err != nil { + log.G(context.TODO()).WithError(err).Error("Error reloading ip6tables chains") + } + }) } } @@ -590,19 +579,6 @@ func (d *driver) configure(option map[string]interface{}) error { return d.initStore() } -func setupHashNetIpset(name string, family uint8) error { - if err := netlink.IpsetCreate(name, "hash:net", netlink.IpsetCreateOptions{ - Replace: true, - Family: family, - }); err != nil { - return fmt.Errorf("creating ipset %s: %w", name, err) - } - if err := netlink.IpsetFlush(name); err != nil { - return fmt.Errorf("flushing ipset %s: %w", name, err) - } - return nil -} - func (d *driver) getNetwork(id string) (*bridgeNetwork, error) { d.Lock() defer d.Unlock() diff --git a/libnetwork/drivers/bridge/bridge_linux_test.go b/libnetwork/drivers/bridge/bridge_linux_test.go index 0537134ffc..126e989e76 100644 --- a/libnetwork/drivers/bridge/bridge_linux_test.go +++ b/libnetwork/drivers/bridge/bridge_linux_test.go @@ -27,7 +27,6 @@ import ( "github.com/docker/docker/libnetwork/types" "github.com/vishvananda/netlink" "github.com/vishvananda/netns" - "golang.org/x/sys/unix" "gotest.tools/v3/assert" is "gotest.tools/v3/assert/cmp" "gotest.tools/v3/icmd" @@ -1266,9 +1265,6 @@ func TestCleanupIptableRules(t *testing.T) { iptables.IPv6: {EnableIP6Tables: true}, } - assert.NilError(t, setupHashNetIpset(ipsetExtBridges4, unix.AF_INET)) - assert.NilError(t, setupHashNetIpset(ipsetExtBridges6, unix.AF_INET6)) - for _, version := range ipVersions { err := setupIPChains(configs[version], version) assert.NilError(t, err, "version:%s", version) diff --git a/libnetwork/drivers/bridge/setup_ip_tables_linux.go b/libnetwork/drivers/bridge/setup_ip_tables_linux.go index 98da53a5a0..4b90c50322 100644 --- a/libnetwork/drivers/bridge/setup_ip_tables_linux.go +++ b/libnetwork/drivers/bridge/setup_ip_tables_linux.go @@ -13,13 +13,14 @@ import ( "github.com/docker/docker/libnetwork/iptables" "github.com/docker/docker/libnetwork/types" "github.com/vishvananda/netlink" - "github.com/vishvananda/netlink/nl" ) // DockerChain: DOCKER iptable chain name const ( DockerChain = "DOCKER" DockerForwardChain = "DOCKER-FORWARD" + DockerBridgeChain = "DOCKER-BRIDGE" + DockerCTChain = "DOCKER-CT" // Isolation between bridge networks is achieved in two stages by means // of the following two chains in the filter table. The first chain matches @@ -33,11 +34,6 @@ const ( IsolationChain1 = "DOCKER-ISOLATION-STAGE-1" IsolationChain2 = "DOCKER-ISOLATION-STAGE-2" - - // ipset names for IPv4 and IPv6 bridge subnets that don't belong - // to --internal networks. - ipsetExtBridges4 = "docker-ext-bridges-v4" - ipsetExtBridges6 = "docker-ext-bridges-v6" ) // Path to the executable installed in Linux under WSL2 that reports on @@ -92,6 +88,30 @@ func setupIPChains(config configuration, version iptables.IPVersion) (retErr err } }() + _, err = iptable.NewChain(DockerBridgeChain, iptables.Filter) + if err != nil { + return fmt.Errorf("failed to create FILTER chain %s: %v", DockerBridgeChain, err) + } + defer func() { + if retErr != nil { + if err := iptable.RemoveExistingChain(DockerBridgeChain, iptables.Filter); err != nil { + log.G(context.TODO()).Warnf("failed on removing iptables FILTER chain %s on cleanup: %v", DockerBridgeChain, err) + } + } + }() + + _, err = iptable.NewChain(DockerCTChain, iptables.Filter) + if err != nil { + return fmt.Errorf("failed to create FILTER chain %s: %v", DockerCTChain, err) + } + defer func() { + if retErr != nil { + if err := iptable.RemoveExistingChain(DockerCTChain, iptables.Filter); err != nil { + log.G(context.TODO()).Warnf("failed on removing iptables FILTER chain %s on cleanup: %v", DockerCTChain, err) + } + } + }() + _, err = iptable.NewChain(IsolationChain1, iptables.Filter) if err != nil { return fmt.Errorf("failed to create FILTER isolation chain: %v", err) @@ -130,14 +150,32 @@ func setupIPChains(config configuration, version iptables.IPVersion) (retErr err // Make sure the filter-FORWARD chain has rules to accept related packets and // jump to the isolation and docker chains. (Re-)insert at the top of the table, // in reverse order. - ipsetName := ipsetExtBridges4 - if version == iptables.IPv6 { - ipsetName = ipsetExtBridges6 + if err := iptable.EnsureJumpRule("FORWARD", DockerForwardChain); err != nil { + return err } + if err := iptable.EnsureJumpRule(DockerForwardChain, DockerBridgeChain); err != nil { + return err + } + if err := iptable.EnsureJumpRule(DockerForwardChain, IsolationChain1); err != nil { + return err + } + if err := iptable.EnsureJumpRule(DockerForwardChain, DockerCTChain); err != nil { + return err + } + + if err := mirroredWSL2Workaround(config, version); err != nil { + return err + } + // Delete rules that may have been added to the FORWARD chain by moby 28.0.0. + ipsetName := "docker-ext-bridges-v4" + if version == iptables.IPv6 { + ipsetName = "docker-ext-bridges-v6" + } if err := iptable.DeleteJumpRule("FORWARD", DockerChain, "-m", "set", "--match-set", ipsetName, "dst"); err != nil { - return fmt.Errorf("%w (kernel module netfilter_xt_set is required)", err) + log.G(context.TODO()).WithFields(log.Fields{"error": err, "set": ipsetName}).Debug( + "deleting legacy ipset dest match rule") } if err := iptable.DeleteJumpRule("FORWARD", IsolationChain1); err != nil { return err @@ -146,28 +184,8 @@ func setupIPChains(config configuration, version iptables.IPVersion) (retErr err "-m", "set", "--match-set", ipsetName, "dst", "-m", "conntrack", "--ctstate", "RELATED,ESTABLISHED", ); err != nil { - return err - } - // Create rules in the DockerForward chain. - if err := iptable.EnsureJumpRule("FORWARD", DockerForwardChain); err != nil { - return err - } - if err := iptable.EnsureJumpRule(DockerForwardChain, DockerChain, - "-m", "set", "--match-set", ipsetName, "dst"); err != nil { - return err - } - if err := iptable.EnsureJumpRule(DockerForwardChain, IsolationChain1); err != nil { - return err - } - if err := iptable.EnsureJumpRule(DockerForwardChain, "ACCEPT", - "-m", "set", "--match-set", ipsetName, "dst", - "-m", "conntrack", "--ctstate", "RELATED,ESTABLISHED", - ); err != nil { - return err - } - - if err := mirroredWSL2Workaround(config, version); err != nil { - return err + log.G(context.TODO()).WithFields(log.Fields{"error": err, "set": ipsetName}).Debug( + "deleting legacy ipset conntrack rule") } return nil @@ -221,11 +239,6 @@ func (n *bridgeNetwork) setupIPTables(ipVersion iptables.IPVersion, maskedAddr * // Pickup this configuration option from driver hairpinMode := !driverConfig.EnableUserlandProxy - ipsetName := ipsetExtBridges4 - if ipVersion == iptables.IPv6 { - ipsetName = ipsetExtBridges6 - } - if config.Internal { if err = setupInternalNetworkRules(config.BridgeName, maskedAddr, config.EnableICC, true); err != nil { return fmt.Errorf("Failed to Setup IP tables: %w", err) @@ -260,28 +273,26 @@ func (n *bridgeNetwork) setupIPTables(ipVersion iptables.IPVersion, maskedAddr * return err } - cidr, _ := maskedAddr.Mask.Size() - if cidr == 0 { - return fmt.Errorf("no CIDR for bridge %s addr %s", config.BridgeName, maskedAddr) - } - ipsetEntry := &netlink.IPSetEntry{ - IP: maskedAddr.IP, - CIDR: uint8(cidr), - } - if err := netlink.IpsetAdd(ipsetName, ipsetEntry); err != nil { - if !errors.Is(err, nl.IPSetError(nl.IPSET_ERR_EXIST)) { - return fmt.Errorf("failed to add bridge %s (%s) to ipset: %w", - config.BridgeName, maskedAddr, err) - } - // Re-adding an IP address to an ipset on firewalld reload is expected, not an error. - log.G(context.TODO()).WithFields(log.Fields{ - "ipset": ipsetName, - "bridge": config.BridgeName, - "subnet": maskedAddr, - }).Debug("Subnet was already in the ipset") + ctRule := iptables.Rule{IPVer: ipVersion, Table: iptables.Filter, Chain: DockerCTChain, Args: []string{ + "-o", config.BridgeName, + "-m", "conntrack", "--ctstate", "RELATED,ESTABLISHED", + "-j", "ACCEPT", + }} + if err := appendOrDelChainRule(ctRule, "bridge ct related", true); err != nil { + return err } n.registerIptCleanFunc(func() error { - return netlink.IpsetDel(ipsetName, ipsetEntry) + return appendOrDelChainRule(ctRule, "bridge ct related", false) + }) + jumpToDockerRule := iptables.Rule{IPVer: ipVersion, Table: iptables.Filter, Chain: DockerBridgeChain, Args: []string{ + "-o", config.BridgeName, + "-j", DockerChain, + }} + if err := appendOrDelChainRule(jumpToDockerRule, "jump to docker", true); err != nil { + return err + } + n.registerIptCleanFunc(func() error { + return appendOrDelChainRule(jumpToDockerRule, "jump to docker", false) }) } return nil diff --git a/libnetwork/drivers/bridge/setup_ip_tables_linux_test.go b/libnetwork/drivers/bridge/setup_ip_tables_linux_test.go index 5a7ea8fb95..54a6ccf4f3 100644 --- a/libnetwork/drivers/bridge/setup_ip_tables_linux_test.go +++ b/libnetwork/drivers/bridge/setup_ip_tables_linux_test.go @@ -16,7 +16,6 @@ import ( "github.com/docker/docker/libnetwork/netlabel" "github.com/docker/docker/libnetwork/types" "github.com/vishvananda/netlink" - "golang.org/x/sys/unix" "gotest.tools/v3/assert" is "gotest.tools/v3/assert/cmp" ) @@ -97,22 +96,15 @@ func TestSetupIPChains(t *testing.T) { createTestBridge(config, br, t) assertBridgeConfig(config, br, d, t) - // The purpose of this test is unclear but, now there's an ipset of bridges, it's - // an error to create a bridge that's already been created. That can't happen in - // normal running. So, just flush the set between each step. - assert.NilError(t, netlink.IpsetFlush(ipsetExtBridges4)) config.EnableIPMasquerade = true assertBridgeConfig(config, br, d, t) - assert.NilError(t, netlink.IpsetFlush(ipsetExtBridges4)) config.EnableICC = true assertBridgeConfig(config, br, d, t) - assert.NilError(t, netlink.IpsetFlush(ipsetExtBridges4)) config.EnableIPMasquerade = false assertBridgeConfig(config, br, d, t) - assert.NilError(t, netlink.IpsetFlush(ipsetExtBridges4)) } func getBasicTestConfig() *networkConfiguration { @@ -164,14 +156,10 @@ func assertIPTableChainProgramming(rule iptables.Rule, descr string, t *testing. func assertChainConfig(d *driver, t *testing.T) { var err error - err = setupHashNetIpset(ipsetExtBridges4, unix.AF_INET) - assert.NilError(t, err) err = setupIPChains(d.config, iptables.IPv4) assert.NilError(t, err) if d.config.EnableIP6Tables { - err = setupHashNetIpset(ipsetExtBridges6, unix.AF_INET6) - assert.NilError(t, err) err = setupIPChains(d.config, iptables.IPv6) assert.NilError(t, err) } @@ -474,8 +462,6 @@ func TestMirroredWSL2Workaround(t *testing.T) { restoreWslinfoPath := simulateWSL2MirroredMode(t, tc.loopback0, tc.wslinfoPerm) defer restoreWslinfoPath() - assert.NilError(t, setupHashNetIpset(ipsetExtBridges4, unix.AF_INET)) - config := configuration{EnableIPTables: true} if tc.userlandProxy { config.UserlandProxyPath = "some-proxy" diff --git a/libnetwork/testdata/TestUserChain_iptables-true_append-false_dockerfwdafter4 b/libnetwork/testdata/TestUserChain_iptables-true_append-false_dockerfwdafter4 index 67db902da9..af14da8ba6 100644 --- a/libnetwork/testdata/TestUserChain_iptables-true_append-false_dockerfwdafter4 +++ b/libnetwork/testdata/TestUserChain_iptables-true_append-false_dockerfwdafter4 @@ -1,4 +1,4 @@ -N DOCKER-FORWARD --A DOCKER-FORWARD -m set --match-set docker-ext-bridges-v4 dst -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT +-A DOCKER-FORWARD -j DOCKER-CT -A DOCKER-FORWARD -j DOCKER-ISOLATION-STAGE-1 --A DOCKER-FORWARD -m set --match-set docker-ext-bridges-v4 dst -j DOCKER +-A DOCKER-FORWARD -j DOCKER-BRIDGE diff --git a/libnetwork/testdata/TestUserChain_iptables-true_append-false_dockerfwdafter6 b/libnetwork/testdata/TestUserChain_iptables-true_append-false_dockerfwdafter6 index 3435aee061..af14da8ba6 100644 --- a/libnetwork/testdata/TestUserChain_iptables-true_append-false_dockerfwdafter6 +++ b/libnetwork/testdata/TestUserChain_iptables-true_append-false_dockerfwdafter6 @@ -1,4 +1,4 @@ -N DOCKER-FORWARD --A DOCKER-FORWARD -m set --match-set docker-ext-bridges-v6 dst -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT +-A DOCKER-FORWARD -j DOCKER-CT -A DOCKER-FORWARD -j DOCKER-ISOLATION-STAGE-1 --A DOCKER-FORWARD -m set --match-set docker-ext-bridges-v6 dst -j DOCKER +-A DOCKER-FORWARD -j DOCKER-BRIDGE diff --git a/libnetwork/testdata/TestUserChain_iptables-true_append-false_dockerfwdinit4 b/libnetwork/testdata/TestUserChain_iptables-true_append-false_dockerfwdinit4 index 67db902da9..af14da8ba6 100644 --- a/libnetwork/testdata/TestUserChain_iptables-true_append-false_dockerfwdinit4 +++ b/libnetwork/testdata/TestUserChain_iptables-true_append-false_dockerfwdinit4 @@ -1,4 +1,4 @@ -N DOCKER-FORWARD --A DOCKER-FORWARD -m set --match-set docker-ext-bridges-v4 dst -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT +-A DOCKER-FORWARD -j DOCKER-CT -A DOCKER-FORWARD -j DOCKER-ISOLATION-STAGE-1 --A DOCKER-FORWARD -m set --match-set docker-ext-bridges-v4 dst -j DOCKER +-A DOCKER-FORWARD -j DOCKER-BRIDGE diff --git a/libnetwork/testdata/TestUserChain_iptables-true_append-false_dockerfwdinit6 b/libnetwork/testdata/TestUserChain_iptables-true_append-false_dockerfwdinit6 index 3435aee061..af14da8ba6 100644 --- a/libnetwork/testdata/TestUserChain_iptables-true_append-false_dockerfwdinit6 +++ b/libnetwork/testdata/TestUserChain_iptables-true_append-false_dockerfwdinit6 @@ -1,4 +1,4 @@ -N DOCKER-FORWARD --A DOCKER-FORWARD -m set --match-set docker-ext-bridges-v6 dst -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT +-A DOCKER-FORWARD -j DOCKER-CT -A DOCKER-FORWARD -j DOCKER-ISOLATION-STAGE-1 --A DOCKER-FORWARD -m set --match-set docker-ext-bridges-v6 dst -j DOCKER +-A DOCKER-FORWARD -j DOCKER-BRIDGE diff --git a/libnetwork/testdata/TestUserChain_iptables-true_append-true_dockerfwdafter4 b/libnetwork/testdata/TestUserChain_iptables-true_append-true_dockerfwdafter4 index 67db902da9..af14da8ba6 100644 --- a/libnetwork/testdata/TestUserChain_iptables-true_append-true_dockerfwdafter4 +++ b/libnetwork/testdata/TestUserChain_iptables-true_append-true_dockerfwdafter4 @@ -1,4 +1,4 @@ -N DOCKER-FORWARD --A DOCKER-FORWARD -m set --match-set docker-ext-bridges-v4 dst -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT +-A DOCKER-FORWARD -j DOCKER-CT -A DOCKER-FORWARD -j DOCKER-ISOLATION-STAGE-1 --A DOCKER-FORWARD -m set --match-set docker-ext-bridges-v4 dst -j DOCKER +-A DOCKER-FORWARD -j DOCKER-BRIDGE diff --git a/libnetwork/testdata/TestUserChain_iptables-true_append-true_dockerfwdafter6 b/libnetwork/testdata/TestUserChain_iptables-true_append-true_dockerfwdafter6 index 3435aee061..af14da8ba6 100644 --- a/libnetwork/testdata/TestUserChain_iptables-true_append-true_dockerfwdafter6 +++ b/libnetwork/testdata/TestUserChain_iptables-true_append-true_dockerfwdafter6 @@ -1,4 +1,4 @@ -N DOCKER-FORWARD --A DOCKER-FORWARD -m set --match-set docker-ext-bridges-v6 dst -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT +-A DOCKER-FORWARD -j DOCKER-CT -A DOCKER-FORWARD -j DOCKER-ISOLATION-STAGE-1 --A DOCKER-FORWARD -m set --match-set docker-ext-bridges-v6 dst -j DOCKER +-A DOCKER-FORWARD -j DOCKER-BRIDGE diff --git a/libnetwork/testdata/TestUserChain_iptables-true_append-true_dockerfwdinit4 b/libnetwork/testdata/TestUserChain_iptables-true_append-true_dockerfwdinit4 index 67db902da9..af14da8ba6 100644 --- a/libnetwork/testdata/TestUserChain_iptables-true_append-true_dockerfwdinit4 +++ b/libnetwork/testdata/TestUserChain_iptables-true_append-true_dockerfwdinit4 @@ -1,4 +1,4 @@ -N DOCKER-FORWARD --A DOCKER-FORWARD -m set --match-set docker-ext-bridges-v4 dst -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT +-A DOCKER-FORWARD -j DOCKER-CT -A DOCKER-FORWARD -j DOCKER-ISOLATION-STAGE-1 --A DOCKER-FORWARD -m set --match-set docker-ext-bridges-v4 dst -j DOCKER +-A DOCKER-FORWARD -j DOCKER-BRIDGE diff --git a/libnetwork/testdata/TestUserChain_iptables-true_append-true_dockerfwdinit6 b/libnetwork/testdata/TestUserChain_iptables-true_append-true_dockerfwdinit6 index 3435aee061..af14da8ba6 100644 --- a/libnetwork/testdata/TestUserChain_iptables-true_append-true_dockerfwdinit6 +++ b/libnetwork/testdata/TestUserChain_iptables-true_append-true_dockerfwdinit6 @@ -1,4 +1,4 @@ -N DOCKER-FORWARD --A DOCKER-FORWARD -m set --match-set docker-ext-bridges-v6 dst -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT +-A DOCKER-FORWARD -j DOCKER-CT -A DOCKER-FORWARD -j DOCKER-ISOLATION-STAGE-1 --A DOCKER-FORWARD -m set --match-set docker-ext-bridges-v6 dst -j DOCKER +-A DOCKER-FORWARD -j DOCKER-BRIDGE