Libvirt overwrites the existing iptables rules
From WBITT's Cooker!
(→First, the problem) |
(→First, the problem) |
||
| Line 15: | Line 15: | ||
* http://www.gossamer-threads.com/lists/xen/users/175478 | * http://www.gossamer-threads.com/lists/xen/users/175478 | ||
| - | === | + | ===Summary=== |
The problem is faced, when a virtual machine needs to be accessed from the outside-network, instead of a virtual machine accessing the outside-network. [Note the difference, here]. The virtual machine in discussion is on a (non-routeable) private network, inside a XEN or KVM host, connected to virbr0 interface of the physical host. This is the type of setup, which is used in situations, where you have limitations/restrictions connecting your VMs directly to the outside-network. ServerBeach, a reputed server-rental company / hosting provider, is one example. | The problem is faced, when a virtual machine needs to be accessed from the outside-network, instead of a virtual machine accessing the outside-network. [Note the difference, here]. The virtual machine in discussion is on a (non-routeable) private network, inside a XEN or KVM host, connected to virbr0 interface of the physical host. This is the type of setup, which is used in situations, where you have limitations/restrictions connecting your VMs directly to the outside-network. ServerBeach, a reputed server-rental company / hosting provider, is one example. | ||
Revision as of 11:32, 1 February 2011
Please send your thoughts/comments to: kamran at wbitt dot com.
Contents |
The observation: XEN overwrites the existing iptables rules
no. Not exactly. It is actually libvirt which is the culprit, not XEN.
- https://bugzilla.redhat.com/show_bug.cgi?id=227011
- http://lists.fedoraproject.org/pipermail/virt/2010-January/001792.html
- http://forums.gentoo.org/viewtopic-p-6209192.html?sid=1089acac70de96d68aa856d758d7cdfe
- http://wiki.libvirt.org/page/Networking
- http://libvirt.org/formatnetwork.html#examplesPrivate
Other readings:
- http://people.gnome.org/~markmc/virtual-networking.html
- http://www.standingonthebrink.com/index.php/ipv6-ipv4-and-arp-on-xen-for-vps/
- http://www.gossamer-threads.com/lists/xen/users/175478
Summary
The problem is faced, when a virtual machine needs to be accessed from the outside-network, instead of a virtual machine accessing the outside-network. [Note the difference, here]. The virtual machine in discussion is on a (non-routeable) private network, inside a XEN or KVM host, connected to virbr0 interface of the physical host. This is the type of setup, which is used in situations, where you have limitations/restrictions connecting your VMs directly to the outside-network. ServerBeach, a reputed server-rental company / hosting provider, is one example.
Below is an example of such situation where a Web Server and a Mail Server runs as virtual machines, in a publicly accessible server (XEN), rented from a server rental company.
In such scenario, naturally, the administrator of the physical host, would create certain forwarding (or DNAT) rules, to allow the traffic coming from the outside, to be redirected to a VM. While doing so, it is observed that whenever the services libvirt or xen, are restarted, or the physical host is restarted, any iptables rules written by the administrator are over-written (by some un-known process). For long, XEN has been (incorrectly) accused for doing so.
In this text, we will explain that it is not XEN, which is over-writing the iptables rules. It is actually "libvirt" which is doing so. And so far, by the time of this writing, there is no solution for it. It is a known bug and still in the OPEN state at redhat and fedora bugzilla websites. (https://bugzilla.redhat.com/show_bug.cgi?id=227011)
As similar restrictions may exist in smaller networks and setups, the diagram below, which is a typical lab setup, will help to explain, reproduce and analyse the problem.
In brief words, here is the problem. I have the following rules configured in my iptables firewall. (The two rules shown here are just example. Don't use them on any of the production servers of yours.) Note: that the services (iptables, libvirtd,xend) are set to ON on system boot.
[root@xenhost ~]# cat /etc/sysconfig/iptables *nat :PREROUTING ACCEPT [10:312] :POSTROUTING ACCEPT [3:260] :OUTPUT ACCEPT [3:260] COMMIT *filter :INPUT ACCEPT [962:71063] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [584:64182] -A INPUT -i eth0 -p tcp -m tcp --dport 80 -j REJECT -A FORWARD -o eth0 -p tcp -m tcp --dport 80 -j REJECT COMMIT [root@xenhost ~]#
[root@xenhost ~]# iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination REJECT tcp -- anywhere anywhere tcp dpt:http reject-with icmp-port-unreachable Chain FORWARD (policy ACCEPT) target prot opt source destination REJECT tcp -- anywhere anywhere tcp dpt:http reject-with icmp-port-unreachable Chain OUTPUT (policy ACCEPT) target prot opt source destination [root@xenhost ~]#
Now, as soon as we restart the physical host. Bammm.... Our iptables rules are gone... replaced with some new (strange?) iptables rules.
[root@xenhost ~]# iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination ACCEPT udp -- anywhere anywhere udp dpt:domain ACCEPT tcp -- anywhere anywhere tcp dpt:domain ACCEPT udp -- anywhere anywhere udp dpt:bootps ACCEPT tcp -- anywhere anywhere tcp dpt:bootps REJECT tcp -- anywhere anywhere tcp dpt:http reject-with icmp-port-unreachable Chain FORWARD (policy ACCEPT) target prot opt source destination ACCEPT all -- anywhere 192.168.122.0/24 state RELATED,ESTABLISHED ACCEPT all -- 192.168.122.0/24 anywhere ACCEPT all -- anywhere anywhere REJECT all -- anywhere anywhere reject-with icmp-port-unreachable REJECT all -- anywhere anywhere reject-with icmp-port-unreachable REJECT tcp -- anywhere anywhere tcp dpt:http reject-with icmp-port-unreachable ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-in vif1.0 Chain OUTPUT (policy ACCEPT) target prot opt source destination [root@xenhost ~]#
Objective / goal of this document
The objective of this document is to identify/clarify the following:
- What are these specific iptable rules?
- Why do we care? and, When do we care?
- Does it matter if we lose these rules?
- Does it matter when we have our virtual machines on a bridged interface, connecting directly to our physical LAN, xenbr0 or br0?
- Does it matter when we have our virtual machines connected only on the private network inside the physical host, virbr0?
- How do we circumvent any problems related to this scenario?
The details
It is observed, that systems which provide KVM or XEN virtualization technologies, sometimes have their iptables firewall rules changed automatically in Dom-0, overwritten by another set of rules.
Note: KVM doesn't have Dom-0. The base OS on the (KVM) physical host will be considered Dom-0 for ease of understanding. (Please. No flame war on this one.). I will use "Physical Host" and "Dom-0" interchangeably for ease of understanding, for a system which hosts one or more virtual machines (i.e. KVM or XEN).
This happens only on those systems, which run libvirtd service. Mistakenly, many people think this is XEN problem. Whereas it is not. First, I would explain, what is the default iptables firewall ruleset on the physical host.
Interestingly this is not much of an issue for the Virtual machines which share the physical ethernet device of the physical host. This is more of an issue, for virutal machines, connected to a virtual (private) network/bridge (virbr0) (192.168.122.0/24) inside the physical host. This means, their traffic passes through NAT configured on the same physical host, whenever they need to go out to the physical interface.
As we are aware that in the shared network device mode, it is easier (aka painless), to access the virtual machines, from the main physical network. This is because they (the virtual machines) have the IP from the same scheme/range/subnet as of the physical device of the physical host they are on. This is ideal for so called server mode, where all the virtual machines, are actually serving some services on the LAN.
On the other hand, in the virtual network mode, it is difficult (aka painful), to access the virtual machines, from the main physical network. It is because, they (the virtual machines) have the IP from a private scheme (192.168.122.0/24), and they sit behind a NAT router which is created by the virtual machine manager, on the physical host. That means, their traffic can go out of the network, NATed, and related traffic can get back in. But, if we want to access these (virtual servers (running on private IPs) from the LAN, we need to configure some traffic redirection (DNAT), on the physical host. This mode, of creating virtual machines on the private network, is ideal for private testing of various type of stuff. BUT, when you are in a situation, where you need your private server to be acccessed from outside, and you cannot use the "shared network device" mode for certain/whatever reasons, then you need to figure out a way to pass your traffic coming from outside to your VM. And when you try to do that, you see your iptables rules of your physical host machine, being clobbered/over-written by the libvirtd's rules, then you get frustrated. And that, is the reason, behind this document.
The default iptables rules on a KVM physical host
Here is a default iptables rule-set from a Fedora13 physical host. The default firewall (iptables service) was stopped when libvirtd service was started, at system boot.
[root@training ~]# iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination ACCEPT udp -- anywhere anywhere udp dpt:domain ACCEPT tcp -- anywhere anywhere tcp dpt:domain ACCEPT udp -- anywhere anywhere udp dpt:bootps ACCEPT tcp -- anywhere anywhere tcp dpt:bootps Chain FORWARD (policy ACCEPT) target prot opt source destination ACCEPT all -- anywhere 192.168.122.0/24 state RELATED,ESTABLISHED ACCEPT all -- 192.168.122.0/24 anywhere ACCEPT all -- anywhere anywhere REJECT all -- anywhere anywhere reject-with icmp-port-unreachable REJECT all -- anywhere anywhere reject-with icmp-port-unreachable Chain OUTPUT (policy ACCEPT) target prot opt source destination [root@training ~]# iptables -L -t nat Chain PREROUTING (policy ACCEPT) target prot opt source destination Chain POSTROUTING (policy ACCEPT) target prot opt source destination MASQUERADE all -- 192.168.122.0/24 !192.168.122.0/24 Chain OUTPUT (policy ACCEPT) target prot opt source destination
Let's save these rules in a file, so we can load the defaults any time we need to.
[root@training ~]# iptables-save > /root/iptables.default.txt
I will show you these rules from this file for easier understanding:
[root@training ~]# cat /root/iptables.default.txt *nat :PREROUTING ACCEPT [661:21364] :POSTROUTING ACCEPT [58069:3670258] :OUTPUT ACCEPT [58069:3670258] -A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE COMMIT *filter :INPUT ACCEPT [1212620:674141323] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [1518464:780474182] -A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT -A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT -A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT -A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT -A FORWARD -d 192.168.122.0/24 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT -A FORWARD -s 192.168.122.0/24 -i virbr0 -j ACCEPT -A FORWARD -i virbr0 -o virbr0 -j ACCEPT -A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable -A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable COMMIT [root@training ~]#
Lets start a VM:
[root@training init.d]# virsh start linuxvm Domain linuxvm started [root@training init.d]# virsh list Id Name State ---------------------------------- 2 linuxvm running
Here is the output of ifconfig command, after a VM is started.
[root@training init.d]# ifconfig | grep -A1 HWaddr
eth0 Link encap:Ethernet HWaddr 00:26:2D:95:AB:FA
inet addr:192.168.1.3 Bcast:192.168.1.255 Mask:255.255.255.0
--
virbr0 Link encap:Ethernet HWaddr 26:45:88:61:E0:CC
inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0
--
vnet0 Link encap:Ethernet HWaddr 26:45:88:61:E0:CC
inet6 addr: fe80::2445:88ff:fe61:e0cc/64 Scope:Link
[root@training init.d]#
The default iptables rules on a XEN physical host
Here is the default iptables rules as seen when a XEN physical host boots up with default configurations. Please note that the firewall service was configured to be stopped on system boot. Which means, that these rules were added by some mechanism (libvirtd) on the XEN host.
[root@xenhost ~]# iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination ACCEPT udp -- anywhere anywhere udp dpt:domain ACCEPT tcp -- anywhere anywhere tcp dpt:domain ACCEPT udp -- anywhere anywhere udp dpt:bootps ACCEPT tcp -- anywhere anywhere tcp dpt:bootps Chain FORWARD (policy ACCEPT) target prot opt source destination ACCEPT all -- anywhere 192.168.122.0/24 state RELATED,ESTABLISHED ACCEPT all -- 192.168.122.0/24 anywhere ACCEPT all -- anywhere anywhere REJECT all -- anywhere anywhere reject-with icmp-port-unreachable REJECT all -- anywhere anywhere reject-with icmp-port-unreachable Chain OUTPUT (policy ACCEPT) target prot opt source destination [root@xenhost ~]# iptables -L -t nat Chain PREROUTING (policy ACCEPT) target prot opt source destination Chain POSTROUTING (policy ACCEPT) target prot opt source destination MASQUERADE all -- 192.168.122.0/24 !192.168.122.0/24 Chain OUTPUT (policy ACCEPT) target prot opt source destination [root@xenhost ~]#
Save these rules for future reference:
[root@xenhost ~]# iptables-save > /root/iptables-default.txt
Have a look at the resultant file to understand the rules better:
[root@xenhost ~]# cat /root/iptables-default.txt *nat :PREROUTING ACCEPT [5:180] :POSTROUTING ACCEPT [6:428] :OUTPUT ACCEPT [6:428] -A POSTROUTING -s 192.168.122.0/255.255.255.0 -d ! 192.168.122.0/255.255.255.0 -j MASQUERADE COMMIT *filter :INPUT ACCEPT [168:13693] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [114:13252] -A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT -A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT -A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT -A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT -A FORWARD -d 192.168.122.0/255.255.255.0 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT -A FORWARD -s 192.168.122.0/255.255.255.0 -i virbr0 -j ACCEPT -A FORWARD -i virbr0 -o virbr0 -j ACCEPT -A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable -A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable COMMIT [root@xenhost ~]#
Now, let's start a VM on the XEN host, configured to run, while connected to the xen host private network, virbr0.
[root@xenhost ~]# virsh list --all Id Name State ---------------------------------- 0 Domain-0 running - miniweb shut off [root@xenhost ~]# virsh start miniweb Domain miniweb started [root@xenhost ~]# virsh list --all Id Name State ---------------------------------- 0 Domain-0 running 1 miniweb idle [root@xenhost ~]#
Let's look at the rules again, after starting the VM:
[root@xenhost ~]# iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination ACCEPT udp -- anywhere anywhere udp dpt:domain ACCEPT tcp -- anywhere anywhere tcp dpt:domain ACCEPT udp -- anywhere anywhere udp dpt:bootps ACCEPT tcp -- anywhere anywhere tcp dpt:bootps Chain FORWARD (policy ACCEPT) target prot opt source destination ACCEPT all -- anywhere 192.168.122.0/24 state RELATED,ESTABLISHED ACCEPT all -- 192.168.122.0/24 anywhere ACCEPT all -- anywhere anywhere REJECT all -- anywhere anywhere reject-with icmp-port-unreachable REJECT all -- anywhere anywhere reject-with icmp-port-unreachable ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-in vif1.0 Chain OUTPUT (policy ACCEPT) target prot opt source destination [root@xenhost ~]# [root@xenhost ~]# iptables -L -t nat Chain PREROUTING (policy ACCEPT) target prot opt source destination Chain POSTROUTING (policy ACCEPT) target prot opt source destination MASQUERADE all -- 192.168.122.0/24 !192.168.122.0/24 Chain OUTPUT (policy ACCEPT) target prot opt source destination [root@xenhost ~]#
Lets save these rules in another file for comparison.
[root@xenhost ~]# iptables-save > /root/iptables-vm-running.txt [root@xenhost ~]# cat /root/iptables-vm-running.txt *nat :PREROUTING ACCEPT [34:2932] :POSTROUTING ACCEPT [18:1292] :OUTPUT ACCEPT [18:1292] -A POSTROUTING -s 192.168.122.0/255.255.255.0 -d ! 192.168.122.0/255.255.255.0 -j MASQUERADE COMMIT *filter :INPUT ACCEPT [549:40513] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [364:41916] -A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT -A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT -A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT -A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT -A FORWARD -d 192.168.122.0/255.255.255.0 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT -A FORWARD -s 192.168.122.0/255.255.255.0 -i virbr0 -j ACCEPT -A FORWARD -i virbr0 -o virbr0 -j ACCEPT -A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable -A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable -A FORWARD -m physdev --physdev-in vif1.0 -j ACCEPT COMMIT
Here I run the diff on the two files on the XEN host and try to see the differences.
[root@xenhost ~]# diff iptables-default.txt iptables-vm-running.txt < :PREROUTING ACCEPT [5:180] < :POSTROUTING ACCEPT [6:428] < :OUTPUT ACCEPT [6:428] --- > :PREROUTING ACCEPT [34:2932] > :POSTROUTING ACCEPT [18:1292] > :OUTPUT ACCEPT [18:1292] --- < :INPUT ACCEPT [168:13693] --- > :INPUT ACCEPT [549:40513] < :OUTPUT ACCEPT [114:13252] --- > :OUTPUT ACCEPT [364:41916] > -A FORWARD -m physdev --physdev-in vif1.0 -j ACCEPT [root@xenhost ~]#
And just for reference and explanation, here is the output of "ifconfig" command on the XEN host.
[root@xenhost ~]# ifconfig | grep -A1 HWaddr
eth0 Link encap:Ethernet HWaddr 54:52:00:73:15:33
inet addr:192.168.1.200 Bcast:192.168.1.255 Mask:255.255.255.0
--
peth0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
--
vif0.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
--
vif1.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
--
virbr0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0
--
xenbr0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
UP BROADCAST RUNNING NOARP MTU:1500 Metric:1
[root@xenhost ~]#
Understanding the rules file created by iptables-save command
Many people consider the output of iptables-save to be very cryptic. It is not. Here is a brief explanation of the iptables rules file, created by iptables-save command. We will use the file created at a XEN host.
Note: Notice that virbr0 (activated through libvirtd) is identical on both KVM and XEN hosts. And since we are here to discuss libvirtd related problems, it is ok to use example from either KVM host or XEN host.
[root@xenhost ~]# iptables-save > /root/iptables-vm-running.txt [root@xenhost ~]# cat /root/iptables-vm-running.txt *nat :PREROUTING ACCEPT [34:2932] :POSTROUTING ACCEPT [18:1292] :OUTPUT ACCEPT [18:1292] -A POSTROUTING -s 192.168.122.0/255.255.255.0 -d ! 192.168.122.0/255.255.255.0 -j MASQUERADE COMMIT *filter :INPUT ACCEPT [549:40513] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [364:41916] -A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT -A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT -A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT -A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT -A FORWARD -d 192.168.122.0/255.255.255.0 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT -A FORWARD -s 192.168.122.0/255.255.255.0 -i virbr0 -j ACCEPT -A FORWARD -i virbr0 -o virbr0 -j ACCEPT -A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable -A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable -A FORWARD -m physdev --physdev-in vif1.0 -j ACCEPT COMMIT
- Lines starting with * (asterisk) are the iptables "tables". Such as "nat" table, and "filter" table, as shown in the output above. Each table has "chains" defined in them.
- Lines starting with : (colon) are the name of chains. These lines contain following four pieces of information about a chain. ":ChainName POLICY [Packets:Bytes]"
- The name of the chain right next to the starting colon. e.g. "INPUT" , or "PREROUTING"
- The default policy of the chain. e.g. ACCEPT or DROP or REJECT. 99% of time, you will see ACCEPT here.
- The two values inside the square brackets are number of packets passed through this chain, so far, as well as the number of bytes. e.g. [364:41916] means 364 packets "or" 41916 bytes, have passed through this chain till this point in time.
- The lines starting with a - (hyphen/minus sign) are actually the actual rules which you put in here. "-A" would mean "Add" the rule. "-I" (eye) would mean "Insert" the rule.
As you notice, these rules are no different than the standard rules you type on the command line, or in the bash shell script. The limitation of this style of writing rules (as shown here) is, that shell scripting is not possible.
More details on these rules
For this explanation the following ASCII diagram should be helpful.
+--[VM1]
|
[LAN 192.168.1.0/24]--(eth0:192.168.1.11)[PhysicalHost:DNS+DHCP+NAT](virbr0:192.168.122.1)--[VM2]
|
+--[VM3]
Continuing with the same example, now we move to explain the rules we see created by the libvirtd daemon. I have put line numbers myself, in the beginning of each line, in the output below.
[root@xenhost ~]# cat /root/iptables-vm-running.txt 01) *nat 02) :PREROUTING ACCEPT [34:2932] 03) :POSTROUTING ACCEPT [18:1292] 04) :OUTPUT ACCEPT [18:1292] 05) -A POSTROUTING -s 192.168.122.0/255.255.255.0 -d ! 192.168.122.0/255.255.255.0 -j MASQUERADE 06) COMMIT 07) *filter 08) :INPUT ACCEPT [549:40513] 09) :FORWARD ACCEPT [0:0] 10) :OUTPUT ACCEPT [364:41916] 11) -A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT 12) -A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT 13) -A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT 14) -A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT 15) -A FORWARD -d 192.168.122.0/255.255.255.0 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT 16) -A FORWARD -s 192.168.122.0/255.255.255.0 -i virbr0 -j ACCEPT 17) -A FORWARD -i virbr0 -o virbr0 -j ACCEPT 18) -A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable 19) -A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable 20) -A FORWARD -m physdev --physdev-in vif1.0 -j ACCEPT 21) COMMIT
What's happening here is the following:
- Line 05 defines a rule in the POSTROUTING chain in the "nat" table. It says that any traffic originating from 192.168.122.0/255.255.255.0 network, and trying to reach any other network but itself (-d ! 192.168.122.0/255.255.255.0) should be MASQUERADEd. In simple words, if there is any traffic coming from a VM (because only a VM can be on this 192.168.122.0/24 network), trying to go out from the physical LAN interface (eth0) of the physical host , must be masqueraded. This makes sense, as we don't know what will be the IP of the physical interface of the physical host. This rule facilitates the traffic to go out.
- Line 06 and 21: The word COMMIT indicates the end of list of rules for a particular particualr iptables "table".
- Lines 11 to 14: Virtual machines need a mechanism to get IP automatically and name resolution. DNSMASQ is the small service running on the phyical host, serving both DNS and DHCP requests coming in from the VMs. The DNSMASQ service on this physical host is not reachable over the physical LAN by any other physical host. Neither do it interferes with any DHCP service running elsewhere on the physical LAN. Lines 11 to 14 allow this incoming traffic from the VMs to arrive on the virbr0 interface of the physical host.
- Line 15: In case some traffic originated from a VM, and went out on the internet (because of line05), it will now want to reach back to the VM. For example, on the VM, you tried to pull an httpd.rpm file from a CENTOS mirror. So an HTTP request was originated from this VM, reached the CENTOS mirror (on the internet) and now the packets related to this transaction needs to be allowed back in so they can reach the VM. This requires a rule in the FORWARD chain, shown in line 15, which says that any traffic going to any VM on 192.168.122.0/255.255.255.0 network, exiting through/towards the virbr0 interface of the physical host, and is RELATED to some previous traffic "originated" from that particular VM, must be allowed. Without this, the return traffic/packets will never reach back the VM, and you will get all sorts of weird time-outs.
- Line 16: Basically if this rule does not exist, then the rule defined in line 5 will never get the traffic to MASQUERADE. Means, this line (16) says that any traffic originating from VMs on 192.168.122.0/255.255.255.0 , arriving at virbr0 interface of the physical host, trying to go across the FORWARD chain (to go elsewhere), must be ACCEPTed. Basically this is the line/rule, which transports a packet from virbr0 to eth0 (in this single direction only).
- Line 17: This is simply a multi-direction facilitator for all VMs connected to one virtual network. For example, VM1 and VM3 want to exchange some data on TCP/UDP/ICMP, their traffic would naturally traverse the virtual switch, virbr0. This line/rule allows that traffic to be ACCEPTed.
So far, we are able to send the traffic out from the VM and receive it back as well. Now what?
- Line 18: Any traffic which could not satisfy the rules so far, and still interested in reaching the VMs going towards the virbr0 interface, is REJECTed. For example, you start simple web service on port 80 on VM1. If you some how setup a port to be forwarded from the physical interface of your physical host, to this port 80 on the VM, using DNAT, and try to reach that port, you access will be REJECTed because of this rule!
- Line 19: Any traffic coming in from any VM, through virbr0 interface and trying to go out from the physical interface of the physical host (eth0) will be REJECTed. For example, in line /rule 16 we saw that any traffic coming in from a VM on 192.168.122.0/24 network, and trying to go out the physical interface of the physical machine is allowed. But that is allowed, when the source IP is from the 192.168.122.0/24 network! If there is a VM on the same virbr0, but with different IP, and it tries to go through the physical host towards the other side, it would be denied access.
- Line 20: This is interesting. First, notice that this rule is not part of the default libvirt ruleset. Instead, when a VM on XEN physical host was powered on, this rule got added to the ruleset. When the virtual machine is shutdown, this line gets removed. The code doing this stuff resides in /etc/xen/scripts/vif-common.sh file.
- "physdev" is a special match module, made available in 2.6 kernels. It is used to match the bridge's physical in and out ports. Its basic usage is simple. e.g. " iptables -m physdev --physdev-in <bridge-port> -j <TARGET>" . It is used in situations where an interface may, or may not, (or may never), have an IP address. This means physdev goes a level down to Layer 2, and uses the information in ethernet frames to manage/control/filter that type of traffic. Note, physdev only works with bridged interfaces. Check the iptables man page for more detail on "physdev" .
- In XEN, for each virtual machine, there is a vifX.Y. A so called / virtual patch cable runs from the virtual machine to this port on the bridge. In the (XEN) example we are following, VM1 (aka domain-1, or domain with id 1), is connected to the virtual bridge on the physical host, on vif1.0 port. Here is the rule for reference.
20) -A FORWARD -m physdev --physdev-in vif1.0 -j ACCEPT
- XEN adds this rule to the firewall, to make sure that whatever your previous firewall rules on the physical host, the VM can always communicate to the physical host and vice-versa. The script /etc/xen/scripts/vif-common.sh has facility to add this rule either at the end of the current ruleset of the physical host, or, at the top of the rule-set of the physical host. This is do-able by changing the "-A" to "-I" in this script file. Here is the part of the script for reference:
function frob_iptable()
{
if [ "$command" == "online" ]
then
local c="-A"
else
local c="-D"
fi
. . .
. . .
The interesting stuff
Probably the most interesting stuff is none of these rules are required in the first place, if you just want to run the VM in the private network (virbr0).!!!! . That is right. If you flush all sorts of rules from your physical host, (assuming all chains have default policy set to ACCEPT), the VMs will still be able to communicate to the physical host and vice-versa.
Surprised!!! You might be thinking, "why so many rules in the first place?" . The answer is multi-part:
- Both libvirtd and XEN try to be smart, (and XEN ends up being more smarter). Libvirtd simply (stupidly) removes all your rules and implements it's default rules. Whereas XEN simply adds the (physdev) rule at the end of current set of rules, for each virtual machine.
- These rules are needed, because both libvirtd and XEN assumes there are some rules already on the physical host. Both of them also assume that any existing rules may be restrictive in nature, and may block the DNS, DHCP and other normal traffic requests coming from the VMs, on the FORWARD chain in particular. So both try to make sure that the essential traffic between a VM and the physical host, and between two VMs on the same physical host always gets sent/received properly. That is why both libvirtd and XEN adds the iptables rules.
- Further extension to the previous point, the designers of XEN (being more intelligent), assumed two situations,especially for the FORWARD chain.
- 1) They assumes that the FORWARD chain on any physical host, will be configured with default chain policy as DROP and ACCEPTing only specific rules. So, the "-A" mechanism in the /etc/xen/scripts/vif-common.sh script adds a ACCEPT rule for the bridge port of each new VM.
- 2) If it is not the case explained in 1), then there might be a possibility that the default chain policy is ACCEPT, having certain ACCEPT rules in the chain, and then DROPing or REJECTing all other traffic at the bottom of the chain. In such a case, XEN adding the rule at the bottom of the FORWARD ruleset will be useless. In that case, we change "-A" to "-I" in the vif-common.sh script. In that way, the bridge interface of any VM starting up by XEN, will always be allowed to send/receive traffic.
