Libvirt overwrites the existing iptables rules
From WBITT's Cooker!
Contents |
XEN overwrites the existing iptables rules
Not exactly. It is libvirt which is the culprit, not XEN.
- https://bugzilla.redhat.com/show_bug.cgi?id=227011
- http://lists.fedoraproject.org/pipermail/virt/2010-January/001792.html
- http://forums.gentoo.org/viewtopic-p-6209192.html?sid=1089acac70de96d68aa856d758d7cdfe
- http://wiki.libvirt.org/page/Networking
- http://libvirt.org/formatnetwork.html#examplesPrivate
- http://people.gnome.org/~markmc/virtual-networking.html
Objective / goal of this document
The objective of this document is to identify/clarify the following:
- What are these specific iptable rules?
- Why do we care? and, When do we care?
- Does it matter if we lose these rules?
- Does it matter when we have our virtual machines on a bridged interface, connecting directly to our physical LAN, xenbr0 or br0?
- Does it matter when we have our virtual machines connected only on the private network inside the physical host, virbr0?
- How do we circumvent any problems related to this scenario?
The details
It is observed, that systems which provide KVM or XEN virtualization technologies, sometimes have their iptables firewall rules changed automatically in Dom-0, overwritten by another set of rules.
Note: KVM doesn't have Dom-0. The base OS on the (KVM) physical host will be considered Dom-0 for ease of understanding. (Please. No flame war on this one.). I will use "Physical Host" and "Dom-0" interchangeably for ease of understanding, for a system which hosts one or more virtual machines (i.e. KVM or XEN).
This happens only on those systems, which run libvirtd service. Mistakenly, many people think this is XEN problem. Whereas it is not. First, I would explain, what is the default iptables firewall ruleset on the physical host.
More interestingly this is not much of an issue for the Virtual machines which share the physical ethernet device of the physical host. This is more of an issue, for virutal machines, connected to a virtual (private) network/bridge (192.168.122.0/24) inside the physical host. This means, their traffic passes through NAT configured on the same physical host, whenever they need to go out to the physical interface.
As we are aware that in the shared network device mode, it is easier (aka painless), to access the virtual machines, from the main physical network. This is because they (the virtual machines) have the IP from the same scheme/range/subnet as of the physical device of the physical host they are on. This is ideal for so called server mode, where all the virtual machines, are actually serving some services on the LAN.
On the other hand, in the virtual network mode, it is difficult (aka painful), to access the virtual machines, from the main physical network. It is because, they (the virtual machines) have the IP from a private scheme (192.168.122.0/24), and they sit behind a NAT router which is created by the virtual machine manager, on the physical host. That means, their traffic can go out of the network, NATed, and related traffic can get back in. But, if we want to access these (virtual servers (running on private IPs) from the LAN, we need to configure some traffic redirection (DNAT), on the physical host. This mode, of creating virtual machines on the private network, is ideal for private testing of various type of stuff. BUT, when you are in a situation, where you need your private server to be acccessed from outside, and you cannot use the "shared network device" mode for certain/whatever reasons, then you need to figure out a way to pass your traffic coming from outside to your VM. And when you try to do that, you see your iptables rules of your physical host machine, being clobbered/over-written by the libvirtd's rules, then you get frustrated. And that, is the reason, behind this document.
The default iptables rules on a KVM physical host
Here is a default iptables rule-set from a Fedora13 physical host. The default firewall (iptables service) was stopped when libvirtd service was started, at system boot.
[root@training ~]# iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination ACCEPT udp -- anywhere anywhere udp dpt:domain ACCEPT tcp -- anywhere anywhere tcp dpt:domain ACCEPT udp -- anywhere anywhere udp dpt:bootps ACCEPT tcp -- anywhere anywhere tcp dpt:bootps Chain FORWARD (policy ACCEPT) target prot opt source destination ACCEPT all -- anywhere 192.168.122.0/24 state RELATED,ESTABLISHED ACCEPT all -- 192.168.122.0/24 anywhere ACCEPT all -- anywhere anywhere REJECT all -- anywhere anywhere reject-with icmp-port-unreachable REJECT all -- anywhere anywhere reject-with icmp-port-unreachable Chain OUTPUT (policy ACCEPT) target prot opt source destination [root@training ~]# iptables -L -t nat Chain PREROUTING (policy ACCEPT) target prot opt source destination Chain POSTROUTING (policy ACCEPT) target prot opt source destination MASQUERADE all -- 192.168.122.0/24 !192.168.122.0/24 Chain OUTPUT (policy ACCEPT) target prot opt source destination
Let's save these rules in a file, so we can load the defaults any time we need to.
[root@training ~]# iptables-save > /root/iptables.default.txt
I will show you these rules from this file for easier understanding:
[root@training ~]# cat /root/iptables.default.txt *nat :PREROUTING ACCEPT [661:21364] :POSTROUTING ACCEPT [58069:3670258] :OUTPUT ACCEPT [58069:3670258] -A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE COMMIT *filter :INPUT ACCEPT [1212620:674141323] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [1518464:780474182] -A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT -A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT -A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT -A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT -A FORWARD -d 192.168.122.0/24 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT -A FORWARD -s 192.168.122.0/24 -i virbr0 -j ACCEPT -A FORWARD -i virbr0 -o virbr0 -j ACCEPT -A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable -A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable COMMIT [root@training ~]#
Lets start a VM:
[root@training init.d]# virsh start linuxvm Domain linuxvm started [root@training init.d]# virsh list Id Name State ---------------------------------- 2 linuxvm running
Here is the output of ifconfig command, after a VM is started.
[root@training init.d]# ifconfig | grep -A1 HWaddr
eth0 Link encap:Ethernet HWaddr 00:26:2D:95:AB:FA
inet addr:192.168.1.3 Bcast:192.168.1.255 Mask:255.255.255.0
--
virbr0 Link encap:Ethernet HWaddr 26:45:88:61:E0:CC
inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0
--
vnet0 Link encap:Ethernet HWaddr 26:45:88:61:E0:CC
inet6 addr: fe80::2445:88ff:fe61:e0cc/64 Scope:Link
[root@training init.d]#
The default iptables rules on a XEN physical host
Here is the default iptables rules as seen when a XEN physical host boots up with default configurations. Please note that the firewall service was configured to be stopped on system boot. Which means, that these rules were added by some mechanism (libvirtd) on the XEN host.
[root@xenhost ~]# iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination ACCEPT udp -- anywhere anywhere udp dpt:domain ACCEPT tcp -- anywhere anywhere tcp dpt:domain ACCEPT udp -- anywhere anywhere udp dpt:bootps ACCEPT tcp -- anywhere anywhere tcp dpt:bootps Chain FORWARD (policy ACCEPT) target prot opt source destination ACCEPT all -- anywhere 192.168.122.0/24 state RELATED,ESTABLISHED ACCEPT all -- 192.168.122.0/24 anywhere ACCEPT all -- anywhere anywhere REJECT all -- anywhere anywhere reject-with icmp-port-unreachable REJECT all -- anywhere anywhere reject-with icmp-port-unreachable Chain OUTPUT (policy ACCEPT) target prot opt source destination [root@xenhost ~]# iptables -L -t nat Chain PREROUTING (policy ACCEPT) target prot opt source destination Chain POSTROUTING (policy ACCEPT) target prot opt source destination MASQUERADE all -- 192.168.122.0/24 !192.168.122.0/24 Chain OUTPUT (policy ACCEPT) target prot opt source destination [root@xenhost ~]#
Save these rules for future reference:
[root@xenhost ~]# iptables-save > /root/iptables-default.txt
Have a look at the resultant file to understand the rules better:
[root@xenhost ~]# cat /root/iptables-default.txt *nat :PREROUTING ACCEPT [5:180] :POSTROUTING ACCEPT [6:428] :OUTPUT ACCEPT [6:428] -A POSTROUTING -s 192.168.122.0/255.255.255.0 -d ! 192.168.122.0/255.255.255.0 -j MASQUERADE COMMIT *filter :INPUT ACCEPT [168:13693] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [114:13252] -A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT -A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT -A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT -A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT -A FORWARD -d 192.168.122.0/255.255.255.0 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT -A FORWARD -s 192.168.122.0/255.255.255.0 -i virbr0 -j ACCEPT -A FORWARD -i virbr0 -o virbr0 -j ACCEPT -A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable -A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable COMMIT [root@xenhost ~]#
Now, let's start a VM on the XEN host, configured to run, while connected to the xen host private network, virbr0.
[root@xenhost ~]# virsh list --all Id Name State ---------------------------------- 0 Domain-0 running - miniweb shut off [root@xenhost ~]# virsh start miniweb Domain miniweb started [root@xenhost ~]# virsh list --all Id Name State ---------------------------------- 0 Domain-0 running 1 miniweb idle [root@xenhost ~]#
Let's look at the rules again, after starting the VM:
[root@xenhost ~]# iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination ACCEPT udp -- anywhere anywhere udp dpt:domain ACCEPT tcp -- anywhere anywhere tcp dpt:domain ACCEPT udp -- anywhere anywhere udp dpt:bootps ACCEPT tcp -- anywhere anywhere tcp dpt:bootps Chain FORWARD (policy ACCEPT) target prot opt source destination ACCEPT all -- anywhere 192.168.122.0/24 state RELATED,ESTABLISHED ACCEPT all -- 192.168.122.0/24 anywhere ACCEPT all -- anywhere anywhere REJECT all -- anywhere anywhere reject-with icmp-port-unreachable REJECT all -- anywhere anywhere reject-with icmp-port-unreachable ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-in vif1.0 Chain OUTPUT (policy ACCEPT) target prot opt source destination [root@xenhost ~]# [root@xenhost ~]# iptables -L -t nat Chain PREROUTING (policy ACCEPT) target prot opt source destination Chain POSTROUTING (policy ACCEPT) target prot opt source destination MASQUERADE all -- 192.168.122.0/24 !192.168.122.0/24 Chain OUTPUT (policy ACCEPT) target prot opt source destination [root@xenhost ~]#
Lets save these rules in another file for comparison.
[root@xenhost ~]# iptables-save > /root/iptables-vm-running.txt [root@xenhost ~]# cat /root/iptables-vm-running.txt *nat :PREROUTING ACCEPT [34:2932] :POSTROUTING ACCEPT [18:1292] :OUTPUT ACCEPT [18:1292] -A POSTROUTING -s 192.168.122.0/255.255.255.0 -d ! 192.168.122.0/255.255.255.0 -j MASQUERADE COMMIT *filter :INPUT ACCEPT [549:40513] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [364:41916] -A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT -A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT -A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT -A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT -A FORWARD -d 192.168.122.0/255.255.255.0 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT -A FORWARD -s 192.168.122.0/255.255.255.0 -i virbr0 -j ACCEPT -A FORWARD -i virbr0 -o virbr0 -j ACCEPT -A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable -A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable -A FORWARD -m physdev --physdev-in vif1.0 -j ACCEPT COMMIT
Here I run the diff on the two files on the XEN host and try to see the differences.
[root@xenhost ~]# diff iptables-default.txt iptables-vm-running.txt < :PREROUTING ACCEPT [5:180] < :POSTROUTING ACCEPT [6:428] < :OUTPUT ACCEPT [6:428] --- > :PREROUTING ACCEPT [34:2932] > :POSTROUTING ACCEPT [18:1292] > :OUTPUT ACCEPT [18:1292] --- < :INPUT ACCEPT [168:13693] --- > :INPUT ACCEPT [549:40513] < :OUTPUT ACCEPT [114:13252] --- > :OUTPUT ACCEPT [364:41916] > -A FORWARD -m physdev --physdev-in vif1.0 -j ACCEPT [root@xenhost ~]#
And just for reference and explanation, here is the output of "ifconfig" command on the XEN host.
[root@xenhost ~]# ifconfig | grep -A1 HWaddr
eth0 Link encap:Ethernet HWaddr 54:52:00:73:15:33
inet addr:192.168.1.200 Bcast:192.168.1.255 Mask:255.255.255.0
--
peth0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
--
vif0.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
--
vif1.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
--
virbr0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0
--
xenbr0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
UP BROADCAST RUNNING NOARP MTU:1500 Metric:1
[root@xenhost ~]#
Understanding the rules file created by iptables-save command
Many people consider the output of iptables-save to be very cryptic. It is not. Here is a brief explanation of the iptables rules file, created by iptables-save command. We will use the file created at a XEN host.
Note: Notice that virbr0 (activated through libvirtd) is identical on both KVM and XEN hosts. And since we are here to discuss libvirtd related problems, it is ok to use example from either KVM host or XEN host.
[root@xenhost ~]# iptables-save > /root/iptables-vm-running.txt [root@xenhost ~]# cat /root/iptables-vm-running.txt *nat :PREROUTING ACCEPT [34:2932] :POSTROUTING ACCEPT [18:1292] :OUTPUT ACCEPT [18:1292] -A POSTROUTING -s 192.168.122.0/255.255.255.0 -d ! 192.168.122.0/255.255.255.0 -j MASQUERADE COMMIT *filter :INPUT ACCEPT [549:40513] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [364:41916] -A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT -A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT -A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT -A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT -A FORWARD -d 192.168.122.0/255.255.255.0 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT -A FORWARD -s 192.168.122.0/255.255.255.0 -i virbr0 -j ACCEPT -A FORWARD -i virbr0 -o virbr0 -j ACCEPT -A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable -A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable -A FORWARD -m physdev --physdev-in vif1.0 -j ACCEPT COMMIT
- Lines starting with * (asterisk) are the iptables "tables". Such as "nat" table, and "filter" table, as shown in the output above. Each table has "chains" defined in them.
- Lines starting with : (colon) are the name of chains. These lines contain following four pieces of information about a chain. ":ChainName POLICY [Packets:Bytes]"
- The name of the chain right next to the starting colon. e.g. "INPUT" , or "PREROUTING"
- The default policy of the chain. e.g. ACCEPT or DROP or REJECT. 99% of time, you will see ACCEPT here.
- The two values inside the square brackets are number of packets passed through this chain, so far, as well as the number of bytes. e.g. [364:41916] means 364 packets "or" 41916 bytes, have passed through this chain till this point in time.
- The lines starting with a - (hyphen/minus sign) are actually the actual rules which you put in here. "-A" would mean "Add" the rule. "-I" (eye) would mean "Insert" the rule.
As you notice, these rules are no different than the standard rules you type on the command line, or in the bash shell script. The limitation of this style of writing rules (as shown here) is, that shell scripting is not possible.
Understanding the iptables rules created by libvirtd
Continuing with the same example, now we move to explain the rules we see created by the libvirtd daemon. For this explanation the following ASCII diagram should be helpful.
+--[VM1]
|
[LAN 192.168.1.0/24]--(eth0:192.168.1.11)[PhysicalHost:dns+dhcp](virbr0:192.168.122.1)--[VM2]
|
+--[VM3]
