Libvirt overwrites the existing iptables rules
From WBITT's Cooker!
Contents |
XEN overwrites the existing iptables rules
Not exactly. It is libvirt which is the culprit, not XEN.
- https://bugzilla.redhat.com/show_bug.cgi?id=227011
- http://lists.fedoraproject.org/pipermail/virt/2010-January/001792.html
- http://forums.gentoo.org/viewtopic-p-6209192.html?sid=1089acac70de96d68aa856d758d7cdfe
- http://wiki.libvirt.org/page/Networking
- http://libvirt.org/formatnetwork.html#examplesPrivate
Objective / goal of this document
The objective of this document is to identify/clarify the following:
- What are these specific iptable rules?
- Why do we care? and, When do we care?
- Does it matter if we lose these rules?
- Does it matter when we have our virtual machines on a bridged interface, connecting directly to our physical LAN, xenbr0 or br0?
- Does it matter when we have our virtual machines connected only on the private network inside the physical host, virbr0?
- How do we circumvent any problems related to this scenario?
The details
It is observed, that systems which provide KVM or XEN virtualization technologies, sometimes have their iptables firewall rules changed automatically in Dom-0, overwritten by another set of rules.
Note: KVM doesn't have Dom-0. The base OS on the physical host will also be considered Dom-0 for ease of understanding. (Please. No flame war on this one.). I will use "Physical Host" and "Dom-0" interchangeably for ease of understanding, for a system which hosts one or more virtual machines.
This happens only on those systems, which run libvirtd service. Mistakenly, many people think this is XEN problem. Whereas it is not. First, I would explain, what is the default iptables firewall ruleset on the physical host.
The default iptables rules on a KVM physical host
Here is a default iptables rule-set from a Fedora13 physical host. It is assumed that the default firewall (iptables service) was stopped when libvirtd service was started.
[root@training ~]# iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination ACCEPT udp -- anywhere anywhere udp dpt:domain ACCEPT tcp -- anywhere anywhere tcp dpt:domain ACCEPT udp -- anywhere anywhere udp dpt:bootps ACCEPT tcp -- anywhere anywhere tcp dpt:bootps Chain FORWARD (policy ACCEPT) target prot opt source destination ACCEPT all -- anywhere 192.168.122.0/24 state RELATED,ESTABLISHED ACCEPT all -- 192.168.122.0/24 anywhere ACCEPT all -- anywhere anywhere REJECT all -- anywhere anywhere reject-with icmp-port-unreachable REJECT all -- anywhere anywhere reject-with icmp-port-unreachable Chain OUTPUT (policy ACCEPT) target prot opt source destination [root@training ~]# iptables -L -t nat Chain PREROUTING (policy ACCEPT) target prot opt source destination Chain POSTROUTING (policy ACCEPT) target prot opt source destination MASQUERADE all -- 192.168.122.0/24 !192.168.122.0/24 Chain OUTPUT (policy ACCEPT) target prot opt source destination
Let's save these rules in a file, so we can load the defaults any time we need to.
[root@training ~]# iptables-save > /root/iptables.default.txt
I will show you these rules from this file for easier understanding:
[root@training ~]# cat /root/iptables.default.txt # Generated by iptables-save v1.4.7 on Sun Jul 11 19:51:07 2010 *nat :PREROUTING ACCEPT [661:21364] :POSTROUTING ACCEPT [58069:3670258] :OUTPUT ACCEPT [58069:3670258] -A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE COMMIT # Completed on Sun Jul 11 19:51:07 2010 # Generated by iptables-save v1.4.7 on Sun Jul 11 19:51:07 2010 *filter :INPUT ACCEPT [1212620:674141323] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [1518464:780474182] -A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT -A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT -A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT -A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT -A FORWARD -d 192.168.122.0/24 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT -A FORWARD -s 192.168.122.0/24 -i virbr0 -j ACCEPT -A FORWARD -i virbr0 -o virbr0 -j ACCEPT -A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable -A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable COMMIT # Completed on Sun Jul 11 19:51:07 2010 [root@training ~]#
The default iptables rules on a XEN physical host
Here is the default iptables rules as seen when a XEN physical host boots up with default configurations. Please note that the firewall service was configured to be stopped on system boot. Which means, that these rules were added by some mechanism (libvirtd) on the XEN host.
[root@xenhost ~]# iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination ACCEPT udp -- anywhere anywhere udp dpt:domain ACCEPT tcp -- anywhere anywhere tcp dpt:domain ACCEPT udp -- anywhere anywhere udp dpt:bootps ACCEPT tcp -- anywhere anywhere tcp dpt:bootps Chain FORWARD (policy ACCEPT) target prot opt source destination ACCEPT all -- anywhere 192.168.122.0/24 state RELATED,ESTABLISHED ACCEPT all -- 192.168.122.0/24 anywhere ACCEPT all -- anywhere anywhere REJECT all -- anywhere anywhere reject-with icmp-port-unreachable REJECT all -- anywhere anywhere reject-with icmp-port-unreachable Chain OUTPUT (policy ACCEPT) target prot opt source destination [root@xenhost ~]# iptables -L -t nat Chain PREROUTING (policy ACCEPT) target prot opt source destination Chain POSTROUTING (policy ACCEPT) target prot opt source destination MASQUERADE all -- 192.168.122.0/24 !192.168.122.0/24 Chain OUTPUT (policy ACCEPT) target prot opt source destination [root@xenhost ~]#
Save these rules for future reference:
[root@xenhost ~]# iptables-save > /root/iptables-default.txt
Have a look at the resultant file to understand the rules better:
[root@xenhost ~]# cat /root/iptables-default.txt # Generated by iptables-save v1.3.5 on Sun Jul 11 19:08:33 2010 *nat :PREROUTING ACCEPT [5:180] :POSTROUTING ACCEPT [6:428] :OUTPUT ACCEPT [6:428] -A POSTROUTING -s 192.168.122.0/255.255.255.0 -d ! 192.168.122.0/255.255.255.0 -j MASQUERADE COMMIT # Completed on Sun Jul 11 19:08:33 2010 # Generated by iptables-save v1.3.5 on Sun Jul 11 19:08:33 2010 *filter :INPUT ACCEPT [168:13693] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [114:13252] -A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT -A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT -A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT -A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT -A FORWARD -d 192.168.122.0/255.255.255.0 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT -A FORWARD -s 192.168.122.0/255.255.255.0 -i virbr0 -j ACCEPT -A FORWARD -i virbr0 -o virbr0 -j ACCEPT -A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable -A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable COMMIT # Completed on Sun Jul 11 19:08:33 2010 [root@xenhost ~]#
Now, let's start a VM on the XEN host, configured to run, while connected to the xen host private network, virbr0.
[root@xenhost ~]# virsh list --all Id Name State ---------------------------------- 0 Domain-0 running - miniweb shut off [root@xenhost ~]# virsh start miniweb Domain miniweb started [root@xenhost ~]# virsh list --all Id Name State ---------------------------------- 0 Domain-0 running 1 miniweb idle [root@xenhost ~]#
Let's look at the rules again:
[root@xenhost ~]# iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination ACCEPT udp -- anywhere anywhere udp dpt:domain ACCEPT tcp -- anywhere anywhere tcp dpt:domain ACCEPT udp -- anywhere anywhere udp dpt:bootps ACCEPT tcp -- anywhere anywhere tcp dpt:bootps Chain FORWARD (policy ACCEPT) target prot opt source destination ACCEPT all -- anywhere 192.168.122.0/24 state RELATED,ESTABLISHED ACCEPT all -- 192.168.122.0/24 anywhere ACCEPT all -- anywhere anywhere REJECT all -- anywhere anywhere reject-with icmp-port-unreachable REJECT all -- anywhere anywhere reject-with icmp-port-unreachable ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-in vif1.0 Chain OUTPUT (policy ACCEPT) target prot opt source destination [root@xenhost ~]# [root@xenhost ~]# iptables -L -t nat Chain PREROUTING (policy ACCEPT) target prot opt source destination Chain POSTROUTING (policy ACCEPT) target prot opt source destination MASQUERADE all -- 192.168.122.0/24 !192.168.122.0/24 Chain OUTPUT (policy ACCEPT) target prot opt source destination [root@xenhost ~]#
Lets save these rules in another file for comparison.
[root@xenhost ~]# iptables-save > /root/iptables-vm-running.txt [root@xenhost ~]# cat /root/iptables-vm-running.txt # Generated by iptables-save v1.3.5 on Sun Jul 11 19:19:03 2010 *nat :PREROUTING ACCEPT [34:2932] :POSTROUTING ACCEPT [18:1292] :OUTPUT ACCEPT [18:1292] -A POSTROUTING -s 192.168.122.0/255.255.255.0 -d ! 192.168.122.0/255.255.255.0 -j MASQUERADE COMMIT # Completed on Sun Jul 11 19:19:03 2010 # Generated by iptables-save v1.3.5 on Sun Jul 11 19:19:03 2010 *filter :INPUT ACCEPT [549:40513] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [364:41916] -A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT -A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT -A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT -A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT -A FORWARD -d 192.168.122.0/255.255.255.0 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT -A FORWARD -s 192.168.122.0/255.255.255.0 -i virbr0 -j ACCEPT -A FORWARD -i virbr0 -o virbr0 -j ACCEPT -A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable -A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable -A FORWARD -m physdev --physdev-in vif1.0 -j ACCEPT COMMIT # Completed on Sun Jul 11 19:19:03 2010
Here I run the diff on the two files on the XEN host and try to see the differences.
[root@xenhost ~]# diff iptables-default.txt iptables-vm-running.txt 1c1 < # Generated by iptables-save v1.3.5 on Sun Jul 11 19:08:33 2010 --- > # Generated by iptables-save v1.3.5 on Sun Jul 11 19:19:03 2010 3,5c3,5 < :PREROUTING ACCEPT [5:180] < :POSTROUTING ACCEPT [6:428] < :OUTPUT ACCEPT [6:428] --- > :PREROUTING ACCEPT [34:2932] > :POSTROUTING ACCEPT [18:1292] > :OUTPUT ACCEPT [18:1292] 8,9c8,9 < # Completed on Sun Jul 11 19:08:33 2010 < # Generated by iptables-save v1.3.5 on Sun Jul 11 19:08:33 2010 --- > # Completed on Sun Jul 11 19:19:03 2010 > # Generated by iptables-save v1.3.5 on Sun Jul 11 19:19:03 2010 11c11 < :INPUT ACCEPT [168:13693] --- > :INPUT ACCEPT [549:40513] 13c13 < :OUTPUT ACCEPT [114:13252] --- > :OUTPUT ACCEPT [364:41916] 22a23 > -A FORWARD -m physdev --physdev-in vif1.0 -j ACCEPT 24c25 < # Completed on Sun Jul 11 19:08:33 2010 --- > # Completed on Sun Jul 11 19:19:03 2010 [root@xenhost ~]#
