Libvirt overwrites the existing iptables rules
From WBITT's Cooker!
(→The default iptables rules on a KVM physical host) |
|||
| Line 1: | Line 1: | ||
| + | <b> NOTE: This document may contain text, which is horribly wrong. This page is just my scratch pad for my personal notes, on this beast. So until I get it right, you are advised not to read anything in this document. It may shatter your existing iptables/virtualization concepts.</b> | ||
| + | |||
Please send your thoughts/comments to: kamran at wbitt dot com. | Please send your thoughts/comments to: kamran at wbitt dot com. | ||
Revision as of 07:57, 8 February 2011
NOTE: This document may contain text, which is horribly wrong. This page is just my scratch pad for my personal notes, on this beast. So until I get it right, you are advised not to read anything in this document. It may shatter your existing iptables/virtualization concepts.
Please send your thoughts/comments to: kamran at wbitt dot com.
Summary
Traditional way to host any service (web, mail, db, etc), was to acquire separate physical servers and install necessary software on them. However, the world realized that most of the servers are idling around 5% of CPU usage. (Mostly web servers). To avoid wastage of resources, the smart people introduced virtualization, and everyone jumped at it. It turned out that by introducing virtualization to save resources, the world was also solving another problem in parallel. The security problem. By virtualization, everyone (if they want to), can separate various service domains. Such as a DB server running as a VM, a web server running separately as a VM, and a mail server running as a separate VM; all running on a single physical machine. This way, if a cracker would gain access to one VM, the other service domains would remain safe. Thus reducing the level and severity of service outage.
Note: The word "domain" in this text has nothing to do with the "domain/workgroup" concept as used by Microsoft.
XEN and KVM virtualization provides two models, to setup networking on the physical host.
- Shared Physical Network Device (xenbr0 in XEN) (br0 in KVM)
- NAT based Virtual Network (virbr0)
For service hosting purposes, it is best to use the shared device model. In this model, all VMs, share the same physical network card of the host, as well as the IP scheme, to which, the physical network card of the physical host is connected to. This would mean, of-course to have more available IPs of the same network scheme, so they can be assigned to the VMs. In this way, all VMs appear on the network in the same way, as if they were just another physical server. This is the easiest way to connect your VMs to your infrastructure, and make them accessible.
Below is an example of such situation, where a Web Server and a Mail Server runs as virtual machines, in a publicly accessible server (XEN), rented from a server rental company.
At times there are requirements (due to security reasons), or restrictions/limitations from the infrastructure side, and shared device mode cannot be used. In that case, we have to use the NAT device/model. Few cases, when you must use the NAT model, are:
- When your servers are on the public network, such as in a public data center, and you have a limited number of public IPs to use for your servers. You may have to pay extra (along with submitting a justification), for extra public IPs; and you don't want to.
- When your service provider has an infrastructure, which was not built with virtualization in mind, during its design. Such infrastructures have restricted way of billing the servers, services or traffic, normally restricted to the MAC address of the physical network card of your servers.
NAT model has a long list of its restrictions, limitations and problems. Most notably, the way these VMs should be accessed is through a single visible / public IP of the physical host they are hosted on.
The problem is faced, when a virtual machine needs to be accessed from the outside-network, instead of a virtual machine accessing the outside-network. [Note the difference, here]. The virtual machine in discussion is on a (non-routeable) private network, inside a XEN or KVM host, connected to virbr0 interface of the physical host. This is the type of setup, which is used in situations, where you have limitations/restrictions connecting your VMs directly to the outside-network. ServerBeach, a reputed server-rental company / hosting provider, is one example.
In such scenario, naturally, the administrator of the physical host, would create certain forwarding (or DNAT) rules, to allow the traffic coming from the outside, to be redirected to a VM. While doing so, it is observed that whenever the services libvirt or xen, are restarted, or the physical host is restarted, any iptables rules written by the administrator are over-written (by some un-known process). For long, XEN has been (incorrectly) accused for doing so.
In this text, we will explain that it is not XEN, which is over-writing the iptables rules. It is actually "libvirt" which is doing so. And so far, by the time of this writing, there is no solution for it. It is a known bug and still in the OPEN state at redhat and fedora bugzilla websites. (https://bugzilla.redhat.com/show_bug.cgi?id=227011)
As similar restrictions may exist in smaller networks and setups, the diagram below, which is a typical lab setup, will help to explain, reproduce and analyse the problem.
Objective / goal of this document
The objective of this document is to identify/clarify the following:
- What are these specific iptable rules?
- Why do we care? and, When should we care?
- Does it matter if we lose these rules?
- Does it matter when we have our virtual machines on a bridged interface, connecting directly to our physical LAN, xenbr0 or br0?
- Does it matter when we have our virtual machines connected only on the private network inside the physical host, virbr0?
- How do we circumvent any problems related to this scenario?
The test setup
In our test setup, we have two Dell Optiplex PCs, and a laptop to access them, and their VMs, all connected to a switched network. The network is also connected to internet, through DSL. Both Dell PCs have CENTOS 5.5 x86_64, installed on them, and were updated using the "yum update" command. One of them is XEN host, and the other is KVM host.
- XENhost (192.168.1.201) [CENTOS 5.5 64bit]
- KVMhost (192.168.1.202) [CENTOS 5.5 64bit]
- Laptop (192.168.1.5) [Fedora 14, OS on the client machine is irrelevant to the discussion]
Problem analysis
Example/simple firewall/iptables rules on our XEN server
To analyse the problem, first we have created a very simple iptables ruleset. The three LOG rules are added to each of the INPUT, FORWARD and OUTPUT chains of the filter table. These rules do nothing, but log any traffic passing through that chain. There is no advantage of doing so (logging the traffic) in our scenario. However, if later on, these rules disappear, or get changed, will prove our point that it is actually libvirt, which is messing with them.
Here is our rules set. Notice that we also have a rule to block any incoming SMTP request to this server, solely for the sake of example.
[root@xenhost ~]# iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination LOG all -- anywhere anywhere LOG level warning REJECT tcp -- anywhere anywhere tcp dpt:smtp reject-with icmp-port-unreachable Chain FORWARD (policy ACCEPT) target prot opt source destination LOG all -- anywhere anywhere LOG level warning Chain OUTPUT (policy ACCEPT) target prot opt source destination LOG all -- anywhere anywhere LOG level warning [root@xenhost ~]#
These rules are actually coming from the file: /etc/sysconfig/iptables. The iptables service reads this file and applies these rules, when it is started.
[root@xenhost ~]# cat /etc/sysconfig/iptables *filter :INPUT ACCEPT [1447:108674] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [736:80968] -A INPUT -i eth0 -j LOG -A INPUT -i eth0 -p tcp -m tcp --dport 25 -j REJECT --reject-with icmp-port-unreachable -A FORWARD -j LOG -A OUTPUT -o eth0 -j LOG COMMIT *nat :PREROUTING ACCEPT [1208:55184] :POSTROUTING ACCEPT [10:464] :OUTPUT ACCEPT [10:464] COMMIT [root@xenhost ~]#
It is important to note, that on a server which is placed-on / connected-to, Internet, normally controls the incoming and outgoing traffic. (Normally only the incoming traffic). An internet web server, for example, is normally protected by a host-firewall. i.e. iptables rules configured on the server itself, to restrict/control traffic arriving on it's INPUT chain. It is important to note that since the server administrator "trusts" the server itself, it does not have to control the outgoing traffic, which is the OUTPUT chain. Thus no rules are configured on the OUTPUT chain. It is also important to note, that since the server is the end-point of any two way internet communication, there is never a need to configure any rule on the FORWARD chain.
In case of a XEN or KVM host, we would be having one or more virtual machines, behind the said server. The traffic to/from the VMs will have to pass through the FORWARD chain on the physical host. Thus FORWARD chain has significant importance in this case, and needs to be protected.
We would still follow the principle of not hosting any un-necessary services on the physical host itself, such as SMTP, HTTP, etc etc. However, we would still need to protect it from any traffic and attacks directly targeted for it. Such as various forms of ICMP attacks.
Note, that we do not intend to protect the virtual machines hosted on this server, using the iptables rules on the physical host. That is, (a) very bad practice, as it complicates the firewall rules each time a VM is added/removed/started/shutdown, (b) it over-loads the server to perform in the firewall role, in addition to the VM hosting. The best practice is to let the physical host do only the VM management only, and protect the virtual machines using the host firewalls on the VMs themselves. If resources on the physical host permits, then an additional VM can be created working solely as a firewall for all the VMs. However, this is beyond the scope of this document.
The default iptables rules on a XEN physical host
For the time being, for the sake of ease of understanding, we have stopped the iptables service. This will help us observe, what iptables rules are set up, when a XEN or KVM host boots up.
Below are the iptables rules found on a XEN host. Note that there is no VM running on the host at this time.
[root@xenhost ~]# iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination ACCEPT udp -- anywhere anywhere udp dpt:domain ACCEPT tcp -- anywhere anywhere tcp dpt:domain ACCEPT udp -- anywhere anywhere udp dpt:bootps ACCEPT tcp -- anywhere anywhere tcp dpt:bootps Chain FORWARD (policy ACCEPT) target prot opt source destination ACCEPT all -- anywhere 192.168.122.0/24 state RELATED,ESTABLISHED ACCEPT all -- 192.168.122.0/24 anywhere ACCEPT all -- anywhere anywhere REJECT all -- anywhere anywhere reject-with icmp-port-unreachable REJECT all -- anywhere anywhere reject-with icmp-port-unreachable Chain OUTPUT (policy ACCEPT) target prot opt source destination [root@xenhost ~]# [root@xenhost ~]# iptables -L -t nat Chain PREROUTING (policy ACCEPT) target prot opt source destination Chain POSTROUTING (policy ACCEPT) target prot opt source destination MASQUERADE all -- 192.168.122.0/24 !192.168.122.0/24 Chain OUTPUT (policy ACCEPT) target prot opt source destination [root@xenhost ~]#
Save these rules, so we can study them in a different format, as well as restore them when there is a need to:
[root@xenhost ~]# iptables-save > /root/xenhost-iptables-default.txt
Have a look at this file to understand the rules better:
[root@xenhost ~]# cat /root/xenhost-iptables-default.txt *nat :PREROUTING ACCEPT [5:180] :POSTROUTING ACCEPT [6:428] :OUTPUT ACCEPT [6:428] -A POSTROUTING -s 192.168.122.0/255.255.255.0 -d ! 192.168.122.0/255.255.255.0 -j MASQUERADE COMMIT *filter :INPUT ACCEPT [168:13693] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [114:13252] -A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT -A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT -A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT -A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT -A FORWARD -d 192.168.122.0/255.255.255.0 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT -A FORWARD -s 192.168.122.0/255.255.255.0 -i virbr0 -j ACCEPT -A FORWARD -i virbr0 -o virbr0 -j ACCEPT -A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable -A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable COMMIT [root@xenhost ~]#
The default iptables rules on a KVM physical host
Here is a default iptables rule-set from a KVM based CentOS 5.5 physical host. The default firewall (iptables service) was stopped when libvirtd service was started, at system boot. Also note that no VM is running on the KVM host at the moment.
[root@kvmhost ~]# iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination ACCEPT udp -- anywhere anywhere udp dpt:domain ACCEPT tcp -- anywhere anywhere tcp dpt:domain ACCEPT udp -- anywhere anywhere udp dpt:bootps ACCEPT tcp -- anywhere anywhere tcp dpt:bootps Chain FORWARD (policy ACCEPT) target prot opt source destination ACCEPT all -- anywhere 192.168.122.0/24 state RELATED,ESTABLISHED ACCEPT all -- 192.168.122.0/24 anywhere ACCEPT all -- anywhere anywhere REJECT all -- anywhere anywhere reject-with icmp-port-unreachable REJECT all -- anywhere anywhere reject-with icmp-port-unreachable Chain OUTPUT (policy ACCEPT) target prot opt source destination [root@kvmhost ~]# [root@kvmhost ~]# iptables -L -t nat Chain PREROUTING (policy ACCEPT) target prot opt source destination Chain POSTROUTING (policy ACCEPT) target prot opt source destination MASQUERADE all -- 192.168.122.0/24 !192.168.122.0/24 Chain OUTPUT (policy ACCEPT) target prot opt source destination [root@kvmhost ~]#
Let's save these rules in a file, so we can study them in a different format, as well as, load the defaults any time we need to.
[root@kvmhost ~]# iptables-save > /root/kvmhost-iptables.default.txt
Lets look at this file for easier understanding of these rules:
[root@kvmhost ~]# cat /root/kvmhost-iptables.default.txt *nat :PREROUTING ACCEPT [661:21364] :POSTROUTING ACCEPT [58069:3670258] :OUTPUT ACCEPT [58069:3670258] -A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE COMMIT *filter :INPUT ACCEPT [1212620:674141323] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [1518464:780474182] -A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT -A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT -A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT -A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT -A FORWARD -d 192.168.122.0/24 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT -A FORWARD -s 192.168.122.0/24 -i virbr0 -j ACCEPT -A FORWARD -i virbr0 -o virbr0 -j ACCEPT -A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable -A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable COMMIT [root@kvmhost ~]#
Important note about the default iptables service
It is important to note, that in the two examples above, iptables service was disabled only to avoid possible confusion, that the iptables rules shown may have been coming from the iptables service. Otherwise, we do not recommend that you disable your iptables service. If the default iptables rules setup by the iptables service are not suitable for your particular case, then you can adjust them as per your needs. The bottom line is, that you must have some level of protection against unwanted traffic.
Another point to be noted here is that, ideally, on a physical host, you should not be running any publicly accessible service (a.k.a. public serving service) other than ssh, in your production environments. It means that you should not use your physical host to serve out web pages/websites , or run mail services, or FTP, etc. You do not want that a cracker exploits any of the extra services' vulnerabilities on your physical host, gain root access, and in-turn gain access to "all" virtual machines hosted on this physical host. Even SSH should be used with key based authentication only. And further restricted to be accessed only from the IP addresses, of the locations you manage your physical hosts from.
Understanding the rules file created by iptables-save command
Many people consider the output of iptables-save to be very cryptic. It is not. Here is a brief explanation of the iptables rules file, created by iptables-save command. We will use the file created at a XEN host.
Note 1: A small virtual machine was created inside XEN host and was started before executing the commands shown below.
Note 2: Notice that virbr0 (activated through libvirtd) is identical on both KVM and XEN hosts. And since we are here to discuss libvirtd related problems, it is ok to use example from either KVM host or XEN host.
[root@xenhost ~]# iptables-save > /root/iptables-vm-running.txt [root@xenhost ~]# cat /root/iptables-vm-running.txt *nat :PREROUTING ACCEPT [34:2932] :POSTROUTING ACCEPT [18:1292] :OUTPUT ACCEPT [18:1292] -A POSTROUTING -s 192.168.122.0/255.255.255.0 -d ! 192.168.122.0/255.255.255.0 -j MASQUERADE COMMIT *filter :INPUT ACCEPT [549:40513] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [364:41916] -A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT -A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT -A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT -A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT -A FORWARD -d 192.168.122.0/255.255.255.0 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT -A FORWARD -s 192.168.122.0/255.255.255.0 -i virbr0 -j ACCEPT -A FORWARD -i virbr0 -o virbr0 -j ACCEPT -A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable -A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable -A FORWARD -m physdev --physdev-in vif1.0 -j ACCEPT COMMIT
- Lines starting with * (asterisk) are the iptables "tables". Such as "nat" table, and "filter" table, as shown in the output above. Each table has "chains" defined in them.
- Lines starting with : (colon) are the name of chains. These lines contain following four pieces of information about a chain. ":ChainName POLICY [Packets:Bytes]"
- The name of the chain right next to the starting colon. e.g. "INPUT" , or "PREROUTING"
- The default policy of the chain. e.g. ACCEPT or DROP or REJECT. 99% of time, you will see ACCEPT here.
- The two values inside the square brackets are number of packets passed through this chain, so far, as well as the number of bytes. e.g. [364:41916] means 364 packets "or" 41916 bytes, have passed through this chain till this point in time.
- The lines starting with a - (hyphen/minus sign) are actually the actual rules which you put in here. "-A" would mean "Add" the rule. "-I" (eye) would mean "Insert" the rule.
As you notice, these rules are no different than the standard rules you type on the command line, or in the bash shell script. The limitation of this style of writing rules (as shown here) is, that you cannot use loops and conditions, as you would normally do in a bash/shell script.
More details on these rules
For this explanation the following ASCII diagram should be helpful.
+--[VM1]
|
[LAN 192.168.1.0/24]--(eth0:192.168.1.201)[PhysicalHost:DNS+DHCP+NAT](virbr0:192.168.122.1)--[VM2]
|
+--[VM3]
Continuing with the same example, we move to explain the iptables rules, created by the libvirtd daemon. I have put line numbers myself, in the beginning of each line, in the output below. (You can use cat -n file, or cat -b file, to include line numbers).
[root@xenhost ~]# cat /root/iptables-vm-running.txt 01) *nat 02) :PREROUTING ACCEPT [34:2932] 03) :POSTROUTING ACCEPT [18:1292] 04) :OUTPUT ACCEPT [18:1292] 05) -A POSTROUTING -s 192.168.122.0/255.255.255.0 -d ! 192.168.122.0/255.255.255.0 -j MASQUERADE 06) COMMIT 07) *filter 08) :INPUT ACCEPT [549:40513] 09) :FORWARD ACCEPT [0:0] 10) :OUTPUT ACCEPT [364:41916] 11) -A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT 12) -A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT 13) -A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT 14) -A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT 15) -A FORWARD -d 192.168.122.0/255.255.255.0 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPT 16) -A FORWARD -s 192.168.122.0/255.255.255.0 -i virbr0 -j ACCEPT 17) -A FORWARD -i virbr0 -o virbr0 -j ACCEPT 18) -A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable 19) -A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable 20) -A FORWARD -m physdev --physdev-in vif1.0 -j ACCEPT 21) COMMIT
What's happening here is the following:
- Line 05 defines a rule in the POSTROUTING chain in the "nat" table. It says that any traffic originating from 192.168.122.0/255.255.255.0 network, and trying to reach any other network but itself (-d ! 192.168.122.0/255.255.255.0) should be MASQUERADEd. In simple words, if there is any traffic coming from a VM (because only a VM can be on this 192.168.122.0/24 network), trying to go out from the physical LAN interface (eth0) of the physical host , must be masqueraded. This makes sense, as we don't know what will be the IP of the physical interface of the physical host. This rule facilitates the traffic to go out.
- Line 06 and 21: The word COMMIT indicates the end of list of rules for a particular particualr iptables "table".
- Lines 11 to 14: Virtual machines need a mechanism to get IP automatically and name resolution. DNSMASQ is the small service running on the phyical host, serving both DNS and DHCP requests coming in from the VMs. The DNSMASQ service on this physical host is not reachable over the physical LAN by any other physical host. Neither do it interferes with any DHCP service running elsewhere on the physical LAN. Lines 11 to 14 allow this incoming traffic from the VMs to arrive on the virbr0 interface of the physical host.
- Line 15: In case some traffic originated from a VM, and went out on the internet (because of line05), it will now want to reach back to the VM. For example, on the VM, you tried to pull an httpd.rpm file from a CENTOS mirror. So an HTTP request was originated from this VM, reached the CENTOS mirror (on the internet) and now the packets related to this transaction needs to be allowed back in so they can reach the VM. This requires a rule in the FORWARD chain, shown in line 15, which says that any traffic going to any VM on 192.168.122.0/255.255.255.0 network, exiting through/towards the virbr0 interface of the physical host, and is RELATED to some previous traffic "originated" from that particular VM, must be allowed. Without this, the return traffic/packets will never reach back the VM, and you will get all sorts of weird time-outs.
- Line 16: Basically if this rule does not exist, then the rule defined in line 5 will never get the traffic to MASQUERADE. Means, this line (16) says that any traffic originating from VMs on 192.168.122.0/255.255.255.0 , arriving at virbr0 interface of the physical host, trying to go across the FORWARD chain (to go elsewhere), must be ACCEPTed. Basically this is the line/rule, which transports a packet from virbr0 to eth0 (in this single direction only).
- Line 17: This is simply a multi-direction facilitator for all VMs connected to one virtual network. For example, VM1 and VM3 want to exchange some data on TCP/UDP/ICMP, their traffic would naturally traverse the virtual switch, virbr0. This line/rule allows that traffic to be ACCEPTed.
So far, we are able to send the traffic out from the VM and receive it back as well. Now what?
- Line 18: This line is read as : Any traffic coming from any source address, and going out to outgoing interface virbr0, will be rejected with a ICMP "port unreachable" message. Since we have already dealt with traffic coming in from virbr0, or any of the other VMs on the same machine, in the iptables rule before #18, this rule is for the traffic coming from "outside network" / "physical LAN". In other words, any traffic which could not satisfy the rules so far, and still interested in reaching the VMs going towards the virbr0 interface, is REJECTed. For example, you start simple web service on port 80 on VM1. If you somehow setup a port to be forwarded from the physical interface of your physical host, to this port 80 on the VM, using DNAT, and try to reach that port, you access will be REJECTed because of this rule!
- Line 19: Any traffic coming in from any VM, through virbr0 interface and trying to go out from the physical interface of the physical host (eth0) will be REJECTed. For example, in line /rule 16 we saw that any traffic coming in from a VM on 192.168.122.0/24 network, and trying to go out the physical interface of the physical machine is allowed. True, but please note, that is only allowed, when the source IP is from the 192.168.122.0/24 network! If there is a VM on the same virtual switch/bridge (virbr0), but with different IP, (say 10.1.1.1) , and it tries to go through the physical host towards the other side, it would be denied access. This is the last iptables rule as defined by libvirt. The next rule is from XEN.
- Line 20: This is interesting. First, notice that this rule is not part of the default libvirt ruleset. Instead, when a VM on XEN physical host was powered on, this rule got added to the ruleset. (Doesn't appear on KVM host). When the virtual machine is shutdown, this line gets removed. The code controlling this behavior resides in /etc/xen/scripts/vif-common.sh file.
- "PHYSDEV" is a special match module, made available in 2.6 kernels. It is used to match the bridge's physical in and out ports. Its basic usage is simple. e.g. " iptables -m physdev --physdev-in <bridge-port> -j <TARGET>" . It is used in situations where an interface may, or may not, (or may never), have an IP address. Check the iptables man page for more detail on "physdev".
- In XEN, for each virtual machine, there is a vifX.Y. A so called / virtual patch cable runs from the virtual machine to this port on the bridge. In the (XEN) example we are following, VM1 (aka domain-1, or domain with id #1), is connected to the virtual bridge on the physical host, on vif1.0 port. Here is the rule for reference.
20) -A FORWARD -m physdev --physdev-in vif1.0 -j ACCEPT
- Since these iptables rules (and rule #20 in particular) are added at the XEN (physical) host, we will read and understand the physdev-in and physdev-out, taking XEN host as a reference. Thus, physical-device-in (--physdev-in vif1.0) means that traffic coming in (received) from vif1.0 port on the bridge, towards the XEN host.
- The script /etc/xen/scripts/vif-common.sh has facility to add this rule either at the end of the current ruleset of the physical host, or, at the top of the rule-set of the physical host. This is do-able by changing the "-A" to "-I" in this script file. Here is the part of the script for reference:
function frob_iptable()
{
if [ "$command" == "online" ]
then
local c="-A"
else
local c="-D"
fi
. . .
. . .
Note: The purpose of this rule (line #20) seems to be: to make sure that whatever your previous firewall rules on the physical host, (assuming they are "sane"), the VM can always send traffic to/through the physical host. However, during my testing, this rule doesn't seem to work, with any traffic coming in from vif1.0, or going out to vif1.0. During some tests, it was made sure that the PHYSDEV rule was at the top of FORWARD chain (using the c="-I"), instead of being added to the end of the rules, to ensure any possible match. However, I could not get any traffic to get registered on this rule. In other words, no traffic matched this rule. And the counters at this rule always remained at zero.
[root@xenhost ~]# iptables -L -v
. . .
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- any any anywhere anywhere PHYSDEV match --physdev-in vif1.0
25 30517 ACCEPT all -- any virbr0 anywhere 192.168.122.0/24 state RELATED,ESTABLISHED
26 1487 ACCEPT all -- virbr0 any 192.168.122.0/24 anywhere
0 0 ACCEPT all -- virbr0 virbr0 anywhere anywhere
1 60 REJECT all -- any virbr0 anywhere anywhere reject-with icmp-port-unreachable
0 0 REJECT all -- virbr0 any anywhere anywhere reject-with icmp-port-unreachable
. . .
[root@xenhost ~]#
This leads us to another possibility. XEN may be connecting the vif1.0 interface to the xenbr0, and since our VM (vm1), is connected to virbr0, instead of xenbr0, this rule is not seeing any traffic.
The /etc/xen/xend-config.sxp specifies a directive (network-script network-bridge), which means that XEN will set up network-bridge. The other options are network-nat and network-route. network-bridge directive expects a bridge (xenbr0), sharing the network device (peth0). Thus, XEN's PHYSDEV match rule, in the situation, where libvirt is providing the network-nat service (not XEN), may be totally useless/un-necessary.
Therefore, we will not bother if XEN adds the PHYSDEV match at the beginning of rules, or at the end. We will leave it to default behaviour, i.e. added at the end of rules-set.
We will see in other tests, by creating a separate VM and connecting it to xenbr0, and then check, if this particular rule gets any traffic or not.
The interesting stuff
Probably the most interesting stuff is none of these rules are required in the first place, if you just want to run a VM in the private network (virbr0).!!!! . That is right. If you flush all sorts of rules from your physical host, (assuming all chains have default policy set to ACCEPT), the VMs will still be able to communicate to the physical host and vice-versa.
Surprised!!! You might be thinking, "why so many rules in the first place?" . The answer is multi-part:
- These rules (added by libvirtd), become important, when the default FORWARD policy is set to DROP. In that case, it is necessary to make sure that the traffic between the VMs and the physical host does not get blocked/dropped.
- Both libvirtd and XEN try to be smart. Libvirtd simply removes all your rules and implements it's default rules. Whereas XEN does not remove your iptables rules, and simply adds the (physdev) rule at the end (or beginning) of current set of rules, for each virtual machine. It is however observed that the PHYSDEV match is never made and thus no packet is seen traversing it.
- These rules are needed, because both libvirtd and XEN assumes there are some rules already on the physical host. Both of them also assume that any existing rules may be restrictive in nature, and may block the DNS, DHCP and other normal traffic requests coming from the VMs, on the FORWARD chain in particular. So both try to make sure that the essential traffic between a VM and the physical host, and between two VMs on the same physical host always gets sent/received properly. That is why both libvirtd and XEN adds the iptables rules.
- Further extension to the previous point, the designers of XEN assumed two situations, especially for the FORWARD chain.
- 1) They assumes that the FORWARD chain on any physical host, will be configured with default chain policy as DROP and ACCEPTing only specific rules. So, the "-A" mechanism in the /etc/xen/scripts/vif-common.sh script adds a ACCEPT rule for the bridge port of each new VM, which is started/powered-on.
- 2) If it is not the case explained in (1), then there might be a possibility that the default chain policy is ACCEPT, having certain ACCEPT rules in the chain, and then DROPing or REJECTing all other traffic at the bottom of the chain. In such a case, XEN adding the rule at the bottom of the FORWARD ruleset will be useless. In that case, we change "-A" to "-I" in the vif-common.sh script. In that way, the bridge interface of any VM starting up by XEN, will always be allowed to send/receive traffic.
References
- https://bugzilla.redhat.com/show_bug.cgi?id=227011
- http://lists.fedoraproject.org/pipermail/virt/2010-January/001792.html
- http://forums.gentoo.org/viewtopic-p-6209192.html?sid=1089acac70de96d68aa856d758d7cdfe
- http://wiki.libvirt.org/page/Networking
- http://libvirt.org/formatnetwork.html#examplesPrivate
Other readings:
