Libvirt overwrites the existing iptables rules - Revision history

Kamran: Redirected page to XEN, KVM, Libvirt and IPTables

Kamran — Sun, 20 Feb 2011 06:58:59 GMT

Redirected page to XEN, KVM, Libvirt and IPTables

← Older revision		Revision as of 06:58, 20 February 2011
Line 1:		Line 1:
		+	#REDIRECT [[XEN, KVM, Libvirt and IPTables]]

	<b> DISCLAIMER: This document may contain text, which is horribly wrong. This page is just my scratch pad for my personal notes, on this beast. So until I get it right, you are advised not to read anything in this document. It may shatter your existing iptables/virtualization concepts. READ AND/OR USE AT YOUR OWN RISK.</b>		<b> DISCLAIMER: This document may contain text, which is horribly wrong. This page is just my scratch pad for my personal notes, on this beast. So until I get it right, you are advised not to read anything in this document. It may shatter your existing iptables/virtualization concepts. READ AND/OR USE AT YOUR OWN RISK.</b>

Kamran: /* References */

Kamran — Sat, 19 Feb 2011 08:25:56 GMT

References

← Older revision		Revision as of 08:25, 19 February 2011
Line 1,329:		Line 1,329:
	* http://libvirt.org/formatnetwork.html#examplesPrivate		* http://libvirt.org/formatnetwork.html#examplesPrivate
	* http://ddevnet.net/wiki/index.php/How_KVM_or_libVirt_IPtables_work		* http://ddevnet.net/wiki/index.php/How_KVM_or_libVirt_IPtables_work
-
-	~~Other readings:~~
	* http://www.redhat.com/archives/libvir-list/2007-April/msg00033.html		* http://www.redhat.com/archives/libvir-list/2007-April/msg00033.html
	* http://people.gnome.org/~markmc/virtual-networking.html		* http://people.gnome.org/~markmc/virtual-networking.html
	* http://www.standingonthebrink.com/index.php/ipv6-ipv4-and-arp-on-xen-for-vps/		* http://www.standingonthebrink.com/index.php/ipv6-ipv4-and-arp-on-xen-for-vps/
	* http://www.gossamer-threads.com/lists/xen/users/175478		* http://www.gossamer-threads.com/lists/xen/users/175478
-
-
	* http://xen.1045712.n5.nabble.com/PATCH-vif-common-sh-prevent-physdev-match-using-physdev-out-in-the-OUTPUT-FORWARD-and-POSTROUTING-che-td3255945.html		* http://xen.1045712.n5.nabble.com/PATCH-vif-common-sh-prevent-physdev-match-using-physdev-out-in-the-OUTPUT-FORWARD-and-POSTROUTING-che-td3255945.html
	* https://bugzilla.redhat.com/show_bug.cgi?id=512206		* https://bugzilla.redhat.com/show_bug.cgi?id=512206

Kamran: /* The simplest iptables rule-set for your XEN/KVM host */

Kamran — Sat, 19 Feb 2011 07:23:22 GMT

The simplest iptables rule-set for your XEN/KVM host

← Older revision		Revision as of 07:23, 19 February 2011
Line 1,291:		Line 1,291:

	# DROP ICMP packets size larger than 56(84) bytes. This should come before other ICMP rules.		# DROP ICMP packets size larger than 56(84) bytes. This should come before other ICMP rules.
-	${IPTABLES} -A INPUT -p icmp --icmp-type echo-request -m length --length 85: -j ~~REJECT~~	+	${IPTABLES} -A INPUT -p icmp --icmp-type echo-request -m length --length 85: -j DROP

	# Allow maximum two incoming ICMP echo-request packets per second		# Allow maximum two incoming ICMP echo-request packets per second

Kamran: /* The simplest iptables rule-set for your XEN/KVM host */

Kamran — Sat, 19 Feb 2011 06:56:14 GMT

The simplest iptables rule-set for your XEN/KVM host

← Older revision		Revision as of 06:56, 19 February 2011
Line 1,290:		Line 1,290:
	# to protect your physical host (XEN/KVM)		# to protect your physical host (XEN/KVM)

-	# DROP ICMP packets size larger than 56(84) bytes	+	# DROP ICMP packets size larger than 56(84) bytes. This should come before other ICMP rules.
	${IPTABLES} -A INPUT -p icmp --icmp-type echo-request -m length --length 85: -j REJECT		${IPTABLES} -A INPUT -p icmp --icmp-type echo-request -m length --length 85: -j REJECT

-	# Allow maximum two incoming ICMP packets per second	+	# Allow maximum two incoming ICMP echo-request packets per second
	${IPTABLES} -A INPUT -p icmp --icmp-type echo-request -m limit --limit 1/s -j ACCEPT		${IPTABLES} -A INPUT -p icmp --icmp-type echo-request -m limit --limit 1/s -j ACCEPT
		+	${IPTABLES} -A INPUT -p icmp --icmp-type echo-reply -m limit --limit 1/s -j ACCEPT
		+
		+	# It is important to drop all other incoming ICMP traffic.
		+	${IPTABLES} -A INPUT -p icmp -m icmp --icmp-type any -j DROP

	exit 0		exit 0

Kamran: /* References */

Kamran — Wed, 16 Feb 2011 17:04:07 GMT

References

← Older revision		Revision as of 17:04, 16 February 2011
Line 1,324:		Line 1,324:
	* http://wiki.libvirt.org/page/Networking		* http://wiki.libvirt.org/page/Networking
	* http://libvirt.org/formatnetwork.html#examplesPrivate		* http://libvirt.org/formatnetwork.html#examplesPrivate
		+	* http://ddevnet.net/wiki/index.php/How_KVM_or_libVirt_IPtables_work

	Other readings:		Other readings:

Kamran: /* More details on the rules file created by iptables-save command */

Kamran — Wed, 16 Feb 2011 16:54:05 GMT

More details on the rules file created by iptables-save command

← Older revision		Revision as of 16:54, 16 February 2011
Line 436:		Line 436:
	* Line 19: Any traffic coming in from any VM, through virbr0 interface and trying to go out from the physical interface of the physical host (eth0) will be REJECTed. For example, in line /rule 16 we saw that any traffic coming in from a VM on 192.168.122.0/24 network, and trying to go out the physical interface of the physical machine is allowed. True, but please note, that is only allowed, when the source IP is from the 192.168.122.0/24 network! If there is a VM on the same virtual switch/bridge (virbr0), but with different IP, (say 10.1.1.1) , and it tries to go through the physical host towards the other side, it would be denied access. This is the last iptables rule as defined by libvirt. The next rule is from XEN.		* Line 19: Any traffic coming in from any VM, through virbr0 interface and trying to go out from the physical interface of the physical host (eth0) will be REJECTed. For example, in line /rule 16 we saw that any traffic coming in from a VM on 192.168.122.0/24 network, and trying to go out the physical interface of the physical machine is allowed. True, but please note, that is only allowed, when the source IP is from the 192.168.122.0/24 network! If there is a VM on the same virtual switch/bridge (virbr0), but with different IP, (say 10.1.1.1) , and it tries to go through the physical host towards the other side, it would be denied access. This is the last iptables rule as defined by libvirt. The next rule is from XEN.

-	Note 1: Rules 18 and 19, shown here, act as a default <b>reject-all</b> or <b>drop-all</b> policy for the FORWARD chain.	+	<b>Note 1:</b> Rules 18 and 19, shown here, act as a default <b>reject-all</b> or <b>drop-all</b> policy for the FORWARD chain.

-	Note 2: The PHYSDEV match rule, has it's own explanation, purely related to XEN. Thus it is discussed separately in the next section.	+	<b>Note 2:</b> The PHYSDEV match rule, has it's own explanation, purely related to XEN. Thus it is discussed separately in the next section.
		+
		+	<b>Note: 3:</b> The rules, from sequence #15 to #20, (all FORWARD rules listed above), are totally un-necessary, "if" the policy of the FORWARD chain is set to ACCEPT, and there is no other rule restricting any traffic. You will still be able to communicate with all VMS, from outside to inside, inside to outside, and VM to VM. This is explained in the <i>Trivia</i> section at the end of this paper.

	====The PHYSDEV match rule====		====The PHYSDEV match rule====

Kamran: /* The simplest iptables rule-set for your XEN/KVM host */

Kamran — Wed, 16 Feb 2011 16:38:22 GMT

The simplest iptables rule-set for your XEN/KVM host

← Older revision		Revision as of 16:38, 16 February 2011
Line 1,281:		Line 1,281:
	${IPTABLES} -A INPUT -i ${PRIVATEBRIDGE} -p udp -m udp --dport 67 -j ACCEPT		${IPTABLES} -A INPUT -i ${PRIVATEBRIDGE} -p udp -m udp --dport 67 -j ACCEPT
	${IPTABLES} -A INPUT -i ${PRIVATEBRIDGE} -p tcp -m tcp --dport 67 -j ACCEPT		${IPTABLES} -A INPUT -i ${PRIVATEBRIDGE} -p tcp -m tcp --dport 67 -j ACCEPT
-		+	# Note: Add any other INPUT chain rules in Section 2, below.

	# Section 2: Physical host firewall		# Section 2: Physical host firewall

Kamran: /* The simplest iptables rule-set for your XEN/KVM host */

Kamran — Wed, 16 Feb 2011 16:37:18 GMT

The simplest iptables rule-set for your XEN/KVM host

← Older revision		Revision as of 16:37, 16 February 2011
Line 1,205:		Line 1,205:

	====The simplest iptables rule-set for your XEN/KVM host====		====The simplest iptables rule-set for your XEN/KVM host====
-	~~The script below~~ can ~~act as~~ your single ~~point~~ of ~~reference to enable host wide iptables rules~~. This can also work as a life saver, when you totally mess up your rule-set in /etc/sysconfig/iptables file, and, the only solution seem to you is restart the server, which you don't want to do! Running the following script, with certain adjustments as per your setup, will help you save your day.	+	You can use the following script to manage all your iptables needs on your XEN/KVM host, in a single location, instead of managing them in separate locations. This script can also work as a life saver, when you totally mess up your rule-set in /etc/sysconfig/iptables file; and the only solution seem to you is restart the server; which you don't want to do! Running the following script, with certain adjustments as per your setup, will help you save your day. Many people now-a-days use their laptops as a (XEN/KVM) host, to setup virtual machines for testing. This script should be safe to run on such laptops too. It is however assumed, that the laptop, if connected to any network, is connected through eth0. You should still exercise care and caution, while using this script.

	<pre>		<pre>
Line 1,213:		Line 1,213:
	# Created : 2011-02-16		# Created : 2011-02-16
	# Revised : 2011-02-16		# Revised : 2011-02-16
-	# Summary : Simplest possible rule-set for your iptables requirements, related to your virtualization host.	+	# Summary : Simplest possible rule-set for your iptables requirements,
		+	# related to your virtualization host.
	# Usage : Can be executed manually, from the command prompt.		# Usage : Can be executed manually, from the command prompt.
	# Can also be executed as a init script at start sequence number 99.		# Can also be executed as a init script at start sequence number 99.
	# chkconfig: 2345 99 02		# chkconfig: 2345 99 02
	# NOTICE : USE AT YOUR OWN RISK!		# NOTICE : USE AT YOUR OWN RISK!
-	# This script will erase all rules from your host, and set the default policies of all chains to ACCEPT.	+	# This script will erase all rules from your host,
-	~~################~~###################################################################################################	+	# and set the default policies of all chains to ACCEPT.
		+	###################################################################################################


Line 1,247:		Line 1,249:


-	# Enable traffic redirection for any ports you want to forward from your ${PUBLICIF} (normally eth0), to any of the private VM.	+	# Enable traffic redirection for any ports you want to forward from your ${PUBLICIF} (normally eth0),
		+	# to any of the private VM.
	# You will need to adjust the PREROUTING rules accordingly.		# You will need to adjust the PREROUTING rules accordingly.
	# If you do not want any traffic to come in from outside, to any of the VM inside this physical host,		# If you do not want any traffic to come in from outside, to any of the VM inside this physical host,
	# you can skip the PREROUTING configuration.		# you can skip the PREROUTING configuration.
-	# In the PREROUTING rules below, .41 and .42 are two different VMs, both running web service on port 80,	+	# In the PREROUTING rules below, .41 and .42 are two different VMs,
		+	# both running web service on port 80,
	# but reachable from outside over port 80 and 8080 respectively. Please adjust as required.		# but reachable from outside over port 80 and 8080 respectively. Please adjust as required.

Line 1,258:		Line 1,262:


-	# Enable MASQUERADE on POSTROUTING, for packets coming from VMs on private network, going outside the physical host.	+	# Enable MASQUERADE on POSTROUTING, for packets coming from VMs on private network,
		+	# going outside the physical host.
	${IPTABLES} -t nat -A POSTROUTING -s ${PRIVATENET} -d ! ${PRIVATENET} -j MASQUERADE		${IPTABLES} -t nat -A POSTROUTING -s ${PRIVATENET} -d ! ${PRIVATENET} -j MASQUERADE

Line 1,280:		Line 1,285:
	# Section 2: Physical host firewall		# Section 2: Physical host firewall
	#		#
-	# Insert any rules below, which you want to use on your INPUT chain, to protect your physical host (XEN/KVM)	+	# Insert any rules below, which you want to use on your INPUT chain,
		+	# to protect your physical host (XEN/KVM)

	# DROP ICMP packets size larger than 56(84) bytes		# DROP ICMP packets size larger than 56(84) bytes

Kamran: /* Scenario 2: The Rich-Man's Setup, and The Solution */

Kamran — Wed, 16 Feb 2011 16:27:21 GMT

Scenario 2: The Rich-Man's Setup, and The Solution

← Older revision		Revision as of 16:27, 16 February 2011
Line 1,030:		Line 1,030:
	# by the libvirtd rules, you will need to add them in the rc.local or a script of your choice,		# by the libvirtd rules, you will need to add them in the rc.local or a script of your choice,
	# which should run after xend.		# which should run after xend.
-	# -A POSTROUTING -s 192.168.122.41 -d ! 192.168.122.0/255.255.255.0 -j SNAT --to-source 192.168.1.~~201~~	+	# -A POSTROUTING -s 192.168.122.41 -d ! 192.168.122.0/255.255.255.0 -j SNAT --to-source 192.168.1.211
-	# -A POSTROUTING -s 192.168.122.42 -d ! 192.168.122.0/255.255.255.0 -j SNAT --to-source 192.168.1.~~211~~	+	# -A POSTROUTING -s 192.168.122.42 -d ! 192.168.122.0/255.255.255.0 -j SNAT --to-source 192.168.1.212
	COMMIT		COMMIT
	[root@xenhost ~]#		[root@xenhost ~]#
Line 1,044:		Line 1,044:
	In essence we have just kept our traffic redirector rules in the PREROUTING chain. And rest of the file has been emptied. When the system boots next time, iptables service will load these rules. Then, when the libvirt service runs, it will add it's default set of rules. The only rule left now is the one which move the traffic from outside, to the VMs across the FORWARD chain. We wish we could control that from the same /etc/sysconfig/iptables file. The solution is to either add this rules in /etc/rc.local file, or create a small simple rc script (a service file), and configure it to start right after libvirtd/xend service.		In essence we have just kept our traffic redirector rules in the PREROUTING chain. And rest of the file has been emptied. When the system boots next time, iptables service will load these rules. Then, when the libvirt service runs, it will add it's default set of rules. The only rule left now is the one which move the traffic from outside, to the VMs across the FORWARD chain. We wish we could control that from the same /etc/sysconfig/iptables file. The solution is to either add this rules in /etc/rc.local file, or create a small simple rc script (a service file), and configure it to start right after libvirtd/xend service.

-	The readers might be thinking that: <i>Can't we add all our custom rules to rc.local, or proposed new service file?</i> True. That is do-able. However, ~~we recommend against that. The~~ /etc/sysconfig/iptables file is still an excellent place to add more rules easily, in case there is a need to, especially for the INPUT chain. It is easy to add server protection rules to it (this file), instead of managing them at different places. Besides, our NEW rule is totally generic in nature, and you will probably never need to change it throughout the service lifetime of your physical host. Same is the case with optional SNAT rule. So they can be kept in either rc.local, or a small service script. We will show both methods below.	+	The readers might be thinking that: <i>Can't we add all our custom rules to rc.local, or proposed new service file?</i> True. That is do-able. However, the /etc/sysconfig/iptables file is still an excellent place to add more rules easily, in case there is a need to, especially for the INPUT chain. It is easy to add server protection rules to it (this file), instead of managing them at different places. Besides, our NEW rule is totally generic in nature, and you will probably never need to change it throughout the service lifetime of your physical host. Same is the case with optional SNAT rule. So they can be kept in either rc.local, or a small service script. We will show both methods below.

	: <b>Note:</b> A very simple version of complete rule-set is shown at the end of the <i>Solutions</i> section, just in case you want to totally eliminate any rules from libvirtd or the iptables services, and want to setup everything of your own, at only one location.		: <b>Note:</b> A very simple version of complete rule-set is shown at the end of the <i>Solutions</i> section, just in case you want to totally eliminate any rules from libvirtd or the iptables services, and want to setup everything of your own, at only one location.

Kamran: /* Scenario 1: The Poor-Man's Setup, and The Solution */

Kamran — Wed, 16 Feb 2011 16:26:05 GMT

Scenario 1: The Poor-Man's Setup, and The Solution

← Older revision		Revision as of 16:26, 16 February 2011
Line 695:		Line 695:
	</pre>		</pre>

-	The state (NEW) of the traffic is not a must to mention. We can have a simpler rule, without specifying the state, and it will still work. Also, since all the common services run on TCP, we have restricted the traffic type to TCP. Again, you can have a simpler rule, to not specify the traffic type in terms of TCP/UDP/ICMP. You can also remove the restriction of incoming interface. This way, in case you have two physical network cards, you will be allowing traffic from both network cards to reach the VMs.	+	The state (NEW) of the traffic is not a must to mention. We can have a simpler rule, without specifying the state, and it will still work. Also, since all the common services run on TCP, we have restricted the traffic type to TCP. Again, you can have a simpler rule, to not specify the traffic type in terms of TCP/UDP/ICMP. You can also remove the restriction of incoming interface. This way, in case you have two physical network cards, you will be allowing traffic from both network cards to reach the VMs. Below is the simpler version of the same rule discussed just now.
	<pre>		<pre>
	# iptables -I FORWARD -o virbr0 -j ACCEPT		# iptables -I FORWARD -o virbr0 -j ACCEPT
	</pre>		</pre>

-	After inserting the rule shown above, we try to access the VM from outside, and see the following ~~webpage~~, being pulled successfully:	+	After inserting the rule shown above, we try to access the VM from outside, and see the following web-page, being pulled successfully:
	<pre>		<pre>
	[kamran@kworkhorse tmp]$ wget http://192.168.1.201 -O index.html; cat index.html		[kamran@kworkhorse tmp]$ wget http://192.168.1.201 -O index.html; cat index.html
Line 809:		Line 809:
	# and the rule below will be pushed down, rendering it useless.		# and the rule below will be pushed down, rendering it useless.
	# You may want to add it to rc.local, or a script of your choice, which should run after xend is started.		# You may want to add it to rc.local, or a script of your choice, which should run after xend is started.
-	# -A POSTROUTING -s 192.168.122.41 -d ! 192.168.122.0/255.255.255.0 -j SNAT --to-source 192.168.1.201	+	# -A POSTROUTING -s 192.168.122.0/255.255.255.0 -d ! 192.168.122.0/255.255.255.0 -j SNAT --to-source 192.168.1.201
	COMMIT		COMMIT
	*filter		*filter
Line 821:		Line 821:
	As you would notice, in-essence we have just kept our traffic redirector rules in the PREROUTING chain. And rest of the file has been emptied. When the system boots next time, iptables service will load these rules. Then, when the libvirt service runs, it will add it's default set of rules. The only rule left now is the one which move the traffic from outside, to the VMs across the FORWARD chain. We wish we could control that from the same /etc/sysconfig/iptables file. The solution is to either add this rules in /etc/rc.local file, or create a small simple <i>rc</i> script (a service file), and configure it to start right after libvirtd/xend service.		As you would notice, in-essence we have just kept our traffic redirector rules in the PREROUTING chain. And rest of the file has been emptied. When the system boots next time, iptables service will load these rules. Then, when the libvirt service runs, it will add it's default set of rules. The only rule left now is the one which move the traffic from outside, to the VMs across the FORWARD chain. We wish we could control that from the same /etc/sysconfig/iptables file. The solution is to either add this rules in /etc/rc.local file, or create a small simple <i>rc</i> script (a service file), and configure it to start right after libvirtd/xend service.

-	The readers might be thinking that: <i>Can't we add all our custom rules to rc.local, or proposed new service file?</i> True. That is do-able. However, ~~we recommend against that. The~~ /etc/sysconfig/iptables file is still an excellent place to add more rules easily, in case there is a need to, especially for the INPUT chain. It is easy to add server protection rules to it (this file), instead of managing them at different places. Besides, our NEW rule is totally generic in nature, and you will probably never need to change it throughout the service lifetime of your physical host. Same is the case with optional SNAT rule. So they can be kept in either rc.local, or a small service script. We will show both methods below.	+	The readers might be thinking that: <i>Can't we add all our custom rules to rc.local, or proposed new service file?</i> True. That is do-able. However, the /etc/sysconfig/iptables file is still an excellent place to add more rules easily, in case there is a need to, especially for the INPUT chain. It is easy to add server protection rules to it (this file), instead of managing them at different places. Besides, our NEW rule is totally generic in nature, and you will probably never need to change it throughout the service lifetime of your physical host. Same is the case with optional SNAT rule. So they can be kept in either rc.local, or a small service script. We will show both methods below.

	<b>Note:</b> A very simple version of complete rule-set is shown at the end of the <i>Solutions</i> section, just in case you want to totally eliminate any rules from libvirtd or the iptables services, and want to setup everything of your own, at only one location.		<b>Note:</b> A very simple version of complete rule-set is shown at the end of the <i>Solutions</i> section, just in case you want to totally eliminate any rules from libvirtd or the iptables services, and want to setup everything of your own, at only one location.