[OpenWrt-Devel] [PATCH 3/3] IP Virtual Server - Load Balancer - OpenWrt configuration
Mauro Mozzarelli
mauro at ezplanet.net
Wed Feb 24 13:35:42 EST 2016
All,
Part 3/3 includes /etc/init.d/ipvsadm script, uci configuration and cron job to check for real servers' availability and
re-configure the ipvs tables accordingly (using ipvsadm).
This patch depends on the ipvs kernel modules patch 1/3 and ipvsadm tool 2/3
This last patch is not necessary like the previous 2 to add load balancing capability to OpenWrt. One could write his own
scripts and configuration files, however I thought it would be helpful to provide an OpenWrt standard configuration using uci.
This can be further extended to support more or all ipvs options.
I tested the configuration file and scripts on a router with a static subnet on the wan side. I run both internet facing and
intranet facing load balanced servers and it works perfectly. It does not include NAT Masquerade configuration. This can be
easily added, however since personally I do not use ipvs NAT (and I would not advise to do so because the real servers would
not be able to track the original client's source IP) I have not implemented it.
Should you find issues please feel free just not to push this last patch into the mainstream trunk.
However I recommend to push the code as is because
1) it provides finely working Load Balancing Capability and
2) it would be an incentive for further improvements and eventually the implementation of luci configuration.
Mauro
commit 56e49cc8ef439ff7e1c225c3902654a4716e1a3f
Author: Mauro Mozzarelli <mauro at ezplanet.net>
Date: Wed Feb 24 18:18:25 2016 +0000
This ipvs package adds Linux IP Virtual Server capability to OpenWrt
IPVS (IP Virtual Server) implements transport-layer load balancing inside the Linux kernel, so called Layer-4 switching.
IPVS running on a host acts as a load balancer at the front of a cluster of real servers, it can direct requests for
TCP/UDP based services to the real servers, and makes services of the real servers to appear as a virtual service on a
single IP address.
This ipvs package contains:
- uci ipvs sample configuration file
- ipvsadm service startup/shutdown script
- scheduler checking real servers' availability and re-configuring ipvs tables accordingly
Dependencies:
- ipvs kernel modules
- ipvsadm tool
Signed-off-by: Mauro Mozzarelli <mauro at ezplanet.net>
diff --git a/package/network/utils/ipvsadm/Makefile b/package/network/utils/ipvsadm/Makefile
index 6a38867..6056da7 100644
--- a/package/network/utils/ipvsadm/Makefile
+++ b/package/network/utils/ipvsadm/Makefile
@@ -43,6 +43,7 @@ define Package/ipvsadm/install
$(INSTALL_BIN) $(PKG_BUILD_DIR)/ipvsadm $(1)/sbin/
$(INSTALL_BIN) $(PKG_BUILD_DIR)/ipvsadm-save $(1)/sbin/
$(INSTALL_BIN) $(PKG_BUILD_DIR)/ipvsadm-restore $(1)/sbin/
+ $(CP) ./files/* $(1)/
endef
$(eval $(call BuildPackage,$(PKG_NAME)))
diff --git a/package/network/utils/ipvsadm/files/etc/config/ipvs b/package/network/utils/ipvsadm/files/etc/config/ipvs
new file mode 100644
index 0000000..9d6be66
--- /dev/null
+++ b/package/network/utils/ipvsadm/files/etc/config/ipvs
@@ -0,0 +1,44 @@
+# Sample Configuration
+# please see the documentation at www.linuxvirtualserver.org to learn how to
+# configure real servers.
+# This configuration is used by /etc/init.d/ipvsadm to setup and load balance
+# virtual servers to available real servers.
+# A cron job is used to activate/deactivate real servers based on their availability (ping)
+#
+# Currently this configuration defaults to persistent gateway mode
+# Currently NAT masquerading and other options provided by ipvsadm are not implemented by this configuration
+# If virtual and real servers have internet static IPs, the wan interface must be part of a static IP Subnet
+# ipvsadm is fully functional for manual configuration
+
+config vipvs globals
+ option enabled 0
+# role master|slave
+ option role 'master'
+ option interface 'lan'
+# valid schedulers are included in the kernel schedulers list
+ list scheduler wlc
+
+# for each vip an alias ip is initialized on the wan/lan router interface
+# the vip must not be the same as the router's ip
+config virtual
+ option enabled 0
+ option name admin
+ option vip '192.168.10.10'
+ option interface lan
+ option alias 10
+ option scheduler 'wlc'
+ list real '192.168.10.100'
+ list real '192.168.10.101'
+
+# lists protocols/ports
+config admin
+ option protocol tcp
+ option src_port 80
+ option dest_port 80
+
+config admin
+ option protocol udp
+# if only 'port' is specified then source and destination ports are the same
+# port 0 means all ports are forwarded to the real servers
+ option port 0
+
diff --git a/package/network/utils/ipvsadm/files/etc/init.d/ipvsadm b/package/network/utils/ipvsadm/files/etc/init.d/ipvsadm
new file mode 100755
index 0000000..506dd32
--- /dev/null
+++ b/package/network/utils/ipvsadm/files/etc/init.d/ipvsadm
@@ -0,0 +1,190 @@
+#!/bin/sh /etc/rc.common
+
+################################################################################
+# Author: Mauro Mozzarelli
+# Description: builds ipvs configuration from uci openwrt parameters
+# load/unload ip_vs kernel modules
+# start/stop ipvs scheduler
+# start/stop cron job scheduler
+# Dependencies: /etc/config/ipvs; ipvsadm; ip_vs Kernel modules; cron
+# Notes: Firewall must be configured separately
+# Configuration files in /tmp/ipvsadm.d:
+# virt.* = virtual servers
+# *.stop = real server in down state ipvs table
+# *.start = real server in up state ipvs table
+# vserv.* = virtual servers ipvs table
+################################################################################
+
+START=70
+STOP=10
+
+USE_PROCD=1
+
+BASE_DIR=/tmp/ipvsadm.d
+TMP_DIR=$BASE_DIR
+REAL_SERVERS=realServers
+KERNEL_RELEASE=`uname -r`
+MODULES=/lib/modules/$KERNEL_RELEASE
+IPVSMOD=ip_vs
+# Cron job used to check if real servers are available and update ipvs
+# scheduler accordingly
+CRON_SCHEDULER='* * * * * /usr/sbin/checkRealServers # IPVS_SCHEDULER'
+
+# Check if real server is in the list and add if not present
+update_real_server () {
+ realServer=$1
+ gotit=0
+ for s in `cat $BASE_DIR/$REAL_SERVERS`; do
+ if [ $r == $s ]; then
+ gotit=1
+ break
+ fi
+ done
+ if [ $gotit -eq 0 ]; then
+ echo $r >> $BASE_DIR/$REAL_SERVERS
+ fi
+}
+
+start_service () {
+ enabled=`uci -q get ipvs.globals.enabled`
+ case $enabled in
+ 0)
+ exit
+ ;;
+ 1)
+ ;;
+ *)
+ echo Invalid initialization parameter: enabled=$enabled
+ exit
+ ;;
+ esac
+ if [ -d $BASE_DIR ]; then
+ cd $BASE_DIR
+ else
+ mkdir -p $BASE_DIR
+ cd $BASE_DIR
+ fi
+ rm -f $BASE_DIR/*
+ rm -f $TMP_DIR/*.down
+ touch $BASE_DIR/$REAL_SERVERS
+ modprobe $IPVSMOD
+ cm=0
+ for m in `uci -q get ipvs.globals.scheduler`; do
+ cm=$((cm+1))
+ modprobe ${IPVSMOD}_$m
+ done
+ # Default to wlc if no schedulers
+ if [ $cm -eq 0 ]; then
+ modprobe ${IPVSMOD}_wlc
+ fi
+ cv=0
+ while [ x`uci -q get ipvs. at virtual[$cv]` == x'virtual' ]; do
+ enabled=`uci get ipvs. at virtual[$cv].enabled`
+ if [ $enabled -eq 0 ]; then
+ cv=$((cv+1))
+ continue
+ fi
+ name=`uci get ipvs. at virtual[$cv].name`
+ vip=`uci get ipvs. at virtual[$cv].vip`
+ interface=`uci get ipvs. at virtual[$cv].interface`
+ device=`uci -P /var/state get network.${interface}.ifname`
+ alias=`uci get ipvs. at virtual[$cv].alias`
+ scheduler=`uci get ipvs. at virtual[$cv].scheduler`
+ # ifconfig
+ ifconfig $device:$alias $vip netmask 255.255.255.255 up
+ #
+ real=`uci get ipvs. at virtual[$cv].real`
+ if [ ! -z "$real" -a "$real" != " " ]; then
+ vip_done=no
+ for r in $real; do
+ update_real_server $r
+ cd=0
+ while [ x`uci -q get ipvs.@$name[$cd]` == x$name ]; do
+ protocol=`uci -q get ipvs.@$name[$cd].protocol`
+ src_port=`uci -q get ipvs.@$name[$cd].src_port`
+ dest_port=`uci -q get ipvs.@$name[$cd].dest_port`
+ case $protocol in
+ tcp)
+ protocol="-t"
+ ;;
+ udp)
+ protocol="-u"
+ ;;
+ *)
+ # Default to tcp protocol
+ protocol="-t"
+ ;;
+ esac
+ if [ -z $src_port ]; then
+ port=`uci -q get ipvs.@$name[$cd].port`
+ if [ -z $port ]; then
+ port=0
+ fi
+ src_port=$port
+ fi
+ if [ -z $dest_port ]; then
+ dest_port=$src_port
+ fi
+ if [ x$vip_done == x'no' ];then
+ echo "-A $protocol $vip:$src_port -s $scheduler -p" >> $BASE_DIR/vserv.start.$interface
+ echo "-D $protocol $vip:$src_port" >> $BASE_DIR/vserv.stop.$interface
+ fi
+ echo "-a $protocol $vip:$src_port -r $r:$dest_port -g -w 1" >> $BASE_DIR/$r.start
+ echo "-d $protocol $vip:$src_port -r $r" >> $BASE_DIR/$r.stop
+ cd=$((cd+1))
+ done
+ vip_done=yes
+ done
+ fi
+ cv=$((cv+1))
+ done
+ for i in `cat $BASE_DIR/$REAL_SERVERS`; do
+ date > $TMP_DIR/$i.down
+ done
+ for i in ${BASE_DIR}/vserv.start.* ; do
+ ipvsadm -R < $i
+ done
+
+ grole=`uci get ipvs.globals.role`
+ gintf=`uci get ipvs.globals.interface`
+ ipvsadm --start-daemon $grole --mcast-interface `uci -P /var/state get network.$gintf.ifname`
+ if [ $? -gt 0 ]; then
+ logger -p info "ERROR: IPVSADM daemon failed to start"
+ exit
+ fi
+ logger -p info "IPVSADM daemon started"
+
+ # Clean up any left over crontab entry
+ # Add real server check scheduler to cron
+ (crontab -l|grep -v IPVS_SCHEDULER ; echo "$CRON_SCHEDULER")|crontab -
+}
+
+stop_service () {
+ for i in ${BASE_DIR}/*.stop ; do
+ ipvsadm -R < $i
+ done
+ for i in ${BASE_DIR}/vserv.stop.* ; do
+ ipvsadm -R < $i
+ done
+ n=0
+ while [ x`uci -q get ipvs. at virtual[$n]` == x'virtual' ]; do
+ interface=`uci get ipvs. at virtual[$n].interface`
+ device=`uci -P /var/state get network.${interface}.ifname`
+ alias=`uci get ipvs. at virtual[$n].alias`
+ ifconfig $device:$alias down
+ n=$((n+1))
+ done
+ ipvsadm --stop-daemon `uci get ipvs.globals.role`
+ rm -f $TMP_DIR/*.down
+ rm -rf $BASE_DIR
+ for m in `uci -q get ipvs.globals.scheduler`; do
+ cm=$((cm+1))
+ rmmod ${IPVSMOD}_$m
+ done
+ # Default to wlc if no schedulers
+ if [ $m -eq 0 ]; then
+ rmmod ${IPVSMOD}_wlc
+ fi
+ rmmod $IPVSMOD
+ crontab -l |grep -v IPVS_SCHEDULER|crontab -
+}
diff --git a/package/network/utils/ipvsadm/files/usr/sbin/checkRealServers
b/package/network/utils/ipvsadm/files/usr/sbin/checkRealServers
new file mode 100755
index 0000000..f801c4b
--- /dev/null
+++ b/package/network/utils/ipvsadm/files/usr/sbin/checkRealServers
@@ -0,0 +1,36 @@
+#!/bin/sh
+################################################################################
+# Author: Mauro Mozzarelli
+# Description: checks whether a real server is up and running and if it is
+# then it starts the services
+# Dependencies: /tmp/ipvsadm.d
+# configuration files created by /etc/init.d/ipvsadm
+# virt.* = virtual servers
+# *.stop = real server in down state ipvs table
+# *.start = real server in up state ipvs table
+# *.down = real server down status
+################################################################################
+
+BASE_DIR=/tmp/ipvsadm.d
+TMP_DIR=$BASE_DIR
+REAL_SERVERS=realServers
+
+if [ -f $BASE_DIR/$REAL_SERVERS ]; then
+ for i in `cat $BASE_DIR/$REAL_SERVERS`; do
+ ping -qc 3 $i > /dev/null
+ RESULT=$?
+ if [ $RESULT -gt 0 ]; then
+ if [ ! -f $TMP_DIR/$i.down ]; then
+ ipvsadm -R < $BASE_DIR/$i.stop
+ date > $TMP_DIR/$i.down
+ logger -p info "IPVS Server $i is down"
+ fi
+ else
+ if [ -f $TMP_DIR/$i.down ]; then
+ ipvsadm -R < $BASE_DIR/$i.start
+ rm $TMP_DIR/$i.down
+ logger -p info "IPVS Server $i is on-line"
+ fi
+ fi
+ done
+fi
_______________________________________________
openwrt-devel mailing list
openwrt-devel at lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel
More information about the openwrt-devel
mailing list