[OpenWrt-Devel] [PATCH 3/3] IP Virtual Server - Load Balancer - OpenWRT configuration - IGNORE PREVIOUS
Mauro Mozzarelli
mauro at ezplanet.net
Tue Feb 23 15:19:26 EST 2016
All,
Please ignore the previous patch 3/3 as I have pasted 2/3 again by mistake.
This is the CORRECT PATCH 3/3
Part 3/3 includes /etc/init.d/ipvsadm script, uci configuration and cron job to check for real servers' availability and
re-configure the ipvs tables accordingly (using ipvsadm).
This patch depends on the ipvs kernel modules patch 1/3 and ipvsadm tool 2/3
This last patch is not necessary like the previous 2 to add load balancing capability to OpenWRT. One could write his own
scripts and configuration files, however I thought it would be helpful to provide an OpenWRT standard configuration using uci.
This can be further extended to support more or all ipvs options.
I tested the configuration file and scripts on a router with a static subnet on the wan side. I run both internet facing and
intranet facing load balanced servers and it works perfectly. It does not include NAT Masquerade configuration. This can be
easily added, however since personally I do not use ipvs NAT (and I would not advise to do so because the real servers would
not be able to track the original client's source IP) I have not implemented it.
Should you find issues with the scripts please feel free just not to push this last patch into the mainstream trunk.
Mauro
Author: Mauro Mozzarelli <mauro at ezplanet.net>
Date: Tue Feb 23 19:30:59 2016 +0000
This ipvs package adds Linux IP Virtual Server capability to OpenWRT
IPVS (IP Virtual Server) implements transport-layer load balancing inside the Linux kernel, so called Layer-4 switching.
IPVS running on a host acts as a load balancer at the front of a cluster of real servers, it can direct requests for
TCP/UDP based services to the real servers, and makes services of the real servers to appear as a virtual service on a
single IP address.
This ipvs package contains:
- uci ipvs sample configuration file
- ipvsadm service startup/shutdown script
- scheduler checking real servers' availability and re-configuring ipvs tables accordingly
Dependencies:
- ipvs kernel modules
- ipvsadm tool
Signed-off-by: Mauro Mozzarelli <mauro at ezplanet.net>
diff --git a/package/network/utils/ipvsadm/Makefile b/package/network/utils/ipvsadm/Makefile
index 8abde77..6ce06a1 100644
--- a/package/network/utils/ipvsadm/Makefile
+++ b/package/network/utils/ipvsadm/Makefile
@@ -54,6 +54,7 @@ define Package/ipvsadm/install
$(INSTALL_BIN) $(PKG_BUILD_DIR)/ipvsadm $(1)/usr/sbin/
$(INSTALL_BIN) $(PKG_BUILD_DIR)/ipvsadm-save $(1)/usr/sbin/
$(INSTALL_BIN) $(PKG_BUILD_DIR)/ipvsadm-restore $(1)/usr/sbin/
+ $(CP) ./files/* $(1)/
endef
$(eval $(call BuildPackage,$(PKG_NAME)))
diff --git a/package/network/utils/ipvsadm/files/etc/config/ipvs b/package/network/utils/ipvsadm/files/etc/config/ipvs
new file mode 100644
index 0000000..c392e39
--- /dev/null
+++ b/package/network/utils/ipvsadm/files/etc/config/ipvs
@@ -0,0 +1,44 @@
+# Sample Configuration
+# please see the documentation at www.linuxvirtualserver.org to learn how to
+# configure real servers.
+# This configuration is used by /etc/init.d/ipvsadm to setup and load balance
+# virtual servers to available real servers.
+# A cron job is used to activate/deactivate real servers based on their availability (ping)
+#
+# Currently this configuration defaults to persistent gateway mode
+# Currently NAT masquerading and other options provided by ipvsadm are not implemented by this configuration
+# If virtual and real servers have internet static IPs, the wan interface must be part of a static IP Subnet
+# ipvsadm is fully functional for manual configuration
+
+config ipvs globals
+ option enabled 0
+# role master|slave
+ option role 'master'
+ option interface 'lan'
+# valid schedulers are included in the kernel schedulers list
+ list scheduler wlc
+
+# for each vip an alias ip is initialized on the wan/lan router interface
+# the vip must not be the same as the router's ip
+config virtual
+ option enabled 0
+ option name admin
+ option vip '192.168.10.10'
+ option interface lan
+ option alias 10
+ option scheduler 'wlc'
+ list real '192.168.10.100'
+ list real '192.168.10.101'
+
+# lists protocols/ports
+config admin
+ option protocol tcp
+ option src_port 80
+ option dest_port 80
+
+config admin
+ option protocol udp
+# if only 'port' is specified then source and destination ports are the same
+# port 0 means all ports are forwarded to the real servers
+ option port 0
+
diff --git a/package/network/utils/ipvsadm/files/etc/init.d/ipvsadm b/package/network/utils/ipvsadm/files/etc/init.d/ipvsadm
new file mode 100755
index 0000000..7c8c378
--- /dev/null
+++ b/package/network/utils/ipvsadm/files/etc/init.d/ipvsadm
@@ -0,0 +1,187 @@
+#!/bin/sh /etc/rc.common
+
+################################################################################
+# Author: Mauro Mozzarelli
+# Description: builds ipvs configuration from uci openwrt parameters
+# load/unload ip_vs kernel modules
+# start/stop ipvs scheduler
+# start/stop cron job scheduler
+# Dependencies: /etc/config/ipvs; ipvsadm; ip_vs Kernel modules; cron
+# Notes: Firewall must be configured separately
+# Configuration files in /tmp/ipvsadm.d:
+# virt.* = virtual servers
+# *.stop = real server in down state ipvs table
+# *.start = real server in up state ipvs table
+# vserv.* = virtual servers ipvs table
+################################################################################
+
+START=70
+STOP=10
+
+USE_PROCD=1
+
+BASE_DIR=/tmp/ipvsadm.d
+TMP_DIR=$BASE_DIR
+REAL_SERVERS=realServers
+KERNEL_RELEASE=`uname -r`
+MODULES=/lib/modules/$KERNEL_RELEASE
+IPVSMOD=ip_vs
+# Cron job used to check if real servers are available and update ipvs
+# scheduler accordingly
+CRON_SCHEDULER='* * * * * /usr/sbin/checkRealServers # IPVS_SCHEDULER'
+
+# Check if real server is in the list and add if not present
+update_real_server () {
+ realServer=$1
+ gotit=0
+ for s in `cat $BASE_DIR/$REAL_SERVERS`; do
+ if [ $r == $s ]; then
+ gotit=1
+ break
+ fi
+ done
+ if [ $gotit -eq 0 ]; then
+ echo $r >> $BASE_DIR/$REAL_SERVERS
+ fi
+}
+
+start_service () {
+ enabled=`uci -q get ipvs.globals.enabled`
+ case $enabled in
+ 0)
+ exit
+ ;;
+ 1)
+ ;;
+ *)
+ echo Invalid initialization parameter: enabled=$enabled
+ exit
+ ;;
+ esac
+ if [ -d $BASE_DIR ]; then
+ cd $BASE_DIR
+ else
+ mkdir -p $BASE_DIR
+ cd $BASE_DIR
+ fi
+ rm -f $BASE_DIR/*
+ rm -f $TMP_DIR/*.down
+ touch $BASE_DIR/$REAL_SERVERS
+ modprobe $IPVSMOD
+ cm=0
+ for m in `uci -q get ipvs.globals.scheduler`; do
+ cm=$((cm+1))
+ modprobe ${IPVSMOD}_$m
+ done
+ # Default to wlc if no schedulers
+ if [ $cm -eq 0 ]; then
+ modprobe ${IPVSMOD}_wlc
+ fi
+ cv=0
+ while [ x`uci -q get ipvs. at virtual[$cv]` == x'virtual' ]; do
+ enabled=`uci get ipvs. at virtual[$cv].enabled`
+ if [ $enabled -eq 0 ]; then
+ cv=$((cv+1))
+ continue
+ fi
+ name=`uci get ipvs. at virtual[$cv].name`
+ vip=`uci get ipvs. at virtual[$cv].vip`
+ interface=`uci get ipvs. at virtual[$cv].interface`
+ device=`uci -P /var/state get network.${interface}.ifname`
+ alias=`uci get ipvs. at virtual[$cv].alias`
+ scheduler=`uci get ipvs. at virtual[$cv].scheduler`
+ # ifconfig
+ ifconfig $device:$alias $vip netmask 255.255.255.255 up
+ #
+ real=`uci get ipvs. at virtual[$cv].real`
+ if [ ! -z "$real" -a "$real" != " " ]; then
+ vip_done=no
+ for r in $real; do
+ update_real_server $r
+ cd=0
+ while [ x`uci -q get ipvs.@$name[$cd]` == x$name ]; do
+ protocol=`uci -q get ipvs.@$name[$cd].protocol`
+ src_port=`uci -q get ipvs.@$name[$cd].src_port`
+ dest_port=`uci -q get ipvs.@$name[$cd].dest_port`
+ case $protocol in
+ tcp)
+ protocol="-t"
+ ;;
+ udp)
+ protocol="-u"
+ ;;
+ *)
+ # Default to tcp protocol
+ protocol="-t"
+ ;;
+ esac
+ if [ -z $src_port ]; then
+ port=`uci -q get ipvs.@$name[$cd].port`
+ if [ -z $port ]; then
+ port=0
+ fi
+ src_port=$port
+ fi
+ if [ -z $dest_port ]; then
+ dest_port=$src_port
+ fi
+ if [ x$vip_done == x'no' ];then
+ echo "-A $protocol $vip:$src_port -s $scheduler -p" >> $BASE_DIR/vserv.start.$interface
+ echo "-D $protocol $vip:$src_port" >> $BASE_DIR/vserv.stop.$interface
+ fi
+ echo "-a $protocol $vip:$src_port -r $r:$dest_port -g -w 1" >> $BASE_DIR/$r.start
+ echo "-d $protocol $vip:$src_port -r $r" >> $BASE_DIR/$r.stop
+ cd=$((cd+1))
+ done
+ vip_done=yes
+ done
+ fi
+ cv=$((cv+1))
+ done
+ for i in `cat $BASE_DIR/$REAL_SERVERS`; do
+ date > $TMP_DIR/$i.down
+ done
+ for i in ${BASE_DIR}/vserv.start.* ; do
+ ipvsadm -R < $i
+ done
+
+ grole=`uci get ipvs.globals.role`
+ gintf=`uci get ipvs.globals.interface`
+ ipvsadm --start-daemon $grole --mcast-interface `uci -P /var/state get network.$gintf.ifname`
+ if [ $? -gt 0 ]; then
+ logger -p info "ERROR: IPVSADM daemon failed to start"
+ exit
+ fi
+ (crontab -l ; echo "$CRON_SCHEDULER")|crontab -
+ logger -p info "IPVSADM daemon started"
+}
+
+stop_service () {
+ for i in ${BASE_DIR}/*.stop ; do
+ ipvsadm -R < $i
+ done
+ for i in ${BASE_DIR}/vserv.stop.* ; do
+ ipvsadm -R < $i
+ done
+ n=0
+ while [ x`uci -q get ipvs. at virtual[$n]` == x'virtual' ]; do
+ interface=`uci get ipvs. at virtual[$n].interface`
+ device=`uci -P /var/state get network.${interface}.ifname`
+ alias=`uci get ipvs. at virtual[$n].alias`
+ ifconfig $device:$alias down
+ n=$((n+1))
+ done
+ ipvsadm --stop-daemon `uci get ipvs.globals.role`
+ rm -f $TMP_DIR/*.down
+ rm -rf $BASE_DIR
+ for m in `uci -q get ipvs.globals.scheduler`; do
+ cm=$((cm+1))
+ rmmod ${IPVSMOD}_$m
+ done
+ # Default to wlc if no schedulers
+ if [ $m -eq 0 ]; then
+ rmmod ${IPVSMOD}_wlc
+ fi
+ rmmod $IPVSMOD
+ crontab -l |grep -v IPVS_SCHEDULER|crontab -
+}
diff --git a/package/network/utils/ipvsadm/files/usr/sbin/checkRealServers
b/package/network/utils/ipvsadm/files/usr/sbin/checkRealServers
new file mode 100755
index 0000000..f801c4b
--- /dev/null
+++ b/package/network/utils/ipvsadm/files/usr/sbin/checkRealServers
@@ -0,0 +1,36 @@
+#!/bin/sh
+################################################################################
+# Author: Mauro Mozzarelli
+# Description: checks whether a real server is up and running and if it is
+# then it starts the services
+# Dependencies: /tmp/ipvsadm.d
+# configuration files created by /etc/init.d/ipvsadm
+# virt.* = virtual servers
+# *.stop = real server in down state ipvs table
+# *.start = real server in up state ipvs table
+# *.down = real server down status
+################################################################################
+
+BASE_DIR=/tmp/ipvsadm.d
+TMP_DIR=$BASE_DIR
+REAL_SERVERS=realServers
+
+if [ -f $BASE_DIR/$REAL_SERVERS ]; then
+ for i in `cat $BASE_DIR/$REAL_SERVERS`; do
+ ping -qc 3 $i > /dev/null
+ RESULT=$?
+ if [ $RESULT -gt 0 ]; then
+ if [ ! -f $TMP_DIR/$i.down ]; then
+ ipvsadm -R < $BASE_DIR/$i.stop
+ date > $TMP_DIR/$i.down
+ logger -p info "IPVS Server $i is down"
+ fi
+ else
+ if [ -f $TMP_DIR/$i.down ]; then
+ ipvsadm -R < $BASE_DIR/$i.start
+ rm $TMP_DIR/$i.down
+ logger -p info "IPVS Server $i is on-line"
+ fi
+ fi
+ done
+fi
_______________________________________________
openwrt-devel mailing list
openwrt-devel at lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel
More information about the openwrt-devel
mailing list