Asterisk integration with Huawei Soft-Switch

I needed to configure a SIP trunk with Asterisk to a provider using a Huawei soft switch.

It was not painless….

Firstly there was a well known error regarding silence suppression:

In the SIP INVITE asterisk sends: “a=silenceSupp:off – – – – –

It is too difficult for Huawei’s coders to accomodate this, so it sends back:

Got SIP response 500 “Server Internal Error” back from XXX.XXX.XXX.XXX

So the way this can be circumvented is by doing this:

So my example use asterisk 10.2.0

Go look at your asterisk source file in chan_sip.c around line 11814

 

Comment out this section:

/* ast_build_string(&a_audio_next, &a_audio_left, "a=silenceSupp:off - - - - -\r\n"); */


Recompile and install.
Now that has fixed the problem with the silence suppression.

Another error appeared that the Huawei soft switch is quite pedantic with session expiry.

Outgoing calls from asterisk to Huawei will work, but incoming calls will be dropped when answered.
this is due to this sip communication:

This is fixed by adding : session-timers=refuse” in my sip.conf

sip.conf file that works for me:

[myprovider]
type=peer
host=XX.XX.6.5
disallow=all
allow=g729
allow=alaw
insecure=port,invite
qualify=yes
nat=no</span>
context=from-myprovider
canreinvite=no
t38pt_udptl = yes
session-timers=refuse

Time will tel how stable this config is, but it looks promising.
Hope this helps somebody out there.

 

Updating Ruby in Centos 5.X

Many thanx to John Alberts whom created this script.

I found this invaluable while working with Metasploit.

Originally from: https://raw.github.com/albertsj1/Misc-Scripts/master/install_gems.sh

 

#!/bin/sh

########################################
# Created by John Alberts
# Last modified: 04/21/2011
#
# Error Codes:
#  1 - Not running as root
#  2 - Invalid hostname
#  3 - Failed to get remove Ruby OS packages
#  4 - Failed to compile and install Ruby
#
# NOTES:
#  This only works on CentOS 5.  Only tested on x86_64
#
#########################################

#RUBY_SOURCE_URL="ftp://ftp.ruby-lang.org/pub/ruby/1.9/ruby-1.9.2-p136.tar.gz"
RUBY_SOURCE_URL="http://ftp.ruby-lang.org/pub/ruby/1.9/ruby-1.9.2-p180.tar.gz"
# The below URL only works from within the exlibrisgroup network.  Anyone else should use the URL above.
# RUBY_SOURCE_URL="https://helpdesk.hosted.exlibrisgroup.com/downloads/ruby-1.9.2-p180.tar.gz"

if ! ( whoami | grep root > /dev/null 2>&1); then
  echo "YOU MUST BE ROOT TO RUN THIS SCRIPT"'!'
  exit 1
fi

if ! ( ping -c1 -q `hostname -f` > /dev/null 2>&1 ); then
  echo "hostname -f must be a valid fqdn for Chef to work properly"'!'
  exit 2
fi

echo "Removing already installed Ruby OS packages..."
PKGLIST="$(yum list | grep installed | grep ruby | sed -n 's/\([^.]*\)\.\(x86_64\|i386\).*$/\1/p' | tr '\n' ' ')"
if [[ $PKGLIST != "" ]]; then
  yum -y erase $PKGLIST
  RETVAL=$?
else
  RETVAL=0
fi

echo;echo
if [[ ${RETVAL} -ne 0 ]]; then
  echo "Failed to remove Ruby OS packages"'!'
  exit 3
fi

echo "Installing Ruby and dependencies..."
yum -y install gcc gcc-c++ zlib-devel openssl-devel readline-devel make

mkdir /tmp/sources
cd /tmp/sources

# Get # cpu's to make this faster
if [[ ! $CPUS ]]; then
  CPUS="$(grep processor /proc/cpuinfo | wc -l)"
fi

wget "${RUBY_SOURCE_URL}"
tar -xvzf $(basename ${RUBY_SOURCE_URL})
cd $(basename ${RUBY_SOURCE_URL/.tar.gz})
./configure
make -j $CPUS
make -j $CPUS install
RETVAL=$?

echo;echo

if [[ ${RETVAL} -ne 0 ]]; then
  echo "RUBY INSTALLATION FAILED"'!'
  exit 4
fi

echo 'gem: --no-ri --no-rdoc' > /root/.gemrc

echo "Installation completed."

Many thanx to John!

Dealing with outgoing smtp problems

In some of my networks there are a huge number users connected through a  internet connection with NAT.
Users do pick up virus’s and bots from time to time. The result is lots of port 25 smtp traffic streaming out the network and the gateway ip obviously gets blacklisted soon.

Seems like SMTP-Gated offers some help with this.

I start of with building a rpm of smtp-gated.
Getting a rpm building environment is easy:

yum install rpm-build redhat-rpm-config gcc openssl openssl-devel

Add a user. It is essential that you NEVER build rpm’s as root.

useradd devel
password devel

Login as newly created devel user.

Add these to make build directory structure:

mkdir -p ~/rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
echo '%_topdir %(echo $HOME)/rpmbuild' >> ~/.rpmmacros

LUSCA Caching Server

Attempt at LUSCA  High performance caching server with tproxy and store rewrite.
Most of this stuff has been taken from forums, google etc…

1. Install Ubuntu server 10.04.2 64 bit
2. Install dependencies
apt-get install gcc make automake sharutils linux-headers-`uname -r` libcap libcap-dev

Check if correct kernel modules gets loaded:

cat /boot/config-`uname -r` | grep -E '(NF_CONNTRACK=|TPROXY|XT_MATCH_SOCKET|XT_TARGET_TPROXY)'
This is a good result:
CONFIG_NF_CONNTRACK=m
CONFIG_NETFILTER_TPROXY=m
CONFIG_NETFILTER_XT_TARGET_TPROXY=m
CONFIG_NETFILTER_XT_MATCH_SOCKET=m

3. Obtain Lusca Head branch of Squid

cd /usr/src/
wget http://lusca-cache.googlecode.com/files/LUSCA_HEAD-r14809.tar.gz
tar -xvf LUSCA_HEAD-r14809.tar.gz
cd LUSCA_HEAD-r14809
</pre>
4. Build Lusca
# Start configuring and compiling
<code>[bash]
./bootstrap.sh
make distclean

Configure Options Debian:

./configure '--prefix=/usr/local/lusca/' '--enable-async-io' '--with-pthreads' '--enable-storeio=aufs,coss,null' '--enable-linux-netfilter' '--enable-arp-acl' '--enable-epoll' '--enable-removal-policies=lru,heap' '--enable-snmp' '--enable-delay-pools' '--enable-htcp' '--enable-cache-digests' '--enable-referer-log' '--enable-useragent-log' '--enable-follow-x-forwarded-for' '--with-large-files' '--with-maxfd=65536' 'amd64-debian-linux' '--enable-linux-tproxy4' 'build_alias=amd64-debian-linux' 'host_alias=amd64-debian-linux' 'target_alias=amd64-debian-linux' 'CFLAGS=-Wall -g -O2' 'LDFLAGS=-Wl,-Bsymbolic-functions' 'CPPFLAGS='

Configure RH Options

./configure '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--target=x86_64-redhat-linux-gnu'  '--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin'  '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--includedir=/usr/include'  '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--exec_prefix=/usr' '--bindir=/usr/sbin' '--libexecdir=/usr/lib64/squid' '--localstatedir=/var' '--datadir=/usr/share' '--sysconfdir=/etc/squid' '--disable-dependency-tracking' '--disable-arp-acl' '--disable-cache-digests' '--enable-cachemgr-hostname=localhost' '--disable-delay-pools' '--enable-epoll'  '--enable-external-acl-helpers=ip_user,unix_group' '--enable-icap-client'  '--disable-ident-lookups' '--enable-referer-log'  '--enable-removal-policies=heap,lru' '--disable-snmp' '--enable-ssl' '--enable-storeio=aufs,coss' '--disable-useragent-log' '--disable-wccpv2' '--with-aio' '--with-default-user=squid' '--with-dl' '--with-pthreads' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu'  'target_alias=x86_64-redhat-linux-gnu' 'CFLAGS=-fPIE -Os -g -pipe  -fsigned-char -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic'  'LDFLAGS=-pie' 'CXXFLAGS=-fPIE -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2  -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64  -mtune=generic' 'FFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic  -I/usr/lib64/gfortran/modules' '--with-large-files' '--enable-linux-netfilter' '--with-maxfd=32000' '--disable-follow-x-forwarded-for' '--enable-truncate' '--disable-unlinkd'  '--disable-htcp' '--enable-kill-parent-hack'

make all
make install

5. Post install
# Link install location to general file system locations

ln -s /usr/local/lusca/etc /etc/lusca
ln -s /usr/local/lusca/var/logs /var/log/lusca
mkdir /usr/local/lusca/var/cache

# Prepare cache and log dir with correct permissions

chown nobody /usr/local/lusca/var/cache /usr/local/lusca/var/logs
ln -s /usr/local/lusca/sbin/squid /usr/sbin/squid
ln -s /usr/local/lusca/bin/squidclient /usr/bin/squidclient
</pre>
# Setup Cron
4 * * * * /usr/sbin/squid -k rotate

# Copy rc startup script
<code>[bash]
cp /usr/src/LUSCA_HEAD-r14809/contrib/lusca.rc /etc/init.d/lusca
# make executeable
chmod +x /etc/init.d/lusca


6. Configure Lusca:

 

My squid.conf

#######################################################
## Lusca High Performance Configuration
## ====================================================
## By. teukuri...@yahoo.com.sg
## Updated:    April 24,2011
#######################################################
##start of config
#http_port 3128 tproxy transparent
http_port 3128 transparent
#http_port 3128 transparent
#server_http11 on
#icp_port 0
# cache_peer 203.128.88.193 parent 8910 0 no-query no-netdb-exchange
no-digest

# File Squid
#pid_filename /var/run/lusca.pid
#coredump_dir /usr/local/lusca/var/cache
#error_directory /usr/share/squid/errors/English
#icon_directory /usr/share/squid/icons
#mime_table /etc/squid/mime.conf
visible_hostname pcw_cache1
cache_mgr support@example.com

# Log Squid
access_log /var/log/squid/access.log squid
#acces_log none
#cache_log none
cache_store_log none

# Beberapa log yg tidak signifikan karena opsi2-nya jarang digunakan.
log_fqdn off
log_icp_queries off
buffered_logs off
emulate_httpd_log off

#===========================================================================
# TAG: FTP section
#---------------------------------------------------------------------------
# Always use this option to reduce the traffic load. FTP passive mode
# causes the client to not relate directly to an FTP server destination
# and using squid as a relay so that the bandwidth for FTP connection
# can be controlled possibly by a combination of IP filtering (PF (BSD) / IPTABLES).
#===========================================================================
ftp_list_width 32
ftp_passive on
ftp_sanitycheck on

#===================================================================
# TAG: ACL Section
#-------------------------------------------------------------------
# Allow local network(s) on interface(s)
# Example rule allowing access from your local networks.
# Adapt to list your (internal) IP networks from where browsing
# should be allowed
#===================================================================
# acl localnet src "/etc/squid/ip-privadas"    # IP LOCALES
acl localnet src 192.168.0.0/16
acl localnet src 172.16.0.0/12
acl localnet src 172.101.0.0/24
acl localnet src 172.50.4.0/24
acl localnet src 10.0.0.0/8

uri_whitespace strip

#DNS NAMESERVER

dns_nameservers 172.16.1.2
dns_nameservers 8.8.8.8

cache_mem 780 MB
maximum_object_size_in_memory 16 KB
memory_replacement_policy lru
cache_replacement_policy lru

cache_dir aufs /var/spool/squid/ 51200 32 256
cache_dir aufs /cache/cache1 51200 32 256
cache_dir aufs /cache/cache2 51200 32 256
cache_dir aufs /cache/cache3 51200 32 256

minimum_object_size 0 bytes
maximum_object_size 704 MB
offline_mode off
cache_swap_low 80%
cache_swap_high 100%

# Setup some default acls
acl all src 0.0.0.0/0
acl localhost src 127.0.0.1/32
acl Safe_ports port 21 # ftp
acl Safe_ports port 70 # gopher
acl Safe_ports port 80 # http
acl Safe_ports port 81 #
acl Safe_ports port 210 # wais
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 443 # https
acl Safe_ports port 488 # gss-http
acl Safe_ports port 563 #
acl Safe_ports port 591 # filemaker
acl Safe_ports port 631 #
acl Safe_ports port 777 # multiling http
acl Safe_ports port 901 3128 # multiling http
acl Safe_ports port 1025-65535 # unregistered ports

acl sslports port 443 563 81
acl manager proto cache_object
acl purge method PURGE
acl connect method CONNECT
acl dynamic urlpath_regex cgi-bin \?
http_access allow manager localhost all
http_access deny manager
http_access allow purge localhost
http_access deny purge
http_access deny !Safe_ports
http_access deny CONNECT !sslports
# Always allow localhost connections
http_access allow localhost

# Allow local network(s) on interface(s)
http_access allow localnet

### no cachear sitios especificos ###
#acl no_cachear dstdomain "/etc/squid/sitios-excluidos.conf"
#no_cache deny no_cachear
#always_direct allow no_cachear

# Default block all to be sure
via off
forwarded_for off
http_access deny all
#header_access From deny all
#header_access Server deny all
#header_access WWW-Authenticate deny all
#header_access Link deny all
#header_access Cache-Control deny all
#header_access Proxy-Connection deny all
#header_access X-Cache deny all
#header_access X-Cache-Lookup deny all
header_access Via deny all
header_access Forwarded-For deny all
header_access X-Forwarded-For deny all
#header_access Pragma deny all
#header_access Keep-Alive deny all

##follow_x_forwarded_for allow localnet
##follow_x_forwarded_for allow localhost

http_gzip on
http_gzip_types text/plain,text/html,text/xml,text/css,application/xml,application/xhtml+xml,application/rss+xml,application/javascript,application/x-javascript

#=====================================================
# TAG: ZPH
#=====================================================
tcp_outgoing_tos 0x30 localnet
zph_mode tos
zph_local 0x30
zph_parent 0
#zph_option 136

#====================================================
# TAG: STORE RULE
#====================================================
# Caching Youtube
acl store_rewrite_list urlpath_regex \/(get_video|videoplayback\?id|videoplayback.*id)
acl store_rewrite_list urlpath_regex \.(jp(e?g|e|2)|gif|png|tiff?|bmp|ico|flv|wmv|3gp|mp(4|3)|exe|msi|zip|on2|mar)\?
acl store_rewrite_list_domain url_regex ^http:\/\/([a-zA-Z-]+[0-9-]+)\.[A-Za-z]*\.[A-Za-z]*
acl store_rewrite_list_domain url_regex (([a-z]{1,2}[0-9]{1,3})|([0-9]{1,3}[a-z]{1,2}))\.[a-z]*[0-9]?\.[a-z]{3}
acl store_rewrite_list_path urlpath_regex \.(jp(e?g|e|2)|gif|png|tiff?|bmp|ico|flv|avc|zip|mp3|3gp|rar|on2|mar|exe)$
acl store_rewrite_list_domain_CDN url_regex \.rapidshare\.com.*\/[0-9]*\/.*\/[^\/]* ^http:\/\/(www\.ziddu\.com.*\.[^\/]{3,4})\/(.*) \.doubleclick\.net.*
acl store_rewrite_list_domain_CDN url_regex ^http:\/\/[.a-z0-9]*\.photobucket\.com.*\.[a-z]{3}$ quantserve\.com
acl store_rewrite_list_domain_CDN url_regex ^http:\/\/[a-z]+[0-9]\.google\.co(m|\.id)
acl store_rewrite_list_domain_CDN url_regex ^http:\/\/\.www[0-9][0-9]\.indowebster\.com\/(.*)(rar|zip|flv|wm(a|v)|3gp|mp(4|3)|exe|msi|avi|(mp(e?g|a|e|1|2|3|4))|cab|exe)
acl videocache_allow_url url_regex -i \.googlevideo\.com\/videoplayback \.googlevideo\.com\/videoplay \.googlevideo\.com\/get_video\?
acl videocache_allow_url url_regex -i \.google\.com\/videoplayback \.google\.com\/videoplay \.google\.com\/get_video\?
acl videocache_allow_url url_regex -i \.google\.[a-z][a-z]\/videoplayback \.google\.[a-z][a-z]\/videoplay \.google\.[a-z][a-z]\/get_video\?
acl videocache_allow_url url_regex -i proxy[a-z0-9\-][a-z0-9][a-z0-9][a-z0-9]?\.dailymotion\.com\/
acl videocache_allow_url url_regex -i vid\.akm\.dailymotion\.com\/
acl videocache_allow_url url_regex -i [a-z0-9][0-9a-z][0-9a-z]?[0-9a-z]?[0-9a-z]?\.xtube\.com\/(.*)flv
acl videocache_allow_url url_regex -i \.vimeo\.com\/(.*)\.(flv|mp4)
acl videocache_allow_url url_regex -i va\.wrzuta\.pl\/wa[0-9][0-9][0-9][0-9]?
acl videocache_allow_url url_regex -i \.youporn\.com\/(.*)\.flv
acl videocache_allow_url url_regex -i \.msn\.com\.edgesuite\.net\/(.*)\.flv
acl videocache_allow_url url_regex -i \.tube8\.com\/(.*)\.(flv|3gp)
acl videocache_allow_url url_regex -i \.mais\.uol\.com\.br\/(.*)\.flv
acl videocache_allow_url url_regex -i \.blip\.tv\/(.*)\.(flv|avi|mov|mp3|m4v|mp4|wmv|rm|ram|m4v)
acl videocache_allow_url url_regex -i \.break\.com\/(.*)\.(flv|mp4)
acl videocache_allow_url url_regex -i redtube\.com\/(.*)\.flv
acl videocache_allow_dom dstdomain .mccont.com .metacafe.com .cdn.dailymotion.com
acl videocache_deny_dom  dstdomain .download.youporn.com .static.blip.tv
acl dontrewrite url_regex redbot\.org \.php
acl getmethod method GET

storeurl_access deny dontrewrite
storeurl_access deny !getmethod
storeurl_access allow store_rewrite_list_domain_CDN
storeurl_access allow store_rewrite_list
storeurl_access allow store_rewrite_list_domain
storeurl_access allow store_rewrite_list_path
storeurl_access deny videocache_deny_dom
storeurl_access allow videocache_allow_url
storeurl_access allow videocache_allow_dom
storeurl_access deny all

storeurl_rewrite_program /usr/sbin/storeurl.pl
storeurl_rewrite_children 1
storeurl_rewrite_concurrency 140
storeurl_bypass on

hierarchy_stoplist (ini|ui|lst|inf||mh-|sc-)$ (afs.dat|update.txt|vdf.info.gz|captcha|reset.css|gamenotice|ickernew.css)
acl QUERY urlpath_regex -i \.(ini|ui|lst|inf|mh-|sc-)$
acl QUERY urlpath_regex -i (afs.dat|captcha|reset.css|update.txt|gamenotice|vdf.info.gz)
cache deny QUERY

#=========================================================
# TAG: Refresh Pattern
#=========================================================
# 1 year = 525600 mins, 1 month = 43200 mins, 1 day = 1440
#=========================================================
refresh_pattern (get_video\?|videoplayback\?|videodownload\?|\.flv?) 43200 999999% 43200 ignore-no-cache ignore-no-store ignore-private override-expire override-lastmod reload-into-ims store-stale
refresh_pattern (get_video\?|videoplayback\?id|videoplayback.*id|videodownload\?|\.flv?) 43200 999999% 43200 ignore-no-cache ignore-no-store ignore-private override-expire override-lastmod reload-into-ims store-stale
refresh_pattern \.(ico|video-stats) 43200 999999% 43200 override-expire ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth override-lastmod ignore-must-revalidate negative-ttl=10080 store-stale
refresh_pattern \.etology\? 43200 999999% 43200 override-expire ignore-reload ignore-no-cache store-stale
refresh_pattern galleries\.video(\?|sz) 43200 999999% 43200 override-expire ignore-reload ignore-no-cache store-stale
refresh_pattern brazzers\? 43200 999999% 43200 override-expire ignore-reload ignore-no-cache store-stale
refresh_pattern \.adtology\? 43200 999999% 43200 override-expire ignore-reload ignore-no-cache store-stale
refresh_pattern ^.*(utm\.gif|ads\?|rmxads\.com|ad\.z5x\.net|bh\.contextweb\.com|bstats\.adbrite\.com|a1\.interclick\.com|ad\.trafficmp\.com|ads\.cubics\.com|ad\.xtendmedia\.com|\.googlesyndication\.com|advertising\.com|yieldmanager|game-advertising\.com|pixel\.quantserve\.com|adperium\.com|doubleclick\.net|adserving\.cpxinteractive\.com|syndication\.com|media.fastclick.net).* 43200 20% 43200 ignore-no-cache ignore-no-store ignore-private override-expire ignore-reload ignore-auth ignore-must-revalidate store-stale negative-ttl=40320 max-stale=10
refresh_pattern ^.*safebrowsing.*google  43200 999999% 43200 override-expire ignore-reload ignore-no-cache ignore-private ignore-auth ignore-must-revalidate negative-ttl=10080 store-stale
refresh_pattern ^http://((cbk|mt|khm|kh|mlt)[0-9]?)\.google\.co(m|\.uk) 43200 999999% 43200 override-expire ignore-reload ignore-private store-stale negative-ttl=10080
refresh_pattern ytimg\.com.*\.jpg 43200 999999% 43200 override-expire ignore-reload store-stale refresh_pattern images\.friendster\.com.*\.(png|gif) 43200 999999% 43200 override-expire ignore-reload store-stale
refresh_pattern garena\.com 43200 999999% 43200 override-expire reload-into-ims store-stale
refresh_pattern photobucket.*\.(jp(e?g|e|2)|tiff?|bmp|gif|png) 43200 999999% 43200 override-expire ignore-reload store-stale
refresh_pattern vid\.akm\.dailymotion\.com.*\.on2\? 43200 999999% 43200 ignore-no-cache override-expire override-lastmod store-stale
refresh_pattern mediafire.com\/images.*\.(jp(e?g|e|2)|tiff?|bmp|gif|png) 43200 999999% 43200 reload-into-ims override-expire ignore-private store-stale
refresh_pattern ^http:\/\/images|pics|thumbs[0-9]\. 43200 999999% 43200 reload-into-ims ignore-no-cache ignore-no-store ignore-reload override-expire store-stale
refresh_pattern ^http:\/\/www.onemanga.com.*\/ 43200 999999% 43200 reload-into-ims ignore-no-cache ignore-no-store ignore-reload override-expire store-stale
refresh_pattern ^http://v\.okezone\.com/get_video\/([a-zA-Z0-9]) 43200 999999% 43200 override-expire ignore-reload ignore-no-cache ignore-no-store ignore-private ignore-auth override-lastmod ignore-must-revalidate negative-ttl=10080 store-stale

# ANTI VIRUS
refresh_pattern guru.avg.com/.*\.(bin)                                  1440 999999% 10080 ignore-no-cache ignore-no-store ignore-reload  reload-into-ims store-stale
refresh_pattern (avgate|avira).*(idx|gz)$                               1440 999999% 10080 ignore-no-cache ignore-no-store ignore-reload  reload-into-ims store-stale
##refresh_pattern ^http.*(\.kaspersky-labs\.com|\.geo\.kaspersky\.com).*(\.avc|\.kdc|\.klz|\.bz2|\.dat|\.dif) 1440 999999% 10080 ignore-no-cache ignore-no-store ignore-reload reload-into-ims store-stale
##refresh_pattern kaspersky.*\.avc$ 1440 999999% 10080 ignore-no-cache ignore-no-store ignore-reload reload-into-ims store-stale
##refresh_pattern kaspersky 1440 999999% 10080 ignore-no-cache ignore-no-store ignore-reload reload-into-ims store-stale refresh_pattern update.nai.com/.*\.(gem|zip|mcs)                        1440 999999% 10080 ignore-no-cache ignore-no-store ignore-reload  reload-into-ims store-stale
refresh_pattern ^http:\/\/liveupdate.symantecliveupdate.com.*\(zip)     1440 999999% 10080 ignore-no-cache ignore-no-store ignore-reload  reload-into-ims store-stale
refresh_pattern avast.com/.*\.(vpu|vpaa)                                1440 999999% 10080 ignore-no-cache ignore-no-store ignore-reload  reload-into-ims store-stale

## WINDOWS UPDATE
refresh_pattern windowsupdate.com/.*\.(cab|exe)                 10080 999999%  43200 ignore-no-cache ignore-no-store ignore-reload reload-into-ims store-stale
refresh_pattern update.microsoft.com/.*\.(cab|exe)              10080 999999%  43200 ignore-no-cache ignore-no-store ignore-reload reload-into-ims store-stale
refresh_pattern download.microsoft.com/.*\.(cab|exe)            10080 999999%  43200 ignore-no-cache ignore-no-store ignore-reload reload-into-ims store-stale

#images facebook
refresh_pattern -i \.facebook.com.*\.(jpg|png|gif) 129600 999999% 129600 ignore-reload override-expire ignore-no-cache ignore-no-store store-stale
refresh_pattern -i \.fbcdn.net.*\.(jpg|gif|png|swf|mp3) 129600 999999% 129600 ignore-reload override-expire ignore-no-cache ignore-no-store store-stale
refresh_pattern  static\.ak\.fbcdn\.net*\.(jpg|gif|png) 129600 999999% 129600 ignore-reload override-expire ignore-no-cache ignore-no-store store-stale
refresh_pattern ^http:\/\/profile\.ak\.fbcdn.net*\.(jpg|gif|png) 129600 999999% 129600 ignore-reload override-expire ignore-no-cache ignore-no-store store-stale

# games facebook
refresh_pattern ^http:\/\/apps.facebook.com.*\/    10080 999999% 43200 ignore-reload override-expire ignore-no-cache ignore-no-store store-stale
refresh_pattern -i \.zynga.com.*\/      10080 999999% 43200 ignore-reload override-expire ignore-no-cache ignore-no-store ignore-must-revalidate store-stale
refresh_pattern -i \.farmville.com.*\/  10080 999999% 43200 ignore-reload override-expire ignore-no-cache ignore-no-store ignore-must-revalidate store-stale
refresh_pattern -i \.ninjasaga.com.*\/  10080 999999% 43200 ignore-reload override-expire ignore-no-cache ignore-no-store ignore-must-revalidate store-stale
refresh_pattern -i \.mafiawars.com.*\/  10080 999999% 43200 ignore-reload override-expire ignore-no-cache ignore-no-store ignore-must-revalidate store-stale
refresh_pattern -i \.crowdstar.com.*\/  10080 999999% 43200 ignore-reload override-expire ignore-no-cache ignore-no-store ignore-must-revalidate store-stale
refresh_pattern -i \.popcap.com.*\/     10080 999999% 43200 ignore-reload override-expire ignore-no-cache ignore-no-store ignore-must-revalidate store-stale

#banner IIX
refresh_pattern ^http:\/\/openx.*\.(jp(e?g|e|2)|gif|pn[pg]|swf|ico|css|tiff?) 129600 99999% 129600 reload-into-ims  ignore-reload override-expire ignore-no-cache ignore-no-store  store-stale
refresh_pattern ^http:\/\/ads(1|2|3).kompas.com.*\/             43200 99999% 129600 reload-into-ims  ignore-reload override-expire ignore-no-cache ignore-no-store  store-stale
refresh_pattern ^http:\/\/img.ads.kompas.com.*\/                43200 99999% 129600 reload-into-ims  ignore-reload override-expire ignore-no-cache ignore-no-store  store-stale
refresh_pattern .kompasimages.com.*\.(jpg|gif|png|swf)          43200 99999% 129600 reload-into-ims  ignore-reload override-expire ignore-no-cache ignore-no-store  store-stale
refresh_pattern ^http:\/\/openx.kompas.com.*\/                  43200 99999% 129600 reload-into-ims  ignore-reload override-expire ignore-no-cache ignore-no-store  store-stale
refresh_pattern kaskus.\us.*\.(jp(e?g|e|2)|gif|png|swf)         43200 99999% 129600 reload-into-ims  ignore-reload override-expire ignore-no-cache ignore-no-store  store-stale
refresh_pattern ^http:\/\/img.kaskus.us.*\.(jpg|gif|png|swf)    43200 99999% 129600 reload-into-ims  ignore-reload override-expire ignore-no-cache ignore-no-store  store-stale

#IIX DOWNLOAD
refresh_pattern ^http:\/\/\.www[0-9][0-9]\.indowebster\.com\/(.*)(mp3|rar|zip|flv|wmv|3gp|mp(4|3)|exe|msi|zip) 43200 99999% 129600 reload-into-ims  ignore-reload override-expire ignore-no-cache ignore-no-store  store-stale ignore-auth

#All File
refresh_pattern -i \.(class|css|js|tif)(\?.*)?$ 1440 95% 100000080 reload-into-ims override-lastmod
refresh_pattern -i \.(jpe|jpg|jpeg|png|bmp|gif)(\?.*)?$ 0 95% 1000000080 reload-into-ims override-lastmod
refresh_pattern -i \.(tiff|mov|avi|qt|mpeg|3gp)(\?.*)?$ 0 95% 201600000 reload-into-ims override-lastmod
refresh_pattern -i \.(mpg|mpe|wav|au|mid|flv)(\?.*)?$ 0 95% 2016000 reload-into-ims override-lastmod
refresh_pattern -i \.(zip|gz|arj|lha|lzh)(\?.*)?$ 0 95% 2016000 reload-into-ims override-lastmod
refresh_pattern -i \.(rar|tgz|tar|exe|bin)(\?.*)?$ 0 95% 2016000 reload-into-ims override-lastmod
refresh_pattern -i \.(cab|psf|part|0*|swf|gz|grf|gpf)(\?.*)?$ 0 95% 2000160 reload-into-ims override-lastmod
refresh_pattern -i \.(pdf|rtf|doc|swf|txt|inf)(\?.*)?$ 1 95% 2016000 reload-into-ims override-lastmod
refresh_pattern -i \.(inc|cab|ad|hqx|dll)(\?.*)?$ 10080 95% 4320000 reload-into-ims override-lastmod
refresh_pattern -i \.(asp|acgi|pl|shtml|php3|php)(\?.*)?$ 2 20% 432000 reload-into-ims override-lastmod
refresh_pattern -i \.(ini)(\?.*)?$ 2 5% 10800 reload-into-ims override-lastmod

refresh_pattern -i (/cgi-bin/|\?)  0  0%  0
refresh_pattern ^ftp:           1440    20%     10080
refresh_pattern ^gopher:        1440    0%      1440
refresh_pattern .               0       40%     40320

#==============================================
# TAG: Extra Tuning Configuration
#==============================================
header_access Accept-Encoding deny  all
client_persistent_connections off
server_persistent_connections on
half_closed_clients off
strip_query_terms off
quick_abort_min 0 KB
quick_abort_max 0 KB
quick_abort_pct 100
vary_ignore_expire on
reload_into_ims on
pipeline_prefetch on
#range_offset_limit 512 KB
read_timeout 30 minutes
client_lifetime 6 hours
negative_ttl 30 seconds
positive_dns_ttl 2 hours
negative_dns_ttl 1 minutes
pconn_timeout 15 seconds
request_timeout 1 minute
store_avg_object_size 13 KB
log_icp_queries off
ipcache_size 8192
ipcache_low 90
ipcache_high 95
log_fqdn off
fqdncache_size 1024
memory_pools off
forwarded_for on
logfile_rotate 1
store_dir_select_algorithm round-robin
#cache_effective_user nobody
#cache_effective_group nobody
max_filedescriptors 8192

uri_whitespace strip
shutdown_lifetime 10 seconds
load_check_stopen on
load_check_stcreate on
download_fastest_client_speed on

url_rewrite_program /usr/sbin/updxlrator
url_rewrite_children 60

##end of config

7. Configure networking
#Begin Script 1##################################################################################
#!/bin/bash
sysctl net.ipv4.ip_nonlocal_bind=1
sysctl net.ipv4.ip_forward=1
/sbin/modprobe ip_conntrack_ftp
/sbin/modprobe ip_conntrack
/sbin/modprobe nf_conntrack
/sbin/modprobe ipt_state
/sbin/modprobe ipt_limit
/sbin/modprobe ipt_LOG
/bin/echo "262144" > /proc/sys/vm/min_free_kbytes
/bin/echo "60" > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_time_wait
/bin/echo "3276822" > /proc/sys/net/nf_conntrack_max
/bin/echo "15" > /proc/sys/net/ipv4/netfilter/ip_conntrack_udp_timeout
/bin/echo "90" > /proc/sys/net/ipv4/netfilter/ip_conntrack_udp_timeout_stream
/bin/echo "300" > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_established
#End Script 1##################################################################################

#Begin Script 2##################################################################################
#!/bin/bash

CACHE_PORT="8080"
WAN="eth1"
LAN="eth0"
BRIDGE="br0"

/bin/echo "1" > /proc/sys/net/ipv4/ip_forward

#########################################################################################################
# BRIDGE MODE - 2 INTERFACES - TPROXY ON
#########################################################################################################
#/sbin/iptables -t nat -F
#/sbin/iptables -t nat -X
#/sbin/iptables -t mangle -F
#/sbin/iptables -t mangle -X
#/sbin/ebtables -t broute -F
#/sbin/ebtables -t broute -X
#/sbin/iptables -t mangle -N DIVERT
#/sbin/iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
#/sbin/iptables -t mangle -A DIVERT -j MARK --set-mark 1
#/sbin/iptables -t mangle -A DIVERT -j ACCEPT
#/sbin/iptables -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY --tproxy-mark 0x1/0x1 --on-port $CACHE_PORT
#/sbin/ebtables -t broute -A BROUTING -i $LAN -p ipv4 --ip-proto tcp --ip-dport 80 -j redirect --redirect-target ACCEPT
#/sbin/ebtables -t broute -A BROUTING -i $WAN -p ipv4 --ip-proto tcp --ip-sport 80 -j redirect --redirect-target ACCEPT

#########################################################################################################
# BRIDGE MODE - 2 INTERFACES - TPROXY OFF
#########################################################################################################
#/sbin/iptables -t nat -F
#/sbin/iptables -t nat -X
#/sbin/iptables -t mangle -F
#/sbin/iptables -t mangle -X
#/sbin/ebtables -t broute -F
#/sbin/ebtables -t broute -X
#/sbin/ebtables -t broute -A BROUTING -p IPv4 --ip-protocol 6 --ip-destination-port 80 -j redirect --redirect-target ACCEPT
#/sbin/iptables -t nat -A PREROUTING -i $BRIDGE -p tcp --dport 80 -j REDIRECT --to-port $CACHE_PORT

#########################################################################################################
# GATEWAY MODE - 2 INTERFACES - TPROXY ON
#########################################################################################################
#/sbin/iptables -t nat -F
#/sbin/iptables -t nat -X
#/sbin/iptables -t mangle -F
#/sbin/iptables -t mangle -X
#/sbin/ebtables -t broute -F
#/sbin/ebtables -t broute -X
#/sbin/iptables -t mangle -N DIVERT
#/sbin/iptables -t mangle -A DIVERT -j MARK --set-mark 1
#/sbin/iptables -t mangle -A DIVERT -j ACCEPT
#/sbin/iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
#/sbin/iptables -t mangle -A PREROUTING -i $LAN -p tcp --dport 80 -j TPROXY --tproxy-mark 1 --on-port $CACHE_PORT
#/sbin/ip rule add fwmark 1 lookup 100
#/sbin/ip route add local 0.0.0.0/0 dev lo table 100
#/bin/echo 1 > /proc/sys/net/ipv4/conf/$WAN/proxy_arp

#########################################################################################################
# GATEWAY MODE - 2 INTERFACES - TPROXY OFF
#########################################################################################################
#/sbin/iptables -t nat -F
#/sbin/iptables -t nat -X
#/sbin/iptables -t mangle -F
#/sbin/iptables -t mangle -X
#/sbin/ebtables -t broute -F
#/sbin/ebtables -t broute -X
#/sbin/iptables -t nat -A PREROUTING -i $LAN -p tcp --dport 80 -j REDIRECT --to-port $CACHE_PORT
#/sbin/iptables -t nat -A POSTROUTING -o $WAN -j MASQUERADE
#End Script
#########################################################################################################

References:
http://www.icez.net/blog/477/lusca-cdn-installation  # Good startup conf. Need to adapt for ubuntu server

 

Xen based Windows 2003 Server VM

Recently i had to deploy a win2k3 server on  Xen.

I had a few problems along the way.
1. Had a dog-slow old Centos 5.4 based server with several vm’s running on it.
2. Had to relocate one image to a new faster server via disk backup because of various reasons.
3. Had to import an existing Xen production image to the new server.
4. The new faster Centos 5.6 based server was a standard install and not installed with Xen.

So here goes:

Shutdown host on existing server.
command: virsh shutdown mymachine (mymachine the host to be moved)

Backup to external drive witch is formatted with ntfs.
For this to work i needed fuse and fuse-ntfs.

Install rpmforge
rpm –import http://apt.sw.be/RPM-GPG-KEY.dag.txt
rpm -i rpmforge-release-0.5.2-2.el5.rf.*.rpm
yum install fuse fuse-ntfs-3g dkms dkms-fuse

 

Backup existing image
cp -v /var/lib/xen/images/mymachine.img /mnt/exthdd/
umount /mnt/exthdd

Change existing Centos install to Xen host.

yum update
reboot if newer kernel was installed

Install Xen
yum groupinstall virtualization
chkconfig xend on
chkconfig libvirtd on

After reboot, edit the grub configuration file to make sure the system boots with xen based kernel.
vi /boot/grub/grub.conf and set the xen based kernal as the default. In this case 1

##############################################################
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /boot/, eg.
#          root (hd0,0)
#          kernel /vmlinuz-version ro root=/dev/sda3
#          initrd /initrd-version.img
#boot=/dev/sda
default=1
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.18-238.12.1.el5)
root (hd0,0)
kernel /vmlinuz-2.6.18-238.12.1.el5 ro root=LABEL=/
initrd /initrd-2.6.18-238.12.1.el5.img
title CentOS (2.6.18-238.12.1.el5xen)
root (hd0,0)
kernel /xen.gz-2.6.18-238.12.1.el5
module /vmlinuz-2.6.18-238.12.1.el5xen ro root=LABEL=/
module /initrd-2.6.18-238.12.1.el5xen.img
title CentOS (2.6.18-238.9.1.el5xen)
root (hd0,0)
kernel /xen.gz-2.6.18-238.9.1.el5
module /vmlinuz-2.6.18-238.9.1.el5xen ro root=LABEL=/
module /initrd-2.6.18-238.9.1.el5xen.img
title CentOS (2.6.18-238.9.1.el5)
root (hd0,0)
kernel /vmlinuz-2.6.18-238.9.1.el5 ro root=LABEL=/
initrd /initrd-2.6.18-238.9.1.el5.img
title CentOS (2.6.18-194.32.1.el5)
root (hd0,0)
kernel /vmlinuz-2.6.18-194.32.1.el5 ro root=LABEL=/
initrd /initrd-2.6.18-194.32.1.el5.img
##############################################################

Reboot

Check if you are running xen kernel.
Linux myserver 2.6.18-238.12.1.el5xen #1 SMP Tue May 31 14:02:29 EDT 2011 x86_64 x86_64 x86_64  GNU/Linux

Test host disk performance.

Test 1:
#hdparm -tT /dev/sda1

/dev/sda1:
Timing cached reads:   3696 MB in  2.00 seconds = 1847.92 MB/sec
Timing buffered disk reads:  100 MB in  1.19 seconds =  84.15 MB/sec

Pretty descent.

Test 2:
#dd if=/dev/zero of=./test1024M bs=1024k count=1024 conv=fsync
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 16.5377 seconds, 64.9 MB/s

Also pretty descent.

Restore image from backup on new server:

cp -v mymachine.img /var/lib/libvirt/images/

Import existing disk image with Virtual Machine Manager. I used VMM with Ubuntu Maverick Meerkat.

The Windows 2003 server was now up and running after assigning 1 CPU and 1024Mb of ram to the Virtual Machine with VM Manager.

Performance was however still sluggish with high (up to 60.5%wa) in top. Write speed was terrible to say the least.

Found a post on another forum telling me to disable ACPI.

“If your Windows installation is set to use ACPI (this is the default), kvm can be quite slow or not work at all. This is due to Windows heavily using a register that has a very large virtualization penalty.

Fortunately, there is a simple workaround available: disable ACPI support in Windows. The procedure for doing this is:

Select “My Computer” with the right mouse button.
Select “Properties”.
Choose the “Hardware” tab.
Click the “Device Manager” button.
Select the entry under “Computer” with the right mouse button. If it says “Standard PC”, then there’s no need to do anything.
Select “Properties”
Click the “Update Driver” button.
Choose “No, not this time” and click “Next”.
Choose “Install from a list” and click “Next”.
Choose “Don’t search” and click “Next”.
Click “Next”.
Choose “Standard PC” and click “Next”.
Continue clicking “Next” and reboot the virtual machine.
If you’re installing Windows XP/2000, you can get it to use the Standard PC HAL by pressing F7 when it asks for drivers (”Press F6 for…”), there is no visual indication. If you would rather select the right HAL from a list visually then press F5 instead.”

BTW: Thanx for the info testy http://www.peakscale.com/archives/gridvm/xen-vs-kvm/

Then, i installed the Xen drivers found at http://wiki.xensource.com/xenwiki/XenWindowsGplPv
This case i used: http://www.meadowcourt.org/downloads/gplpv_2003x32_0.11.0.238.msi as i was using a 32 bit version of Win2k3.

Results:
Local Backup on the Win2k3 VM using Uranium Backup free before migration:  “Time taken for last execution: 6 Hours, 39 Minutes, 35 Seconds”

Local Backup on the Win2k3 VM using Uranium Backup free after migration, disabling ACPI and installing Xen drivers:  “Time taken for last execution: 0 Hours, 4 Minutes, 53 Seconds”

Size ~ 4Gig

References:
http://southbrain.com/south/2009/08/xen-drivers-for-windows-2003-m.html
http://blog.surfulater.com/2007/01/10/xen-and-the-art-of-virtualization/