VMWare 2.0.x and CentOS 5.4 glibc crashes and instabilities – a workaround solution
If you're running VMWare Server 2.0.x on CentOS 5.4, you'll have noticed that you (most likely) suffer random reboots and other instabilities on the guest operating systems - this is due to a glibc error in vmware's libraries, and has yet to be fixed by vmware (thanks a lot). The real issue is CentOS/Redhat has updated glibc, and vmware just hasn't kept up with it.
Some more details on vmware's forums:
http://communities.vmware.com/thread/230842?tstart=0
The steps to fix this are as follows:
Use the following if you've already upgraded to 5.4, and need to downgrade to 5.3 glibc libraries for vmware to work...
# code
cd /etc/yum.repos.d# copy current .repo to .repo we'll tailor for CentOS 5.3
cp CentOS-Base.repo CentOS53-Base.repo# replace $releasever with 5.3
sed -i 's/$releasever/5.3/g' CentOS53-Base.repo# append '53' to each repo name
sed -i 's/]/53]/g' CentOS53-Base.repo# downgrade glibc, etc.
yum downgrade glibc\* nscd gcc gcc-c++ cpp libstdc++\* nss_ldap# add the following to /etc/yum.conf
exclude=glibc* nscd gcc gcc-c++ cpp libstdc++* nss_ldap# recompile/configure VMWare Server
vmware-config.pl
UPDATE: Mid February 2010 - the 5.3 repos are now out of service, so you can no longer use the mirrorlist!
Since all 5.3 repos are now gone, so the update is as follows to use vault.centos.org backup copy of 5.3 rpm's:
cd /etc/yum.repos.d/
sed -i 's/mirrorlist/#mirrorlist/g' CentOS53-Base.repo
sed -i 's/#baseurl=http:\/\/mirror.centos.org\/centos/baseurl=http:\/\/vault.centos.org/g' CentOS53-Base.repo
to change this:
[base53]
name=CentOS-5.3 - Base
mirrorlist=http://mirrorlist.centos.org/?release=5.3&arch=$basearch&repo=os
#baseurl=http://mirror.centos.org/centos/5.3/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5
to this:
[base53]
name=CentOS-5.3 - Base
#mirrorlist=http://#mirrorlist.centos.org/?release=5.3&arch=$basearch&repo=os
baseurl=http://vault.centos.org/5.3/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5
Part of this is here at vmware's forum as an update to 5.3 repos missing:
http://communities.vmware.com/message/1487959#1487959
Yum update check script – runs via crontab and emails when new updates are available
Here's a quick script that will check yum updates and email you when there are new updates available.
Split into two sections, one script is for all package updates, and the other script is for letting us know if we need to reboot when a new kernel package has been installed.
Script to check for updates:
check-yum-updates.sh
#!/bin/bash
#
# check-yum-updates.sh
#
# checks for yum updates and emails if there are any available
#
#
# Eric Thern
# Zoidial Incorporated
# http://www.zoidial.com
#
# last update:
# Dec 30, 2008
##
# change this to your email
#
email="youremail@youremail.com"#
# no need to change anything below here
#yumtmp="/tmp/yum-check-update.$$"
yum="/usr/bin/yum"$yum check-update >& $yumtmp
yumstatus="$?"
hostname=$(/bin/hostname)
case $yumstatus in
0)
# no updates!
exit 0
;;
*)
date=$(date)
number=$(cat $yumtmp | egrep '(.i386|.x86_64|.noarch|.src)' | wc -l)
updates=$(cat $yumtmp | egrep '(.i386|.x86_64|.noarch|.src)')
echo "
There are $number updates available on host $hostname at $dateThe available updates are:
$updates
" | /bin/mail -s "UPDATE: $number updates available for $hostname" $email
;;
esac# clean up
rm -f /tmp/yum-check-update.*
Script to check kernel (here we use 'ovzkernel' since we're running with an openvz kernel, if you have a stock centos kernel, change this to 'kernel')
check-yum-kernel.sh
#!/bin/bash
email=youremail@youremail.com
latestkernel=$(rpm -q ovzkernel |tail -n1|sed -e 's/kernel-//')echo "$latestkernel"
if uname -a | grep -qv "$latestkernel"; then
echo "Running Kernel is" `uname -r` "but latest installed rpm is ${latestkernel}" |\
mail -s "UPDATE: ${HOSTNAME} reboot required" $email
fi;
Crontab entries:
30 21 * * * /root/bin/check-yum-updates.sh >/dev/null 2>&1
30 21 * * * /root/bin/check-yum-kernel.sh >/dev/null 2>&1
Run at 9:30 every night. Change times and paths to suit.
logrotate tweaks to enhance rsync backups
logrotate, by default (at least on CentOS as of 3, 4, and 5.x), rotates logs by incrementing all numbers on previous logs by one, and moving the current log to log.0
This is all fine and good if you like your logs to all be rotated in order, and have a certain number backed up:
secure --> secure.0.gz --> secure.1.gz --> secure.2.gz ...
the next 'logrotate' will move secure.0.gz to secure.1.gz, secure.1.gz to secure.2.gz, secure.2.gz to secure.3.gz ...
This is very problematic when you use rsync to back up files, and when you include logs in your backups. The issue is that rsync will see all rotated files as "brand new" files, and thus have to download all of them again and again, wasting time, disk I/O, bandwidth, etc.
To alleviate this issue, you can add 'dateext' to /etc/logrotate.conf - which will now back up all logs with a date extension, rather than just an incrementing number. This causes rotated logs to be static, and rsync will not have to download them again and again and again based on their file name changing and their 'contents' differing.
From the logrotate manpage:
dateext
Archive old versions of log files adding a daily extension like YYYYMMDD instead of simply adding a number.
Now your logs look like this:
secure, secure-20091120.gz, secure-20091020.gz, secure-20090920.gz ...
the next 'logrotate' will only add a new file 'secure-20091220.gz' and leave all the others the same.
This means rsync will only have to download a single new file (and the partial "secure" log file which is currently being written to by syslog) rather than having to download the entire list of rotated logs, saving tons of time on backups when you've got lots of logs.
Further Reading
A good article about the problems related to "hindsight log rotation schemes", although with the addition (in 2005?) of the logrotate 'dateext' feature, their suggestion to use syslog-ng is not the only good option. Since a lot of the popular server distros (Redhat/CentOS/Debian) ship with syslogd/rsyslogd and logrotate, most people won't replace their syslog with syslog-ng just for time-based log rotation.
OpenVZ cache update script
Script to update OpenVZ vz cache for any template.
Useful to upgrade to newest security releases.
#!/bin/bash
#
# updates cache of VZ
#
##
# Set These (or pass from command line as below)
#
#TEMPLATE=centos-5-i386-afull
VZID=5999
#
# use this from command line:
#
# ./updatevzcache centos-5-i386-afull
#
TEMPLATE=$1
# VZID=$2#
# change these if needed
#
HOSTNAME=testvps.zoidial.com
IP=10.10.10.100
NAMESERVER=10.10.10.1#
# create vps, add IP address, set hostname, set nameserver, start vps
#vzctl create $VZID --ostemplate $TEMPLATE --ipadd $IP --hostname $HOSTNAME
vzctl set $VZID --nameserver $NAMESERVER --save
vzctl start $VZID
#vzctl enter $VZID#
# run some update commands on the VPS
#vzctl exec $VZID "yum clean all; yum upgrade -y; yum clean all; > /etc/resolv.conf"
#
# stop vps, delete ip addresses
#vzctl stop $VZID
vzctl set $VZID --ipdel all --save#
# zero out all logs in the vps (we want it to look fresh!)
#for i in `ls -1 /vz/private/$VZID/var/log`; do echo "" > /vz/private/$VZID/var/log/$i; done
for i in `ls -1 /vz/private/$VZID/var/log/*/*`; do echo "" > $i; done#
# move old vps template to a backup
#mv /vz/template/cache/$TEMPLATE.tar.gz{,-old}
#
# Enter vps private directory, make a template from this VPS (nice 19 and ionice to limit impact of this tar)
#cd /vz/private/$VZID
nice -19 ionice -c 2 -n 7 tar czf /vz/template/cache/$TEMPLATE.tar.gz .
chmod +rx /vz/template/cache/$TEMPLATE.tar.gz#
# back to /root dir
#cd /root/
#
# destroy this temporary vps and config
#vzctl destroy $VZID
rm -f /etc/vz/conf/$VZID.conf.destroyed#
# double check that the template looks like it's been updated
#ls -lh /vz/template/cache/$TEMPLATE*
Plesk 9.x overuse policy (and how to avoid domains being suspended)
Parallels, in their infinite wisdom, introduced new "overuse" policies into their Plesk control panel as of version 9.0. A great idea when just starting out, but a nightmare when upgrading from a previous version, since it changes the default behavior when clients and domain owners reach their pre-assigned limits. Now, instead of just warning that you're over a limit, it SUSPENDS the account and domain. What a great idea, how useful! Clients *love* that!
So how do you undo this when you have new accounts being added, and existing accounts that don't allow for this type of "overuse" setup, and are suspending clients and domains when they use 1KB more disk space or 1KB more bandwidth than they are supposed to?
New accounts can take advantage of the new "overuse policy" settings within the templates, but existing clients are going to have a tougher time, you'll have to update their policy in a template, then apply the new template, or you can use the CLI utilities and change the overuse policy per client, which is what I opted for here.
Using the Plesk CLI utilities (here running on CentOS 5.x):
replace "clientuser" with your client login name you wish to update
check client preferences for their overuse policy:
/usr/local/psa/bin/client_pref.sh --info clientuser |grep Overuse
Overuse policy: block
- For checking all clients on the system (produces a lot of random output):
for i in `cat /etc/passwd |grep "/var/www/vhosts" | awk -F":" '{print $1}'`; do echo -n "$i -- "; /usr/local/psa/bin/client_pref --info $i |grep Overuse; echo ""; done
update overuse policy to "notify" rather than "block":
/usr/local/psa/bin/client_pref.sh --update clientuser -overuse notify
- updating all your clients at once:
for i in `cat /etc/passwd |grep "/var/www/vhosts" | awk -F":" '{print $1}'`; do echo -n "$i -- "; /usr/local/psa/bin/client_pref --update $i -overuse notify ; echo ""; done
check if it was updated:
/usr/local/psa/bin/client_pref.sh --info clientuser |grep overuse
Overuse policy: notify
No longer need to worry about client accounts and domains being suspended due to being over their limits by mere kb! Will need to keep an eye on their use though, and watch overuse emails, since this does mean users could abuse the "limits" that have been set for them now.
Now on to the domains!
Once you're done with fixing up the client accounts, you will have to double check all those domains if you don't want them being suspended.
The following is for editing ALL domains on the Plesk system, you can just run the domain_pref script on a single domain at a time if you prefer.
check domain preferences for their overuse policy:
for i in `ls -1 /var/named/run-root/var/ | egrep -v '(in-addr.arpa|localhost.rev|saved_by_psa|named.root|make-localhost)'`; do echo -n "$i : "; /usr/local/psa/bin/domain_pref --info $i |grep Overuse; done
update overuse policy to "notify" rather than "block" on all domains:
for i in `ls -1 /var/named/run-root/var/ | egrep -v '(in-addr.arpa|localhost.rev|saved_by_psa|named.root|make-localhost)'`; do echo -n "$i : "; /usr/local/psa/bin/domain_pref --update $i -overuse notify ; done
Background:
Fairly unresolved discussion of this issue here: http://forum.parallels.com/showthread.php?t=85216
From the Plesk documentation
Overuse policy. Specify what should be done to the site when disk space and traffic limits are exceeded: To block the site, select the Overuse is not allowed option. To allow the site to operate, select the Overuse is allowed option. Select the check box Notify the domain owner about reaching the resource limits: When the specified resource usage limits are reached, Parallels Plesk Panel will send a notice to the appropriate client account's e-mail address (by default) or to the domain administrator's account, if that is specified at Home > Notifications (in the Logs & Statistics group) > Resource usage limits exceeded by domain option.Note: The overuse policy does not apply to the limits set on size of mailboxes (mailbox quota). Therefore, even if you enable overuse, be sure to allocate enough disk space to mailboxes.
From the administrators guide:
Overuse policy. Specify what should be done when disk space and monthly bandwidth (traffic) allotments are exceeded. We recommend setting this option to Overuse is allowed. Otherwise, the user account and user's sites will be blocked when the resource limits are exceeded.
Upgrading MailScanner
Easy Upgrading of MailScanner - RPM version
(For Redhat/Centos - updated from the MailScanner documentation)
1) Make a backup copy of your current MailScanner (Linux):
cp -a /etc/MailScanner /etc/MailScanner.$(date +%Y%m%d)
cp -a /usr/lib/MailScanner /usr/lib/MailScanner.$(date +%Y%m%d)
cp -a /usr/sbin/MailScanner /usr/sbin/MailScanner.$(date +%Y%m%d)
2) Download the latest version and extract
http://mailscanner.info/downloads.html
Check the changelog for upgrade notes and new features:
http://mailscanner.info/ChangeLog
Check the integrity of the downloaded file
untar the archive (tar xzf) X/
cd into the created directory
3) Upgrade MailScanner
run the install script (./install)
inspect the output for errors
manage the .rmpnew files
4) Upgrade the configuration files
upgrade_MailScanner_conf
cd /etc/MailScanner
upgrade_MailScanner_conf MailScanner.conf MailScanner.conf.rpmnew > MailScanner.new
mv -f MailScanner.conf MailScanner.old
mv -f MailScanner.new MailScanner.conf
upgrade_languages_conf
cd /etc/MailScanner/reports/en
upgrade_languages_conf languages.conf languages.conf.rpmnew > languages.new
mv -f languages.conf languages.old
mv -f languages.new languages.conf
Don’t forget to upgrade other components periodically as well (SA, DCC, Pyzor, Razor)
5) Restart MailScanner service
service MailScanner restart
6) Check for errors
Check the maillog for errors:
tail -F /var/log/maillog
Run MailScanner with --lint to make sure there are no configuration errors:
MailScanner --lint
Run SpamAssassin with --lint to make sure there are no config errors there either:
spamassassin --lint
- or -
spamassassin --lint -D
(shows output of --lint)
Run check_mailscanner to make sure MailScanner processes are running:
check_mailscanner
7) Send a test mail
If you're really concerned, and want to make sure everything works, it's a good idea to send a couple test mails from external servers to see if they get through your newly updated MailScanner server.
Dell 1650 server and IPMI support on CentOS 5
So I bought a Dell 1650 server on ebay for cheap. I loaded it up with three 36gb U160 10,000 RPM SCSI disks in a RAID 5 array, loaded up CentOS 5 and off I went.
Once it was at the data centre, I wanted to monitor the system a bit. This meant I wanted to keep track of the temperatures, fan speeds, and other information.
To do this, I needed to install IPMI support.
Install IPMI
yum install OpenIPMI OpenIPMI-tools -y
You may also need lm_sensors installed
Start up IPMI
service ipmi start
This should load the kernel modules needed for ipmi
Set up IPMI to start on boot
chkconfig ipmi on
Now we're off to testing IPMI!
Listing IPMI Device info
ipmitool -I open bmc info
Device ID : 0
Device Revision : 0
Firmware Revision : 1.81
IPMI Version : 1.0
Manufacturer ID : 674
Manufacturer Name : Unknown (0x2a2)
Product ID : 1 (0x0001)
Device Available : yes
Provides Device SDRs : yes
Additional Device Support :
Sensor Device
SDR Repository Device
SEL Device
FRU Inventory Device
IPMB Event Receiver
Aux Firmware Rev Info :
0x00
0x00
0x00
0x00
Listing all IPMI variables monitored within the server:
ipmitool -I open sdr list
CPU 1 | 39 degrees C | ok
CPU 2 | 43 degrees C | ok
CPU 3 | disabled | ns
CPU 4 | disabled | ns
CPU Planar | 44 degrees C | ok
Ambient | disabled | ns
CPU | 1.50 Volts | ok
CPU 2 | disabled | ns
CPU 3 | disabled | ns
CPU 4 | disabled | ns
+5 | 4.97 Volts | ok
+12 | 11.84 Volts | ok
+3.3 | 3.32 Volts | ok
Battery | 2.77 Volts | ok
+2.5 | 2.58 Volts | ok
NIC +2.5 | disabled | ns
NIC +1.8 | disabled | ns
MemCard A +2.5 | disabled | ns
MemCard B +2.5 | disabled | ns
MemCard A +1.25 | disabled | ns
MemCard B +1.25 | disabled | ns
Cover Intrusion | 0x00 | ok
Fan Control | 0x2d | ok
Fan 1 | 7920 RPM | ok
Fan 2 | 7920 RPM | ok
Fan 3 | 7920 RPM | ok
Fan 4 | 7920 RPM | ok
Fan 5 | 7440 RPM | ok
Fan 6 | 7800 RPM | ok
Pwr Supply Cntrl | Not Readable | ns
Power Supply | 0x01 | ok
AC Source | Not Readable | ns
AC Switch Ctrl | Not Readable | ns
AC Switch Status | Not Readable | ns
CPU VRM | Not Readable | ns
PCI HPlg Ctrl | Not Readable | ns
PCI HPlg Slot | Not Readable | ns
Memory Card A | disabled | ns
Memory Card B | disabled | ns
Power Button | 0x2d | ok
Bezel Intrusion | Not Readable | ns
ROMB Battery | disabled | ns
CPU | 0x2d | ok
Memory Card | Not Readable | ns
PDB | Not Readable | ns
Strapping ERR | Not Readable | ns
5V VRM | Not Readable | ns
+3.3 Aux | disabled | ns
CPU Termination | disabled | ns
ROMB Battery | 4.23 Volts | ok
Chassis Identify | 0x2d | ok
I/O Planar | disabled | ns
PSDB | 39 degrees C | ok
Pwr Supply Type | 0x00 | ok
NIC +1.5 | disabled | ns
+3.3 PCI | disabled | ns
+1.8 | disabled | ns
CPU IERR | Not Readable | ns
Event Logging | Not Readable | ns
Fan 7 | disabled | ns
+1.5 | disabled | ns
OS Watchdog | Not Readable | ns
BP 5V | disabled | ns
BP 12V | 7.21 Volts | ok
BP 3.3V | disabled | ns
Term Pwr SCSI A | 5.62 Volts | ok
Term Pwr SCSI B | disabled | ns
BP Bottom Temp | disabled | ns
BP Top Temp | 43 degrees C | ok
SCSI A Con | 0xc0 | ok
SCSI B Con | Not Readable | ns
Drive | 0x00 | ok
Drive | 0x00 | ok
Drive | 0x00 | ok
Drive | 0x00 | ok
SAF-TE | 0x01 | ok
SAF-TE | 0x01 | ok
Status | Not Readable | ns
Status | Not Readable | ns
PBay BP 5V | disabled | ns
PBay BP 12V | 7.21 Volts | ok
PBay BP 3.3V | disabled | ns
PBay TPwr SCSI A | 5.65 Volts | ok
PBay TPwr SCSI B | disabled | ns
PBay BP Btm Temp | disabled | ns
PBay BP Top Temp | 43 degrees C | ok
PBay SCSI A Con | 0xc0 | ok
PBay SCSI B Con | Not Readable | ns
PBay Drive | 0x00 | ok
PBay Drive | 0x00 | ok
PBay SAF-TE | 0x01 | ok
PBay Status | Not Readable | ns
RAC TEMP | 167 degrees C | ok
RAC BATT VOLT | 4.05 Volts | ok
RAC EXTPWR VOLT | disabled | ns
RAC +12PCI VOLT | disabled | ns
RAC -12PCI VOLT | -11.87 Volts | ok
RAC +5PCI VOLT | disabled | ns
RAC AUXPCI VOLT | 2.00 Volts | ok
Further reading:
http://www.hollenback.net/index.php/LinuxServerManagementIpmi
http://lonesysadmin.net/2007/06/21/how-to-configure-ipmi-on-a-dell-power...