Sysop Blog

Back Plates keep more then just dust out

I received a call from a old co-worker today needing help to trouble shoot a T1 customer of his. The line kept bouncing up and down. The telco checked the line and said it looked fine up to the CSU/DSU. After talking to him for a few minutes and him explaining the situation to me I told him we should replace the router. I told him to hook it up to a console so I could copy the config off of it. During this conversation he kept telling me how there was rat crap all over the place and how bad the place smelt. When he got the router to a console that I could log into I saw the output that shows below. He also told me that the router was missing one of the WIC card slots and rat crap had actually poured out of the router when he tipped on its back.  Rats in the router anyone?

Copyright (c) 1999 by cisco Systems, Inc.
C1700 platform with 32768 Kbytes of main memory

program load complete, entry point: 0×80008000, size: 0×7ee488
Self decompressing the image : #################################################
################################################################################
######## [OK]

Smart Init is enabled
smart init is sizing iomem
ID            MEMORY_REQ         TYPE
MainBoard       0X00027A80 1720
0X000F3BB0 public buffer pools
0X00211000 public particle pools
0X0002          0X0000DC00 Card in slot 1
TOTAL:          0X0033A230

If any of the above Memory Requirements are
“UNKNOWN”, you may be using an unsupported
configuration or there is a software problem and
system operation may be compromised.

Memory required: 36400364 bytes, Memory available: 33554432 bytes
INSUFFICIENT MEMORY TO LOAD IMAGE!
The difference between Memory required and Memory available is the additional
memory you need to run the router. Alternatel
may cause the memory requirement to decrease, and may allow you to at least
use the router with reduced functionality until you get more memory.

*** System received a Software forced crash ***
signal= 0×17, code= 0×200, context= 0×8000620c
PC = 0xffffffff, Vector = 0×200, SP = 0xffffffff

System Bootstrap, Version 12.0(3)T, RELEASE SOFTWARE (fc1)
Copyright (c) 1999 by cisco Systems, Inc.
C1700 platform with 32768 Kbytes of main memory

$$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $
$$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $
*** line too large ***

$$  large ***

monitor: command “large” not found
$$ und

monitor: command “und” not found
$$ not found

monitor: command “not” not found
$$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $$
$$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $$ $
*** line too large ***

AND it continues to repeat from there….

Stay tuned for pictures of dead rats or whatever we may find inside the router.

When perl and rpm don’t get along

Sometimes when building rpm packages you will get an rpm that requires a file that it already contains. This seems pretty lame (which it is) but here is an example and a workaround.

Building a package for freepbx we see this output:

rpmbuild -ba freepbx.spec
--------snip-----------------
Provides: config(freepbx) = 2.4.0-0
Requires(interp): /bin/sh /bin/sh
Requires(rpmlib): rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1
Requires(post): /bin/sh
Requires(postun): /bin/sh
Requires: /bin/bash /usr/bin/env /usr/bin/perl /usr/bin/php config(freepbx) = 2.4.0-0 perl(DBI) perl(FindBin) perl(retrieve_parse_amportal_conf.pl)

The file we don’t want is: “perl(retrieve_parse_amportal_conf.pl)”

What rpmbuild does is go through the list of files and run “ldd” against all executables to find required libraries.
It also goes through each perl file and looks for “use / require” flags to pull out required perl modules.
When a developer does a legitimate thing like ‘require “retrieve_parse_amportal_conf.pl“‘ to include functions and such into their program, rpmbuild sees that this file is needed adds it too it’s list of required files.

Now rpmbuild also goes through and looks for what packages/files perl programs provide. It does this by scanning through the files and looking for “package“. If you just have an include file with functions, you don’t have a complete module and won’t have the package statement either. Thus rpmbuild will never see your file provides itself! Fortunatly there are a couple work arounds.

In your perl file that rpmbuild is requiring you can define ‘ our $RPM_Provides = “yourfilename.pl” ‘. rpmbuild will pick this up and happily add it to the provided file list. The other method is slightly more complicated but works well if you don’t want to patch the source code.

In your rpm spec file under the %prep section after the %setup add the following code:

cat << \EOF > %{name}-req
#!/bin/sh
%{__perl_requires} $* |\
sed -e '/perl(yourperlfile.pl)/d'
EOF
%define __perl_requires %{_builddir}/%{name}-%{version}/%{name}-req
chmod 755 %{__perl_requires}


Where yourperlfile.pl is the file you want to exclude from the rpm requires check.
This should make your rpm build hapily and exlude that file from the requires check.

If you want to see the actuall files rpmbuild runs take a look at:
/usr/lib/rpm/perl.prov
and
/usr/lib/rpm/perl.req

Tags: exclude perl from requires, perl, requires included file, rpm requires, rpmbuild

3Ware Sucks!

FYI Im not going to provide benchmarks or anything remotely useful here.

3Ware raid controllers suck when it comes to random IO r/w. I recently installed a new mail delivery server with 15 1TB Hitachi hard drives and a 3ware 9650SE-24M8. Raid 6 for the mail spool raid 10 for the mail indexes and raid 1 for / and everything else.

I always heard that Random IO sucked on these cards from other colleagues but brushed it off thinking I could live with less then optimal disk r/w throughput. Little did I know it was SO bad that the machine wasn’t even usable. I/O wait was through the roof! Like over 90%! Waiting to write mail to mail spools was timing out from locks that were still being written. It was REALLY BAD!

I quickly tore down the Hardware Raid and ran the raid 6 and raid 10 JBOD on the 3ware controller with software raid with the same I/O mail load. I/O wait is never over 10% now.

Note: 3Ware cards are pretty solid cards with nice management. They just have some random i/o issues to work out. Their support is not that helpful either on the issue. I turned all the nobs they suggested with little or no results. The cards seem to be tuned and geared towards high sequential I/O. And they do work great for that. 3ware will probably never fix the problem as long as people keep buying them though.

This was done on Centos 5 with dovecot and postfix. Was tested w/ same results on FreeBSD.

BTW: Munin is pretty nice! Thanks to dforbes for turning me on to it.

Filtering WebServer Spam

Lately I have been tasked with filtering outgoing email coming from our webcluster. Why does it seem that nobody has figured out how to effectively filter outgoing email from webservers? Every solution seems to fall flat. The best I have seen so far came from a custom plugin I wrote for Spam Assassin that uses Akismet (Sorry if I was pounding on your servers guys) to do the filtering. The Akismet filter worked probably about 50% or so better then the commercial cloud mark plugin that I tried.  Not to mention it caught way fewer false positives then cloud mark. I wish I had my notes still cloud marks false positive rate was over 25% if I remember correctly. This was with all the Spam Assassin rules turned off by the way for testing on cloud mark and akismet.

My current setup is a sendmail wrapper that adds extra environment variables for outgoing email to use for filtering purposes and also does some script logging. This was pretty easy and I can post if anyone is interested. The rather difficult thing was making sendmail auto add the headers for me based on the environment variables since I didn’t want to manually alter the email. Ill post the sendmail magic here later on.

I have a VOIP project im working on right now that has pulled me away from this but when I get back to it I will post more info on what im doing now.