What the world needs now, is yet another blog

It was the day that Kosovo declared independence and we just happened to be in New York City that day. The city was a mess with people celebrating — which meant they all had to get home, wherever that was, at the end of the day.

Unfortunate timing, since my wife and I had to head home as well. We went to Penn Station to catch an Amtrak train south. The place was mobbed. My wife got a bit in front of me, trying to find a seat. We crammed ourselves into the train, no seats to be found. Yet another crappy train ride standing most of the way.In front of me was a black woman, next to her was an open seat with stuff piled on it. My wife was half-way down the car. I was about to ask the seated woman if the seat was vacant (I was pretty sure it was since there was only one seat check on it). Before doing so, the black woman asked. The asian woman who was seated said something to the black lady, got up, went forward in the car a bit, said something to my wife, came back with my wife, said “Sorry” to the black lady, and my wife sat down.

I figured the woman knew my wife, but I started wondering if that was true, so I sent my wife a text message “Do you know that lady?” She sent me a reply back “Never seen her before in my life.

“Wow, I had a hard time believing what I just witnessed. This Asian woman just simply didn’t want a black person sitting next to her, so she went down the aisle of the train and grabbed the first white person she saw.

At the next stop, as people got off the train, I made sure this lady got herself a seat.

Really pretty disturbing. I would have hoped this sort of thing never happens in this day and age. Guess I was being naive.

Went to Florida today for a conference. As I was going through security, the TSA agent told me “I hope you are coming back before Friday.” I’m like, “Er, no, Sunday… Why?”

So she proceeds to tell me my drivers license expires on Friday and I won’t be able to use it to come back after that.

YOU GOTTA BE KIDDING ME.

So I say I’ll worry about it Sunday and she says “Fine, don’t say I never warned you.”

While waiting for my flight I do some googling for info and find a few horror stories about TSA and expired drivers licenses.

Gessh… a few other accounts told about how doing this can land you on the TSA watch list too, even after you get your license renewed.

Fortunately, I have a valid passport back home and a understanding wife who is going to send it to me here.

Now I just have to pray the passport doesn’t get lost.

I recently got a cool Nokia N90 smart phone but it throws up SSL certificate warnings when used with GoDaddy issued SSL certificates. These certs are nice because they are only $30/year instead of several hundred for Verisign. Allegedly they work with all modern browsers. Sigh, not with the latest Symbian devices.

Not only does it barf on websites that use GoDaddy, but also with my imap and smtp ssl certs used for my email. A pain when reading or sending on the N90.

I ensured the intermediate certificate was installed in the server correctly but that didn’t work.

Then I found a posting on Nokia support forum that said to install the root certificate in the phone using its native browser (not Opera). Which I did, by browsing to GoDaddy’s SSL certificate page and downloading the “Valicert Root — DER Format” root certificate. Once clicked, the phone’s browser prompts to install the cert and that root cert works for email as well!

Now I’m happy again!

This should not have been as difficult to do as it was. Simple concept, add disks and hence LUNs to the scan, and you want to use them on your host without having to umount all disks and reload the scsi driver.

There’s a script to rescan the scsi bus from a website that automates the process, but it didn’t work on one host I tried it.

I found that since we use qlogic drivers there’s a command that has to be run first to let the driver know that a rescan is coming (I guess).


echo "scsi-qlascan" > /proc/scsi/driver-name/adapter-id

After running it, I got this goodness in fdisk for the new device!

Disk /dev/sdd: 2000.4 GB, 2000414572544 bytes


Just so I don’t forget, here’s a link to tell you how to stop Flash popups from Firefox.

In case that link goes dead…

1. Type about:config into the Firefox location bar.
2. Right-click on the page and select New and then Integer.
3. Name it privacy.popups.disable_from_plugins
4. Set the value to 2.

The main data volume on the system at work ran out of PEs due to the old default PE size of 4 MB and 64k of PEs being in the volume group. There’s no way to change that without recreating the volume group, and blowing away everything. So I have to move the 250 gigs of crap to somewhere else, then delete the old volume group, recreate it, then move everything back. Since moving that much data requires several hours and hence downtime to users, that’s not good.

So, the idea is to move the data around live. This requires making a RAID 1 mirror on top of lvm — not usually done this way. It also requires learning enough about mdadm to be able to create a mirror without destroying the good data (ouch). The process of intially creating the mirror will require the regular lvm device be unmounted and the RAID (md) device mounted. But after that is done, the syncing of the mirror can happen live. Hence downtime is kept to an absolute minimum.

There’s a real derth of information out there for doing this type of thing, and maybe for good reason. I’m also disappointed that this is quite a bit of hassle compared to how I used to do this sort of thing on a DG/UX (Data General) system back in the mid 90s.

1. Create a new phys volume on temporary disk
2. Create a new volume group on new disk with larger extent size (32 megs)
3. Mirror the old data and new data together. NOTE: This requires RAID on top of lvm, not lvm on top of RAID
4. Break the mirror after it syncs, leaving data on the temporary disk
5. Create a new PV on the original disk with 32 meg PEs
6. Extend the volume group onto the new PE
7. Move the PEs from the temporary disk to the original disk
8. Shrink the volume group so it lives just on the original disk

Of course, before doing any of this, testing and documentation is required, hence this post. Also, this procedure was used and tested on RHEL 3. For the love of your job, data, sanity, and all that is holy, do not trust what I am saying here. Use it as a guide with other docs and test it on non-production box with data you can afford to lose. Then before doing it on a production box, make sure you have safe backups — preferably multiples.

Some commands to use to get a feel for what is on your disk already

• pvscan
• vgdisplay -v vg_group_name

For purposes of the demo, the good data is on vg san0 and physical device /dev/sdd. The temporary disk will be named san2 on /dev/sdc.

Initial test set up

pvcreate /dev/sdc
vgcreate -s 16m san2 /dev/sdc # NOTE: Using 16m as extent size just for testing
lvcreate -l 60 --name newhome san2

RAID 1 (mirror) background

Idea will be to create a RAID 1 mirror consisting of the main data disk and a “missing” disk. This will start it off in a degraded mode. After that we can add the temporary (new) disk to it as a hot add and it wil rebuild the mirror onto it. After that is done, we mark as bad the original disk and leave ourselves with data on the new disk. Then we kill the mirror and go back to naked lvm.

Finding info on if you could create a mirror and preserve old data, let alone if it’s on top of lvm was difficult, as in, my googling could not find anyone who did it.

Creating the mirror

THE ORDER OF THE DEVICES BELOW IS CRITICAL. The first device will be the master and will copy to the second device.

mdadm --create /dev/md0 -l 1 -n 2 /dev/san0/home /dev/san2/newhome
mdadm --detail /dev/md0
mount /dev/md0 /mnt/home

If that bothers you (and it really should) you can create the array with just the original disk, mount it, then hot add the second disk into the array after you know the data is there. Also, if you’re paranoid after reading the various notes below and intend to fsck the md device before mounting, it’ll save you time if you do it like this since the fsck won’t be beating the disk at the same time as the mirror sync is going.

mdadm --create /dev/md0 -l 1 -n 2 /dev/san0/home missing
mdadm --detail /dev/md0
fsck -f /dev/md0
resize2fs /dev/md0
mount /dev/md0 /mnt/home
mdadm /dev/md0 -a /dev/san2/newhome

At this point the mirror should start rebuilding. Now it will say it has three devices and the missing will still be listed, but when it’s done the rebuild it only shows the two devices (but the total # of devices still says three). So maybe there’s a better way to do this, but it works.

Run the –detail option again to monitor when the drive is done rebuilding. At this point we should be able to break off the original disk and be left with our data on the new disk.

But before doing that, just in case, we should create an mdadm.conf file so if the array has to be stopped and restarted, we don’t have to scan for it.

echo -e 'DEVICE\t/dev/san0/home /dev/san2/newhome' >> /etc/mdadm.conf
mdadm --detail --scan >> /etc/mdadm.conf

Stopping / Restarting the array

mdadm --stop --scan # will stop the array
mdadm --detail --scan # will start the array iff mdadm.conf file above was appended with correct info
mdadm -Ac partitions -m 0 /dev/md0 # will start it back up (if there is no config file)


Breaking the mirror

After (and only after, verify first) that the mirror syncing is done (.e.g. mdadm --detail /dev/md0), the mirror can be broken and we’ll have two identical mountable ext3 lvm partitions. One on the original disk, one on the temporary (new) disk. Before killing the mirror, be sure to unmount any file systems using the mirror. When checking the rebuild status with mdadm, look for a line that says “Rebuild Status :” — if that line is not there, it’s rebuilt.

umount /dev/md0
mdadm --manage --set-faulty /dev/md0 /dev/san0/home
mdadm --manage --set-faulty /dev/md0 /dev/san2/newhome
mdadm --stop --scan

Once this is done we can actually mount and use the parition, but the raid “superblock” will still be associated with each partition and that could lead to it accidently being restarted. To really destroy the mirror, the raid superblock needs to be zeroed out.

mdadm --misc --zero-superblock /dev/san2/newhome
mdadm --misc --zero-superblock /dev/san0/home

IMPORTANT NOTES

NOTE: This procedure goes for minimize downtime at the risk of file system safety. Before starting this you really should fsck the disk you are going to mirror — offline. Also, read notes below.

NOTE: The set-faulty commands above aren’t really needed if you’re just going to zero the superblocks right away but if you don’t, you could technically mount up both partitions, make changes, then unmount them and restart the mirror with probably really bad effects.

NOTE: fsck will fail on the mirror device because the size of the device shrinks a wee bit. However, once the mirror is broken, a fsck should work and pass OK.

NOTE: When you import a disk with an ext3 file system into a mirror like this, the physical size of the partiton shrinks a bit because the md superblock is stored at the end of the partition. Hence while it will mount, it will fail a fsck and there’s the risk (I think) that real data might overwrite the raid superblock, especially if the partition is filled. Hence to be very safe, one should run ‘fsck -f /dev/md0 ; resize2fs /dev/md0′ against the new raid partition before remounting. However, once you do this, when you destroy the mirror the filesystem probably should be rezised out to original size again.

Reference websites

• Software RAID Howto
• Article on RAID

The quest has been given, to determine a good groupware package for where I work. The main component needed is a shared calendar interface. We currently use sendmail on a linux box as an MTA, uw-imap as an imap server, and mailboxes are stored in “mbx” format. For mail clients, we allow any imap client and provide a webmail client from the fine folks at horde.org.

The current version of horde we use has a calendar that is private only. The new version provides many groupware features including shared calendars, however, we are going to look at other groupware products as well.

It’s been basically narrowed down to either Microsoft Exchange or the latest set of Horde apps. One of my techs has set up a test exchange environment and to say it’s bloated is an understatement. Now granted it does a lot, but everything I hear is that most organizations hardly use all of Exchange features. There’s also a huge hardware cost. We do 40,000 email accounts on one dual-xeon 3 ghz box runnign Linux. Apparently we need a farm of several exchange servers to handle that kind of a load. Even only giving 10% of our users exchange would require at least four servers (two front-end and two back-end). Ouch.

Regardless, I have no religion about either platform, as long as the resources are provided to adequately support the chosen platform.

So this blog entry is mainly a place holder to jot down notes and experiences and is not going to be complete. So don’t consider this a formal case study or implementation guide. It will also focus primarily on horde since another tech has done Exchange. Hence it may seem biased towards that product.

NOTE: This is all done on Redhat Enterprise Linux (RHEL) 4

I didn’t exactly like the default behavior of wordpress to send registered users to the dashboard after they log on, so I did a few google searches and found a faq page that looks like it’s out of date. But it got me close enough.

Trick was to go into the wp-login.php file and find the

case login:

… statement, then go down a few lines and change the

$redirect_to = 'wp-admin/'

… statement and change it to just

$redirect_to = '/'

… to get the behavior I wanted.

If your blog isn’t in the root of your domain then that redirect should be to where it sits.

My last Quiznos Meal

You are looking at the last meal I will ever buy at Quiznos. They did the unforgiveable, they switched from Coke products to that Pepsi sludge.

It’s no secret that both Pepsi and Coke work to get exclusive rights to provide their stuff in places like this, by providing “incentives” to the owners to switch. And of course, the people that run these places see dollar signs in their head and switch, thinking that most people don’t care less — and probably that is true — but not this person.

I can’t imagine what the Coke and Pepsi reps tell their customers when working a deal like this. Do they say “You won’t lose any customers because they don’t care?” If they say this, then they are admitting that their product is not so special.

Well, some people care. Nothing beats a nice cold Diet Coke. Diet Pepsi is horrible. It’s bad enough that it does influence my decision. There’s a Subway down the street, and by golly, they toast their stuff now, and they serve Coke products, so I’m switching. I even told the franchise owner I was switching after he told me he couldn’t sell Coke products due to the contract.

Now I’m sure Quiznos won’t care less that ole Weave is switching, and I’m sure they may pick up some diehard Pepsi losers, but that’s not the point. What happened to the customer is always right? What happened to giving the customer what they want? Why not provide them a choice?

WE DESERVE THE RIGHT TO CHOOSE OUR OWN BRAND OF FIZZY DRINK

I know places lose some revenue from doing exclusives. Is it more than the incentive, I don’t know. Is the loss of goodwill worth it? 7-Eleven certainly has both company’s products in their fountains, for example. And for a while, they were making a big deal about giving people a choice.

My employer “used” to have both brands available in our cafeteria, but they switched to an exclusive Pepsi arrangement a year or so ago. And you know what happened? They lose revenue cause of it. How can I make such a claim? Because from actual experience I know I and other Coke lovers now smuggle in our own supply. So instead of dropping $1.25 on a bottle of Diet Coke there a few times a day, I just bring in a 12 pack of my own. The company cafeteria operation now gets squat.

And if the pain of having to wash down my meatball sub with a Diet Pepsi wasn’t bad enough, to make matters worse, it wasn’t even an iTunes “free song” winner.

I’m not the only one who is hoping for this, of course, but my dream ticket for 2008 is Rice vs. Clinton. It would shake up even the most die-hard party faithful.

First of all, there’s the entire problem of a female being President. I remember many in 1984 refusing to vote for the Democrats because there was a woman on the ticket, Geraldine Ferrero. Well, now we could have a race with two women on the #1 spot in 2008. Sweet justice.

Then there’s the entire racism factor. Face it, there are a lot of people in “red states” who would never vote for anyone unless they are White. What will they do? Probably just stay home.

Now, as the icing on this dream cake, I read that Rice is pro-choice, albiet “mildly,” if there is such a thing.

‘In an interview with editors and reporters in the office of the editor in chief at The Washington Times, she said she would not want the government “forcing its views” on abortion.‘

Now you’d have all of the anti-abortion folks staying at home.

So, I guess this opinion makes me one of those raging liberals. I guess so, if you define a liberal as someone who doesn’t drink from the neocon koolaid. But in reality, I’m pro government gridlock.

I most certainly do not want to see any political party in control (or defacto control) of all three branches of government. When that’s the case, things get done, and the definition of things “getting done” in D.C. is new laws being passed and money being spent. Each party is a big spender, they just have different priorities on what they like to spend money on. But one thing in common with all of them is that they spend money to get votes and keep power.

Government works best when it’s divided and arguing with each other. Only then, if something squeaks out of them, it’s usually a good thing (good example, NAFTA, bad example, DMCA). During true emergencies, like 9/11, when fast action is required, both parties do come together, so a divided country is not a weak country.

There are too many voters in this country that are blinded by idealogy. They will support their chosen political party and blast the “other side” no matter what happens. They dream of the day of a “one party system” without realizing that a multi-party system is the foundation of a strong democracy.

This is why, during the 2004 election, I voted mostly Republican (for local elections) and Democrat for President. It is also why I dream of a Rice vs. Clinton ticket. It will slice and dice the country different than red vs. blue and hopefully we’ll return to a country under political deadlock.