Note: this is unedited, stream-of-consciousness babble, containing profanity, a bunch of possibly-incorrect assertions, and a clear Emacs-over-vi bias. I'm frustrated, I need to vent, and that's life. My friend Mark asked for the gory details of my server problems, and it felt so good to spew about them that I thought I'd do so really publicly.

Where to begin?

Background:

I host paulroub.com and openmikes.org at jumpline.net. Various aliased domains are also involved:

        roub.net = paulroub.com
        paulraub.com = paulroub.com     
            (nobody ever spells my name right, so I give up)

        openmikes.info = openmikes.org
        openmics.org = openmikes.org
        openmics.info = openmikes.org

Notice, in particular, that my main email address (paul@roub.net) lives in a n aliased domain. Big deal, right? Ha-ha.

This was all hosted on a couple of RedHat Linux servers, in a typical virtual-domain setup -- my files all lived under /home/paulroub, etc. I had local procmail support, emacs, gcc and other needed tools were there, pretty thorough Perl setup. Life was good.

The Upgrade:

About a week ago, I get a notice that in a couple of days they'll be doing this big server upgrade, moving to Sphera virtual private servers, among other things. OK, that shouldn't be too bad -- having the appearance of my own machine, a bit more control perhaps. They've always been good w/support before, so I figure any problems will be solved quickly, and they must've tested the shit out of all this, right?

Then came Thursday morning.

Thursday would've sucked anyway -- I'd been up until 4 working, slamming things together for a customer demo my boss was doing. This had taken forever because my two Windows machines at home are running 98 or ME, and crash ALL THE TIME when running Delphi. XP and a second hard drive have since been ordered for the main box, soon it will dual-boot (games and Digi 001 still need the toy OS). And yes, I do prefer to use Linux and OpenBSD at home, but my working life revolves around Windows development.

Plus the cable (and hence cable modem) had gone out, so I had to send the program files via phone line. Yee-hah.

So I wake up to check for mail from my boss, and everything has gone to hell.

All my mail account passwords had changed. OK, check the support web site, turns out they've helpfully reset them all to my login password.

Get the passwords fixed, re-download all mail (expected that, new server) -- hey, how come it's all two weeks old? Hmm.

Check database-related pages on both sites. Expect stuff to fail. It does -- new database passwords, etc. I'll just SSH in to openmikes.org and start fixing things.

Oh fuck, no SSH. Turns out that "standard plan" accounts no longer have that. Great.

Edit the appropriate include files via FTP. Still not working. Hey, where'd my /include and /data directories go? Gone. Cool. Restore from my backups.

Much better. Hey, where are my recently-added listings? And page fixes made in the last two weeks? Hmm.

Grab my own backups (done nightly from a Linux box at home), build a Perl CGI script to restore the latest data (since I can't SSH in and use MySQL directly).

Better. Start uploading web page backups to both sites, things are improving.

Hey, how come certain pages still don't work? All Perl pages -- the PHP stuff is fine. Well, let's SSH into paulroub.com (Plus Plan, still have that don't-call-it-a-"feature"-it's-a-basic-goddamn-necessity) and test the scripts locally.

"Can't find module DBI". Hmm. When was the last time you saw a Perl installation without DBI? Must be misplaced.

Let's do a "locate". Not installed, fuck.
Try "find". Hey, no DBI.
Shit, download source, build it locally. No fucking C compiler!
Hey, this is RedHat, let's grab an RPM and install that. No rpm commands installed!
Gaaaahhhh!

OK, let's walk away from that. I'm on 3 hours sleep, and I'm too fucking tired.

Make sure mail is getting through. It's not! Cool! No bounces, though. Huh. Definitely nothing in the mail spool file, nor any activity in the sendmail logs. Are my MX records fucked up?

Uh-oh - what's this?

        [c:]nslookup
        > server ns.jumpline2000.net
        Default Server:  ns.jumpline2000.net
        Address:  66.84.5.1

        > roub.net
        Server:  ns.jumpline2000.net
        Address:  66.84.5.1

        Name:    roub.net
        Address:  209.239.55.116

        > paulroub.com
        Server:  ns.jumpline2000.net
        Address:  66.84.5.1

        Name:    paulroub.com
        Address:  66.84.14.39

Neat! roub.net, where people mail me, and paulroub.com, where I pick it up, are now different! As in, roub.net still goes to the old server. Can I still get there? Why yes I can -- that's where my mail is. Fun. So for the moment I tell Outlook Express and Evolution to check both servers.

Start firing off support requests -- what's wrong with aliases? what the fuck happened to SSH? why are you using two-week-old backups? why are they incomplete, anyway? where is Perl DBI? how 'bout RPM or a C compiler, you fucks?

That was the situation for a couple of days -- dribbles of support responses (they're a bit overwhelmed) -- DBI isn't supported (for no stated reason), but you can still use the old mysql.pm module -- except it isn't there either. Sorry about the old backups, we've restored again from newer ones. Sorry for thereby blowing off all your fixes from the last couple of days.

Nice email-to-all from the Jumpline's Director of Operations -- sorry for the "unforeseen problems". Yeah, who could foresee that old, incomplete backups; missing libraries; out-of-date aliases; etc. could ever cause trouble,

Then, Saturday, the old IP address went completely dead. Fine, except roub.net still points there. Sunday, they say aliases will be fixed "within 48 hours". Yeah, 'cause that kind of thing can't possibly be automated. Maybe the Perl modules they'd need to do it are missing.

Meanwhile, who knows what is happening to my mail?

I give up, buy a year of DNS from Dotster, and point roub.net at the new address. Then I get to go edit all sorts of sendmail configs on the new box so it won't just reject the mail as "relayed".

As of this morning, the new DNS seems to be getting out to the world, and hopefully things will be better.

Still no apparent intention of fixing the DBI problem (question -- what's the use of offering a database if the web's most-popular scripting language can't use it?); they have "grandfathered" us old Standard Plan guys, however, so I have SSH access back to openmikes.org.

So most of openmikes.org works, except the part where you search near a city -- that relies on some Perl DB code. I may be able to rewrite it in PHP, but I don't know if those particular modules are available.

Bunches of things are broken on paulroub.com, 'cause it's mostly Perl scripted. Comments pages, gig listings. Nothing important since I don't have any gigs right now.

Oh, and as of today they still haven't updated DNS for the aliases. Wanna bet they break my hand-tweaked sendmail config when they do?

p.s. Oh, and no fucking emacs, either. pico and vi.

p.p.s. vi!!! gaaaaaahhhh!