The other day we had a clustered server freak out on us. On a four node cluster, one instance appeared to be randomly failing over constantly, never staying in one spot long enough to really do anything. It was maddening, it was frustrating, and it was really hard to even catch up to it before it moved to another node. It also housed the database for a fairly important application.
Eventually I got the instance locked down long enough to get it to just sit still, and was able to look at the logs.
The weird thing was, while I was intently looking at the logs, I also found my mind wandering off in directions that I wouldn’t have expected – at least at that time in the morning, and with the pressure on so much – I mean, the server was just plain down and wouldn’t start. Something was really weird, there was more than a little urgency to get things fixed, and yet my mind was wandering into places it just shouldn’t have been going right then.
Let’s see what happen – and there again… I’m looking at problems software is causing, and ironically, it was – oh, how to put this delicately – another kind of ‘software’ that had my mind wandering.
And then I saw it – and I saw what my subconscious mind was trying to tell me… That the strange error I was seeing was a 42d.
Hmm… SQL would start, appear to run, I’d see a few lines in the errorlog – then boom, it’d failover, the one constant was that whole distracting 42d thing…
Hmm… I did some checking, and found that 42d is the hex equivalent of a logon failure.
What, so SQL can’t log on to the network itself?
No, wait, the Service Account can’t log on.
That’s a network account – and on a hunch I checked to see if for some reason that account was locked out – and sure enough, it was.
My colleagues over in AD land unlocked the account, I fired up SQL, and all was happy.
Obviously, your mileage may vary, you might be facing a totally different issue, but if you find yourself thinking about software that has nothing to do with computers, or a double barreled slingshot about the time you’re trying to troubleshoot a server down, take a look at the SQL errorlog and check with your friends over in AD (that would be ‘Active Directory’) land to see if that service account is locked out.
If it is, either unlock it or have your AD people unlock it, and you should be all set.
So remember, 42d can apply to two different types of software… It’ll be up to you to know which one to use where.