Fix uart boot errors on M1000e blade chassis

Somebody else figured it out in 2009, and I’m late to the party.

Basically, if you are running Red Hat Enterprise Linux (or one of it’s clone siblings) on a Dell M600 blade, you’ll need to modify the default BIOS settings in a non-intuitive way or you’ll get an error on every boot-up.

IRQ handler type mismatch for IRQ 12

Call Trace:
[] setup_irq+0x1b7/0x1cf
[] serial8250_interrupt+0x0/0xfe
[] request_irq+0xb0/0xd6
[] serial8250_startup+0x43d/0x5dc
[] uart_startup+0x76/0x16c
If you have no time to visit a doctor for consulting about the right dose can buy kamagra 100mg oral jelly. tadalafil canadian pharmacy Bad effects of certain medicines can generic levitra mastercard lead to sudden loss of vision. The results of the Hogan Development cipla viagra Survey are delivered in a proper manner. Yes, order generic cialis is trending and spreading around like a fire. [] uart_open+0x19e/0x427
[] tty_open+0x1e8/0x3b0
[] chrdev_open+0x14d/0x183
[] open_namei+0x2be/0x6ba
[] chrdev_open+0x0/0x183
[] __dentry_open+0xd9/0x1dc
[] do_filp_open+0x2a/0x38
[] do_sys_open+0x44/0xbe
[] tracesys+0xd5/0xdf

I knew this was a serial port issue from the second and fifth lines of the trace, but I couldn’t figure out why it was involving IRQ 12, which is normally used for SCSI cards or PS/2 mice.

udev under RHEL6 on a Dell m1000e blade server

The udev subsystem dynamically responds to udev events generated by the kernel and creates device nodes on the fly. This solves the old *nix problem of too many files in /dev that don’t have anything to do with the hardware you actually have. If you see a file named /dev/mouse, the system really truly has a mouse attached to it. Or at least that’s the plan. The kernel calls udev whenever you activate or deactivate any device, not just USB and PCMCIA cards, and this lets linux support absolutely any kind of hot-swappable hardware, even processors.

This is not the first dynamic hardware mapping scheme to hit linux – I think Red Hat EL4 used devfs and there was at least one other one before that. And people have tried to do it using the HAL daemon too.

The udev daemon reads “rules” that tell it how you want your devices named. This is so that different distributions that have historically used different names can all use udev, they just have their own rule sets. This also means that hardware invented tomorrow is trivial to add – just make a new rule.

In Red Hat Enterprise Linux v6, udev rules are read from multiple places in order to be maximally confusing and infuriating.

/lib/udev/rules.d is a “library” of standard rules that udev uses by default.

/etc/udev/rules.d is a place where you can put your own system-specific rules that will override the library. This is based on the names of the files, not the rules themselves.

/dev/.udev/rules.d is a secret hidden directory that seems to exist mostly to make you angry. I haven’t found any clear documentation of precedence of rule files found here. You have to experiment on a test system. The rules in here are usually referred to as “temporary rules” which may very well mean “permanent overrides” in normal English.

udev automatically detects changes to rules files, so changes take effect immediately without requiring udev to be restarted. However, the rules are not re-triggered automatically on already existing devices, so you might have learn to use the udevadm command (or just reboot) if you write any new rules.

But wait! There’s more! It was starting to make sense, and we can’t have that!

If you have a Dell system, a program called biosdevname will be invoked by rules in the library file /lib/udev/rules.d/71-biosdevname.rules whenever the system becomes aware that a network interface exists. The Dell white papers on the subject tell me that this program will rename your network devices and disk drives in a way that’s both counter-intuitive and inaccurately documented, but in reality it just renames the ethernet interfaces on a specific set of Dell systems.

Now, honestly, on some models of Dell server this behaviour makes a decent amount of sense; it labels the embedded ethernet ports on the system’s motherboard as “em1” and “em2” which is a good idea since Dell labeled the ports 1 & 2 (instead of the expected 0 and 1 – Real Computer Scientists can’t count past nine on their fingers). Then it will label any additional interfaces by the PCI bus positions, which is sort of uselessly informative but at least consistent.

On a Dell M1000e blade server, though, running biosdevname is worse than doing nothing. Instead of following their reasonably sane idea of making the labels on the chassis match the labels on the interfaces, the emn and pxpy naming scheme is mapped pseudo-randomly across the network ports. Net fabric A is arbitrarily designated as “embedded” (remember, on the M1000e none of the interfaces are embedded in the blades themselves), P3 is considered fabric B, and P1 is assigned to fabric C – even the order doesn’t match. So, on blade five, the interface names end up looking like this:

This generic cheap viagra circulation controls blood flow into the male sex organ and cause an erection. Men can feel much better erection if they neglect eating grapefruit or avoid drinking side effects cialis grape juice. This medication is available to you purchasing viagra in two low dosage strengths of 2.5mg and 5mg. If overmuch skin is taken away, or a more upright vector not employed, the order levitra human face can assume a pulled-back, “windswept” visual aspect. Blade Chassis Name -> Linux Interface Name
A1-port5 -> em1
A2-port5 -> em2
B1-port5 -> p3p1
B2-port5 -> p3p2
C1-port5 -> p1p1
C2-port5 -> p1p2

This is like purposeful obfuscation of location, and much worse than just nailing down eth0 through eth6 to specific MAC addresses (which RHEL6 would happily do if Dell’s biosdevname software wasn’t sticking its member into the pie).

The best solution I’ve found so far is to create a file named /etc/udev/rules.d/71-biosdevname.rules (thus overriding /lib/udev/rules.d/71-biosdevname.rules) and put a series of udev rules in it that associate specific names with specific MAC addresses. I map the actual Dell chassis labels like so:

Blade Chassis Name -> Linux Interface Name
A1-port5 -> ethA1p5
A2-port5 -> ethA2p5
B1-port5 -> ethB1p5
B2-port5 -> ethB2p5
C1-port5 -> aoeC1p5
C2-port5 -> aoeC2p5

With this setup, you will know which wire the telco guy unplugged as soon the system starts screaming about something gone wrong on ethA1p5 – it’s the Cat 6 ethernet wire in slot A1 port 5, obviously. You can also see which interfaces are intended to be used for our high-speed ATA-over-Ethernet SAN infrastructure.

That’s enough for today…