ipmi jumphost

This document describes setting up an “IPMI jumphost” using a server with an extra network port and a switch with VLAN capabilities in order to create a dedicated and isolated network for each IPMI device(BMC=Baseboard Management Controller). IPMI devices are accessed via ssh tunnels or dedicate commands, no general access is allowed.

Physical connections

  • The host that will be used as the “jump host” should have a spare network interface plugged into an assigned port on the non-public switch.
  • Each IPMI device will be plugged directly into an assigned port on the non-public switch, record the port assignments in a document somwhere so you don’t get confused.

WARNING: most servers with IPMI have “feature” where they automatically bridge a port to the BMC network, usually the eth0 host port. They do this so you only have to plug one cable into the switch, but this can result in putting the horribly insecure BMC on an accessible network where it can be hacked. Always use the server eth1 for the public network, use a dedicated cable in the BMC network port (even if it’s going to the same network as eth0), and if possible disable the BMC/eth0 bridging in the IPMI interface (not always possible with proprietary IPMI firmware).

Switch configuration

On the switch we create a separate VLAN for each IPMI device.

  • The jump host port is assigned “Tagged” to each of the IPMI VLANs. The jump host sends tagged packets to the switch and those tags are respected and sent to the appropriate ports (assuming that VLAN exists, otherwise it’s dropped).
  • Each IPMI port is assigned “Untagged” to it’s own VLAN, and “Forbid” for all the other VLANs. When the IPMI device sends an untagged packet (since we didn’t tell it to be on a VLAN) then that packet gets forwarded to the other ports in that VLAN, which is just the jump host. If an IPMI device attempts to tag a packet with a different VLAN number (like if it gets hacked), because we’ve set “Forbid” on all the ports, the packet will get dropped. The IPMI can only communicate with the jump host.

Some good info about VLANs in this HP switch manual PDF.

Jump host networking configuration

In 

/etc/network/interfaces
add a VLAN device for each IPMI network,

auto eth3.4
iface eth3.4 inet static
  address 192.168.239.1
  netmask 255.255.255.0
  # older releases will need the 'vlan' package installed and:
  #vlan-raw-device eth3

Jump host software configuration

Create a wrapper script

#!/bin/sh
# riseup ssh wrapper script 

USER=`whoami`

case "$SSH_ORIGINAL_COMMAND" in
  "power")
    /usr/sbin/ipmi-power -h ${USER}-ipmi -u ADMIN -P
    ;;
  "console")
    /usr/sbin/ipmi-console -h ${USER}-ipmi -u ADMIN -P
    ;;
  *)
    echo "Sorry. Only these commands are available to you: power, console"
    exit 1
    ;;
esac

For each user:
1) Create a user (we use ‘foo’ for example in the remaining steps)

2) In /etc/hosts create an entry named ${USER}-ipmi that points to the ipmi. For example

foo-ipmi  192.168.123.2

3) setup an authorized_keys file similar to this

command="/usr/local/bin/ipmi-wrapper",no-agent-forwarding,no-X11-forwarding,permitopen="foo-ipmi:22",permitopen="foo-ipmi:80",permitopen="foo-ipmi:443",permitopen="foo-ipmi:5900" ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGAAAAAAAAAyd7vhhZgbAMK2aJ4XAAAAAA+5i3d4gD9j foo@bar

This restricts the user to only run the wrapper command and also setup tunnels to the ipmi interfaces.

4) In sshd_config, enable port forwarding for the user

Match User foo
  AllowTcpForwarding yes

Client configuration

There are a few ways to access the ipmi

Use ipmi tools via ssh

alias ipmipower='ssh -p 4422 -t as250@magpie.riseup.net power'
alias ipmiconsole='ssh -p 4422 -t as250@magpie.riseup.net console'

Here are some handy aliases to setup tunnels

alias ipmihttp='echo "ipmi web at http://localhost:8080  Hit ctrl-c when done";ssh -p 4422 -L 8080:as250-ipmi:80 -N -T as250@magpie.riseup.net'
alias ipmihttps='echo "ipmi web at https://localhost:8443  Hit ctrl-c when done";ssh -p 4422 -L 8443:as250-ipmi:443 -N -T as250@magpie.riseup.net'
alias ipmivnc='echo "ipmi vnc at vnc://localhost:5900  Hit ctrl-c when done";ssh -p 4422 -L 5900:as250-ipmi:5900 -N -T as250@magpie.riseup.net'
alias ipmissh='echo "ipmi ssh at localhost:2222  Hit ctrl-c when done";ssh -p 4422 -L 2222:as250-ipmi:22 -N -T as250@magpie.riseup.net'