Getting Ahead with ESXi on Headless System

A headless system is a system that can be operated without a monitor, keyboard or mouse, the primary interface is a single serial port – or dual if you are lucky enough as me with the ATCA Blade Servers.  With VMware’s ESXi leading in Server Virtualization space does support the operation on Headless systems but with a few difficulties. For example VMware documentation says that ESXi will detect Headless System automatically but that is not the case on ATCA Systems.In my upcoming post i will explain how to do a scripted installation in case of Headless systems. 

ESXi supports four serial port modes. Please note only COM1 & COM2 are supported ,USB & PCI/PCIe based serial ports are not supported for redirection.

  • Logging mode – Logging mode is the default mode in a debug build. Logging mode sends the vmkernel.log over the serial port.
  • GDB mode – Use GDB mode for dedicated debugging.
  • Shell mode – Shell mode is the shell port access, which is similar to SSH.
  • DCUI mode – DCUI mode is a Direct Console User Interface. This is the user interface that is displayed when you boot ESXi using a monitor.

ESXi supports dynamically switching between four different serial port modes.
ESXi supports serial mode dynamic switching in order to provide maximum platform flexibility, and to allow debugging and supportability in the field.

ESXi includes a unique keystroke sequence that allows dynamic serial mode switching.
Dynamic Switching Keytrokes

Once the correct keystroke sequence is entered, the system switches the serial port to the desired mode. Note that these key strokes are after you access the system using serial port. Also note that once you enter the GDB Mode then the other keystrokes will not work, need to change the setting on CLI only as explained below.

  • Logging mode: Ctrl+G, Ctrl+B, 1
  • Shell mode: Ctrl+G, Ctrl+B, 2
  • DCUI mode: Ctrl+G, Ctrl+B, 3
  • GDB mode: Ctrl+G, Ctrl+B, ?

In my case after the first boot after installation ESXi could not redirect the DCUI on the Serial port. So I logged into the system, enabled SSH and then was able to redirect the serial modes over COM1/2. esxcfg-advcfg to set the current serial mode. Caveat is that we need to know the current mode of operation which needs to be set to none before setting the required serial mode of operation. My personal preference is to sett all modes to none before setting the required mode.

  • Logging mode: esxcfg-advcfg -s com1 /Misc/LogPort
  • Shell mode: esxcfg-advcfg -s com1 /Misc/ShellPort
  • DCUI mode: esxcfg-advcfg -s com1 /Misc/ConsolePort
  • GDB mode: esxcfg-advcfg -s com1 /Misc/GDBPort

Example: Example
If the serial mode is set to logging mode, I would do it by executing following commands (on the SSH Window)

  1. §. > esxcfg-advcfg –s none /Misc/LogPort
  2. §. > esxcfg-advcfg –s none /Misc/ShellPort
  3. §. > esxcfg-advcfg –s none /Misc/ConsolePort
  4. §. > esxcfg-advcfg –s none /Misc/GDBPort
  5. §. > esxcfg-advcfg –s com1 /Misc/ConsolePort (to redirect it to COM1)

For Operation of DCUI Following Shortcuts/Alternative Key Combinations are available, These are very useful in case some serial clients do not support the use of Function keys. Fortunately i use Putty as my preferred Client, it supports the use of function keys.

  • Esc + 1 –> F1
  • Esc + 2 –> F2
  • Esc + 3 –> F3
  • Esc + 4 –> F4
  • Esc + 5 –> F5
  • Esc + 6 –> F6
  • Esc + 7 –> F7
  • Esc + 8 –> F8
  • Esc + 9 –> F9
  • Esc + 0 –> F10
  • Esc + ! –> F11
  • Esc + @ –> F12

This is all i got now, the source is of course the vmware documentation. Will come back with the post on scripted installation on headless system. All the best using the Headless system in your Environment.




About jagannaths

A technology professional, with hands-on experience on virtualization (VMware) , wired and wireless (Cellular - LTE) networking infrastructure.
This entry was posted in Virtualization, VMWare Ops. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s