144 KiB
ATEN iKVM RFB Protocol Specification
Reverse-engineered from iKVM__V1.69.21.0x0 (Java), libiKVM64.so (native x86-64 client),
and ikvmserver (ARM32 BMC server binary from Supermicro X9 firmware SMT_X9_339).
Overview
The ATEN iKVM uses a modified RFB (Remote Framebuffer) protocol, version 3.8, for KVM (Keyboard, Video, Mouse) console access over TCP. It extends standard VNC/RFB with:
- ATEN-proprietary video encodings for BMC chipset-specific JPEG-like compression
- Extended keyboard/mouse messages (18 bytes instead of standard 8/6)
- Optional AES-128-CBC encryption for mouse events (keyboard is always cleartext)
- Privilege control, power management, and QoS extensions
- Hardware cursor position tracking
The protocol runs over a single TCP connection to the BMC's KVM port (default 5900,
configurable via the IPMI web interface -- stored in persistent storage at offset
0x1b42 from the PS shared memory region).
Architecture
┌──────────────────────────────────────────────────────┐
│ Java GUI (RemoteVideo, Viewer) │
│ - AWT/Swing keyboard/mouse capture │
│ - BufferedImage framebuffer (32bpp, max 1920x1200) │
│ - HID keycode translation (KeyMap) │
├──────────────────────────────────────────────────────┤
│ JNI Bridge │
│ - RMConnection: init, keepActive, checkValidUser │
│ - RemoteVideo: native methods (44 functions) │
├──────────────────────────────────────────────────────┤
│ Native Library (libiKVM64.so) │
│ - RFBProtocol: handshake, message dispatch │
│ - RFBScreen: video decode, framebuffer management │
│ - RFBKeyboard: key translation, send │
│ - RFBMouse: pointer events, send │
│ - RFBKMCryto: AES-128-CBC for input encryption │
│ - RFBPrivilege: session/privilege management │
│ - NtwStream: buffered network I/O │
│ - Video decoders: AST, Hermon, Yarkon, Pilot3 │
├──────────────────────────────────────────────────────┤
│ TCP Socket (TcpSocket) │
└──────────────────────────────────────────────────────┘
Connection Lifecycle
Client BMC (Server)
│ │
│──────── TCP connect (KVM port) ─────────>│ InitHandShake
│ │
│<──────── "RFB 003.008\n" ───────────────│ ProcVersion
│──────── "RFB 003.008\n" ───────────────>│
│ │
│<──── security_types_count(1) ───────────│ ProcSecurity
│<──── security_type(1) × N ─────────────│
│──── selected_type(1) ──────────────────>│
│ │
│<──── challenge(24) ────────────────────│ Authenticate
│──── username(24) ─────────────────────>│ (ATEN plaintext auth)
│──── password(24) ─────────────────────>│
│<──── auth_result(4) ───────────────────│ 0 = success
│ │
│──── client_init: shared_flag=0x00(1) ──>│ ProcClientInit
│ │
│<──── ServerInit (standard RFB) ────────│ ProcServerInit
│<──── ATEN extensions (session+config) ─│
│ │
│<──── PrivilegeInfo (type 0x39) ────────│ First server message
│ │
│──── FramebufferUpdateRequest ──────────>│ Main loop begins
│<──── FramebufferUpdate (type 0x00) ────│
│<──── CursorPosition (type 0x04) ───────│
│──── KeyEvent / PointerEvent ───────────>│
│<──── KeyboardInfo+MouseInfo (0x35) ────│
│<──── KeepAlive (type 0x16) ────────────│
│──── KeepAlive ACK ─────────────────────>│
│ ... (repeat) ... │
Handshake
The handshake is split into two phases:
- InitHandShake (
RFBProtocol::InitHandShakeat 0x00117fa0): Creates the NtwStream, establishes the TCP connection, then runs ProcVersion + ProcSecurity. - Authenticate (
RFBProtocol::Authenticateat 0x00118190): Performs the ATEN-specific username/password authentication, then runs ProcClientInit + ProcServerInit.
Version Exchange
Server sends exactly 12 bytes: RFB 003.008\n
Client reads this, parses RFB %03d.%03d\n, verifies major=3, minor=8, then
echoes back the identical 12-byte string.
Security Negotiation
Server sends:
┌──────────────────────┐
│ count: u8 │ Number of security types
├──────────────────────┤
│ types: u8[count] │ Available security type IDs
└──────────────────────┘
Client selects a security type (the implementation always picks the last one offered) and sends it back as a single byte.
The native code does not implement any security-type-specific logic (e.g., VNC DES challenge-response). It blindly selects the last type offered and sends it back. The actual authentication is ATEN-proprietary (see below), not standard RFB security types. The security type negotiation appears to be a formality to satisfy the RFB 3.8 protocol structure.
Server-side verification: The server (ikvmserver at FUN_0000f058) always
offers exactly one security type: 0x10 (16) — an ATEN proprietary type.
The server sends u8(1) (count=1) followed by u8(0x10) (type=16), then reads
back the client's selection. A client implementation should send 0x10 back.
ATEN Authentication
After security negotiation, the Authenticate function performs ATEN-specific
plaintext credential authentication. Important: On success, Authenticate
also runs ProcClientInit and ProcServerInit internally -- from the Java
caller's perspective, checkValidUser() returning true means the entire
handshake (including ClientInit, ServerInit, and ATEN privilege setup) is
already complete.
Client BMC (Server)
│ │
│<──── challenge: u8[24] ──────────────────│ Server sends 24-byte challenge
│ │
│──── username: u8[24] ───────────────────>│ Client sends username (null-padded)
│──── password: u8[24] ───────────────────>│ Client sends password (null-padded)
│ │
│<──── auth_result: u32 (big-endian) ─────│ 0 = success, non-zero = failure
│ │
│ If auth_result != 0: │
│<──── error_len: u32 (big-endian) ───────│
│<──── error_msg: u8[error_len] ──────────│
│ Connection should be closed. │
│ │
│ If auth_result == 0: │
│──── ClientInit ─────────────────────────>│ (sent inside Authenticate)
│<──── ServerInit ────────────────────────│ (read inside Authenticate)
│ │
Wire format:
┌─────────────────────────────────────────────┐
│ From server: │
│ challenge: u8[24] │ 24-byte challenge/nonce (discarded)
├─────────────────────────────────────────────┤
│ From client (sent atomically via buffered │
│ write -- both in one StreamWriteFlush): │
│ username: u8[24] │ Null-padded username
│ password: u8[24] │ Null-padded password
├─────────────────────────────────────────────┤
│ From server: │
│ auth_result: u32 (big-endian) │ 0 = success
│ If auth_result != 0: │
│ error_len: u32 (big-endian) │ Length of error message
│ error_msg: u8[error_len] │ Error description string
└─────────────────────────────────────────────┘
Verified from decompilation of RFBProtocol::Authenticate at 0x00118190:
The Userinfo_t struct (48 bytes, passed by value) contains username at
offset 0x00 (24 bytes) and password at offset 0x18 (24 bytes). The JNI
wrapper checkValidUser at 0x00119c60 copies credentials from Java
UserInfo fields using strcpy into local buffers (username: 24 bytes,
password: 96 bytes on stack but only 24 bytes sent on wire).
Security note: The username and password are sent in plaintext (no encryption, no hashing). The 24-byte challenge from the server is read into a local stack buffer and never referenced again -- it is genuinely discarded (confirmed by binary analysis: no DES, no XOR, no hash of the challenge). Maximum username length is 24 bytes; maximum password length is also 24 bytes (both truncated and null-padded to exactly 24 bytes). The error message on auth failure is read but also discarded by the native code -- Java shows a hardcoded "Authentication failed" dialog instead.
Server-side authentication (from ikvmserver at FUN_0000f37c / FUN_0000e638):
The server has two authentication backends, tried in order:
- Web session cookie: Reads
/tmp/sess_<username>— if the file exists and contains a valid session, the user is authenticated from the web UI session. Session file format:<auth_status> <username> <privilege> <display_name> <counter> - IPMI user database: Falls back to
UtilAuthUser(username, password)which checks the IPMI user table (supports LDAP vialibldap_client.soand RADIUS vialibradius_client.so). Requires privilege level >= Operator (level 4) on the LAN channel for KVM access.
After successful auth, the server sends u32(0) (success), reads 1 byte from
the client (the ClientInit shared_flag), then sends the ServerInit message.
Step-by-Step Wire Bytes for Full Handshake
For client implementors, here is the exact byte sequence from TCP connect through the first FramebufferUpdateRequest. All multi-byte integers are big-endian unless noted.
Step 1: TCP connect to BMC KVM port (e.g., 5900)
→ (no application data, just SYN/SYN-ACK/ACK)
Step 2: Read server RFB version (12 bytes)
← 52 46 42 20 30 30 33 2E 30 30 38 0A "RFB 003.008\n"
Parse: sscanf(buf, "RFB %03d.%03d\n", &major, &minor)
Verify: major == 3, minor == 8
Send same string back:
→ 52 46 42 20 30 30 33 2E 30 30 38 0A "RFB 003.008\n"
Step 3: Read security types
← [count:u8] always 01 (server offers exactly 1 type)
← [type:u8] × count always 10 (0x10 = ATEN proprietary auth)
Client MUST select one type and send it back (ATEN always picks the LAST offered):
→ [selected_type:u8] 10 (echo back 0x10)
Step 4: Authentication
← [challenge:24 bytes] Read and DISCARD (not used)
→ [username:24 bytes, null-padded] e.g. "admin\x00\x00...\x00"
→ [password:24 bytes, null-padded] e.g. "password\x00...\x00"
← [auth_result:u32] 00 00 00 00 = success
If auth_result != 0:
← [error_len:u32] Length of error string
← [error_msg:error_len bytes] Human-readable error
Connection should be closed.
Step 5: ClientInit
→ [shared_flag:u8] 00 (exclusive access)
Step 6: Read ServerInit (standard RFB, all discarded by native client code)
← [fb_width:u16] Server sends 0x01E0 (480) — swapped!
← [fb_height:u16] Server sends 0x0280 (640) — swapped!
← [pixel_format:16 bytes]
bits_per_pixel:u8 = 0x20 (32), depth:u8 = 0x18 (24),
big_endian:u8 = 0, true_colour:u8 = 1,
red_max:u16 = 0x00FF, green_max:u16 = 0x00FF, blue_max:u16 = 0x00FF,
red_shift:u8 = 16, green_shift:u8 = 8, blue_shift:u8 = 0,
padding:3 bytes
← [name_length:u32] = 16
← [name:name_length bytes] = "ATEN iKVM Server"
Step 7: Read ATEN ServerInit extensions
← [padding:4 bytes] (skip)
← [session_id:u32]
← [privilege_config:4 bytes] (read as 4 individual u8s)
The session_id and privilege_config are passed to ViewerConfig()
which stores them in globals (g_session_id, g_config) and calls
Java setViewerConfig(session_id, config[4]).
privilege_config byte meanings (verified from Java setViewerConfig AND server):
[0] = video_access (0 = denied -> show error and exit)
Server: controls types 3, 0x14, 0x15, 0x18, 0x19, 0x32, 0x33
[1] = keyboard_mouse (0 = disabled)
Server: controls types 4, 5, 7 (MouseSync), 8, 0x34, 0x35, 0x36, 0x37, 0x3a
[2] = kick_ability (0 = disable kick button)
Server: controls types 0x38, 0x39
[3] = virtual_storage (0 = disabled) — also controls power (0x1a) on server
Step 8: Server sends PrivilegeInfo (type 0x39) -- first server message
← 39
← [session_word_lo:u32]
← [session_word_hi:u32]
← [privilege_data:256 bytes] (read byte-by-byte)
The 256-byte payload is forwarded to Java privilegeCtrl(lo, hi, data).
It is NOT used for AES key setup -- the AES key is hardcoded.
Java interprets it based on session_word_hi:
hi=1: user joined (data = "<session_id> <username> <ip>")
hi=2: user left (data = session_id string)
hi=3: forced logout
hi=4: session kicked
hi=7: LED status sync (data[3] bits: b0=NumLock, b1=CapsLock, b2=ScrollLock)
Step 9: Client sends first FramebufferUpdateRequest
→ 03 00 00 00 00 00 00 00 [width:u16] [height:u16]
(type=0x03, incremental=0, x=0, y=0, width, height)
Step 10: Server sends first FramebufferUpdate (type 0x00)
← 00 type byte
← [skip:3 bytes] padding(1) + num_rects(2), skip all 3
← [x:u16] [y:u16] rectangle position
← [w:i16] [h:i16] rectangle size (can be negative, abs'd)
← [encoding:u32] ATEN encoding (0x57-0x61)
← [frame_number:u32] 1 for first frame, 0 thereafter
← [data_length:u32] compressed data size
← [data:data_length bytes] compressed frame data
frame_number will be 1 for the very first frame.
The encoding_type (0x57-0x61) determines which decoder to use.
Main loop: continue exchanging FramebufferUpdateRequests,
KeepAlive ACKs, KeyEvents, PointerEvents, and server messages.
ClientInit
Client sends a single byte:
┌──────────────────────┐
│ shared_flag: u8 │ 0x00 = exclusive access
└──────────────────────┘
The ATEN implementation always sends 0x00 (non-shared / exclusive).
ServerInit
Server sends the standard RFB ServerInit followed by ATEN-proprietary extensions.
Confirmed from decompilation of RFBProtocol::ProcServerInit at 0x00118040:
Standard RFB ServerInit (read and discarded by native code):
┌─────────────────────────────────────┐
│ fb_width: u16 (big-endian) │ StreamRead16 -- discarded
│ fb_height: u16 │ StreamRead16 -- discarded
│ pixel_format: 16 bytes │
│ bits_per_pixel: u8 │ StreamRead8 -- discarded
│ depth: u8 │ StreamRead8 -- discarded
│ big_endian: u8 │ StreamRead8 -- discarded
│ true_colour: u8 │ StreamRead8 -- discarded
│ red_max: u16 │ StreamRead16 -- discarded
│ green_max: u16 │ StreamRead16 -- discarded
│ blue_max: u16 │ StreamRead16 -- discarded
│ red_shift: u8 │ StreamRead8 -- discarded
│ green_shift: u8 │ StreamRead8 -- discarded
│ blue_shift: u8 │ StreamRead8 -- discarded
│ padding: 3 bytes │ StreamReadSkip(3) -- discarded
│ name_length: u32 │ StreamRead32 -> uVar1
│ name: u8[name_length] │ StreamRead(buf, uVar1) -- read into local_58
└─────────────────────────────────────┘
ATEN Extensions:
┌─────────────────────────────────────┐
│ padding: 4 bytes (skip) │ StreamReadSkip(4) -- discarded
│ session_id: u32 │ StreamRead32 -> uVar2
│ aes_key_seed: u8[4] │ StreamRead8 x4 -> local_5c..local_59
└─────────────────────────────────────┘
Important: The standard RFB ServerInit fields (width, height, pixel format) are all read and discarded. The native code does not use them -- the actual resolution is determined later from the first FramebufferUpdate.
Server-side note (WPCM450/Hermon): The server's SendServerInit at 0xf1ec
sends the width and height in swapped order: Write16BE(0x1E0) (480) for width
and Write16BE(0x280) (640) for height, when the actual default resolution is
640x480. This is a bug in the server firmware but has no practical impact since
the client discards these values. The session_id field is pthread_self() of
the session thread.
The ATEN extension fields are used as follows:
session_id(u32) is passed toRFBPrivilege::ViewerConfig(session_id, aes_key_seed_ptr)which stores it in globalg_session_idviastoreViewerConfig()aes_key_seed(4 bytes read individually) is also passed toViewerConfigwhich stores it in globalg_configviastrncpy(&g_config, ptr, 4)
After reading the ATEN extensions, ProcServerInit creates a new
RFBPrivilege object (0x120 bytes), calls ViewerConfig (vtable offset 0x28)
to store the session_id and key seed, then immediately calls the virtual
destructor (vtable offset 0x00) to destroy the temporary privilege object.
This temporary RFBPrivilege is separate from the one later created in
RMDesktop.
Note: The RFBPrivilege constructor allocates an RFBKMCryto object
(0x2110 bytes) at offset 0x118 within the privilege object. This crypto
instance is available but the SetCipherKey method is a no-op stub
(returns 1 without doing anything), confirming that the actual AES key is
entirely hardcoded, not derived from the server-provided key material.
After ServerInit, the server sends a ProcPrivilegeInfo message (type 0x39) containing session/privilege management data. The PrivilegeInfo received immediately after ServerInit has the same format as subsequent PrivilegeInfo messages -- there is no special "first-time" variant.
Privilege Info (from server, post-handshake)
Sent as server message type 0x39 (see PrivilegeInfo in Server-to-Client Messages).
Important: Despite the field name "aes_key_material" in early documentation,
the 256-byte payload is NOT used for AES key setup. The AES encryption key is
entirely hardcoded in the native code. The 256-byte payload is forwarded to
Java's privilegeCtrl callback, which interprets it as session management
data (user join/leave notifications, LED sync, forced logout) based on the
session_word_hi value.
┌─────────────────────────────────────┐
│ type: u8 = 0x39 │
│ session_word_lo: u32 │ Stored at this+0x10
│ session_word_hi: u32 │ Stored at this+0x14; action code
│ privilege_data: u8[256] │ Stored at this+0x18 (read byte-by-byte)
└─────────────────────────────────────┘
session_word_hi action codes (from Java privilegeCtrl):
1 = user joined session (data = "<session_id> <username> <ip>")
2 = user left session (data = session_id string)
3 = forced logout by another user
4 = session kicked (privilege denied)
7 = LED status sync (data[3]: bit0=NumLock, bit1=CapsLock, bit2=ScrollLock)
Confirmed from decompilation of RFBPrivilege::ProcPrivilegeInfo at 0x0011a120:
StreamRead32()-> store at this+0x10 (session_word_lo)StreamRead32()-> store at this+0x14 (session_word_hi)- Compute combined 64-bit value:
*(long*)(this+0x10)and compare against0x400000001(which issession_word_hi=4, session_word_lo=1in little-endian) - Call
SetThreadNormaleStart(result):- If
(lo, hi) == (0x00000001, 0x00000004): passes 0 (NOT normal start) - If
(lo, hi) != (0x00000001, 0x00000004): passes 1 (normal start)
- If
- Read 256 bytes one at a time (
StreamRead8()in a loop of 0x100 iterations) into this+0x18 through this+0x117 - Call virtual method at vtable offset 0x18 (which is
ExePrivilegeCtrl())
ExePrivilegeCtrl at 0x0011a1b0 simply calls privilegeControl(lo, hi, key_ptr),
which is a JNI helper at 0x00121580 that:
- Attaches to JVM thread
- Creates a Java byte array of 256 bytes
- Copies the 256-byte key material into the Java byte array
- Calls
RemoteVideo.privilegeCtrl(int lo, int hi, byte[] keyData)via JNI
The 256-byte key material is forwarded to Java for AES key setup. However, as noted in the AES encryption section, the actual AES key used for mouse event encryption is hardcoded in the binary (the NIST test vector), so the server-provided key material appears to be used only on the Java side for session management purposes, not for actual cryptographic operations in the native layer.
The SetThreadNormaleStart function at 0x00121640 calls the Java callback
RemoteVideo.setNormalStart(int flag) to signal whether this is a normal
session start or a reconnection/takeover.
Message Framing
All messages are big-endian unless noted. Each message begins with a 1-byte
type identifier. Multi-byte integers are big-endian (network byte order) as per
standard RFB, sent through the NtwStream::StreamWrite{8,16,32} helpers.
Server-to-Client Messages
The RFBProtocol::ProtocolHandler() at 0x001183d0 dispatches server messages
via a jump table at 0x00127790 supporting types 0x00 through 0x3C (61 entries).
The jump table was fully decoded from the binary. Non-default handlers exist for exactly 7 types:
| Type | Hex | Name | Handler | Description |
|---|---|---|---|---|
| 0 | 0x00 | FramebufferUpdate | RFBScreen::ScreenDecode (0x0010E7E8) | Video frame data |
| 4 | 0x04 | CursorPosition | RFBScreen::ScreenCursorPosProc (0x0010E7F8) | Hardware cursor position + shape |
| 22 | 0x16 | KeepAlive Request | (inline: reads 1 byte) | Server ping, expects ACK |
| 53 | 0x35 | KeyboardInfo + MouseInfo | RFBKeyboard::ProcKeyboardInfo (0x0011A720) then RFBMouse::ProcMouseInfo (0x001198C0) | Combined keyboard + mouse config |
| 55 | 0x37 | MouseInfo | RFBMouse::ProcMouseInfo (0x001198C0) | Mouse-only config update |
| 57 | 0x39 | PrivilegeInfo | RFBPrivilege::ProcPrivilegeInfo (0x0011A120) | Session/AES key material |
| 60 | 0x3C | GetScreenUILang | RFBScreen::GetScreenUILang (0x00118CB0) | OSD language config response |
All 54 other type values (1-3, 5-21, 23-52, 54, 56, 58-59, etc.) hit the default handler at 0x001183f5 which simply returns the type byte value to the caller (the Java runImage loop). The byte is not consumed or processed further by the native code.
Corrections from previous analysis: The earlier documentation listed types 0x31 (MouseInfo), 0x33 (KeyboardInfo), 0x35 (PrivilegeInfo), and 0x38 (ATEN Extended) as separate server message types. Binary analysis of the jump table reveals this was incorrect:
- Types 0x31 and 0x33 do NOT exist in the dispatch table
- Type 0x35 handles BOTH keyboard and mouse info in a single message
- Type 0x37 handles mouse info only (a separate mouse-only variant)
- Type 0x38 has NO server-to-client handler (it is client-to-server only: SendKickRequest)
- Type 0x39 is the actual PrivilegeInfo handler (not 0x35)
FramebufferUpdate (type 0x00)
The standard RFB FramebufferUpdate, but with ATEN-proprietary encoding.
Verified from decompilation of RFBScreen::ScreenDecode at 0x00119090 --
exact sequence of StreamRead calls after the type byte:
┌─────────────────────────────────────────────────┐
│ type: u8 = 0x00 │ (consumed by ProtocolHandler)
│ │
│ --- ScreenDecode reads the following --- │
│ │
│ padding: 1 byte (StreamReadSkip(3) reads │ Server sends: u8(0x00) padding
│ num_rects: u16 = 1 all 3 as a group) │ + u16(1) num_rects (always 1)
│ x_position: u16 (StreamRead16) │
│ y_position: u16 (StreamRead16) │
│ width: i16 (StreamRead16) │ Stored at RFBScreen+0x10 (as int)
│ height: i16 (StreamRead16) │ Stored at RFBScreen+0x14 (as int)
│ encoding_type: u32 (StreamRead32) │ Stored at RFBScreen+0x20 (0x57-0x61)
│ sub_encoding: u32 (StreamRead32) │ Stored at RFBScreen+0x50
│ │ Client uses as frame_number (1=first frame)
│ data_length: u32 (StreamRead32) │
│ data: u8[data_length] (StreamRead) │ Read into compressed_data_buf
└─────────────────────────────────────────────────┘
Total wire bytes after type: 3 + 2 + 2 + 2 + 2 + 4 + 4 + 4 + data_length = 23 + data_length.
Byte-by-byte wire dump (for a frame with encoding 0x58, 640x480):
00 type = FramebufferUpdate
XX XX XX 3 bytes skipped (padding + num_rects)
00 00 x_position = 0
00 00 y_position = 0
02 80 width = 640
01 E0 height = 480
00 00 00 58 encoding_type = 0x58 (AST2100+)
00 00 00 01 frame_number = 1 (first frame)
00 01 23 45 data_length = 0x12345 (example)
[data_length bytes] compressed frame data
Important for implementors: The ATEN server always sends exactly ONE
rectangle per FramebufferUpdate. The "3 bytes padding" is actually the
standard RFB padding(1) + number_of_rectangles(2) fields, but the native
client calls StreamReadSkip(3) -- it does not parse the rectangle count.
A client implementation should read these 3 bytes and can expect
num_rects to always be 1.
Frame processing logic (verified from decompilation):
- After reading encoding_type,
RMDecoder::GetDecoder()is called to create/retrieve the singleton decoder for this encoding type. - x, y, width, height, and encoding are stored into the decoder's info struct.
- frame_number is read:
- If
frame_number == 1AND this is the very first frame ever received (RFBScreen+0x50was 0): set resolution_changed flag, clear cursor flag.
- If
- data_length is read:
- If
data_length > 0ANDframe_number == 0: decode immediately by callingdecoder->Decode(UpdateBlocks_t&). Width/height are abs()'d before decode. - If
data_length > 0ANDframe_number != 0: set decoder's "deferred decode" flag (decoder offset 0x1c = 1). The actual decode happens on the next call with frame_number == 0. - If
data_length == 0: no data to read, returns immediately.
- If
KeepAlive Request (type 0x16)
┌───────────────────────────────┐
│ type: u8 = 0x16 │ (consumed by dispatcher)
│ status: u8 │ Read and discarded by native code
└───────────────────────────────┘
Total: 2 bytes.
The server sends type 0x16 periodically as a heartbeat. The ProtocolHandler's
inline handler at 0x00118488 reads one additional byte via StreamRead8() and
discards it (the return value is not stored or checked). The byte's value has
no significance to the native code -- it is consumed solely to advance the
stream position.
The Java side responds by calling sendKeepAliveAck(), which invokes
ProcAlive(1) via vtable offset 0x28. The ACK is always sent with value 1
(true). See KeepAlive ACK below.
CursorPosition (type 0x04)
Server message type 0x04 triggers RFBScreen::ScreenCursorPosProc() at
0x00118fb0 (thunked via 0x0010E7F8), which reads the hardware cursor
position and optionally the cursor image data.
Note: Type 0x04 is dual-purpose by direction -- server-to-client is CursorPosition, client-to-server is KeyEvent. They are separate streams.
┌─────────────────────────────────────────────┐
│ type: u8 = 0x04 │ (consumed by dispatcher)
│ cursor_x: u32 (BE) │ Cursor X position
│ cursor_y: u32 (BE) │ Cursor Y position
│ cursor_width: u32 (BE) │ Cursor image width
│ cursor_height: u32 (BE) │ Cursor image height
│ cursor_type: u32 (BE) │ 1 = cursor data follows
│ If cursor_type == 1: │
│ compositing_mode: u32 (BE) │ 0=XOR/overlay, 1=alpha-blend
│ cursor_pixels: u8[w * h * 2] │ 16bpp ARGB4444 cursor
└─────────────────────────────────────────────┘
Total: 21 bytes minimum (type + 5 x u32), plus variable cursor pixel data if cursor_type == 1. The cursor_type value is NOT stored to the object -- only the consequence matters (whether pixel data was read).
When cursor_type != 1: Only the position is updated. No pixel data is read or stored. The cursor_has_data flag (RFBScreen+0x55) is not set.
When cursor_type == 1: The compositing_mode (stored at RFBScreen+0x2070) determines how the cursor is composited onto the framebuffer:
Cursor pixel format: ARGB4444 (16 bits per pixel, big-endian):
- Bits 15-12: Alpha / control flags
- Bits 11-8: Blue (4 bits)
- Bits 7-4: Green (4 bits)
- Bits 3-0: Red (4 bits)
Compositing mode 0 (XOR/Overlay) -- ASTVideoDecoder::MixedCursor at 0x0010f990:
Each 16-bit cursor pixel is processed as follows:
- Bit 15 clear: Direct color. ARGB4444 expanded to 8-bit RGB channels (R = low nibble << 4, G = nibble 1 << 4, B = nibble 2 << 4)
- Bit 15 set, bit 14 clear: Transparent. Background pixel shows through.
- Bits 15 and 14 both set: XOR/Invert. Background pixel channels are bitwise inverted (~byte), creating a visible cursor on any background.
Compositing mode 1 (Alpha blend): True alpha blending using 4-bit alpha:
alpha = (pixel >> 12) & 0xF // 4-bit alpha (0-15)
R_out = (bg_R * (15 - alpha) + (pixel & 0xF) * 16 * alpha) / 15
G_out = (bg_G * (15 - alpha) + ((pixel>>4) & 0xF) * 16 * alpha) / 15
B_out = (bg_B * (15 - alpha) + ((pixel>>8) & 0xF) * 16 * alpha) / 15
Note: Only ASTVideoDecoder::MixedCursor implements compositing.
HermonVideoDecoder, YarkonVideoDecoder, and Pilot3VideoDecoder all
have stub implementations that return 1 without doing anything.
KeyboardInfo + MouseInfo (type 0x35)
Type 0x35 is a combined message that carries both keyboard and mouse configuration.
The client handler calls ProcKeyboardInfo() first, then falls through to ProcMouseInfo().
┌───────────────────────────────┐
│ type: u8 = 0x35 │ (consumed by dispatcher)
│ kb_encrypt: u8 │ Keyboard encryption flag (-> kb offset 0x18)
│ kb_type: u8 │ Keyboard type (-> kb offset 0x14)
│ mouse_encrypt:u8 │ Mouse encryption (0=clear, !0=AES) (-> mouse 0x18)
│ mouse_mode: u8 │ Mouse mode (-> mouse offset 0x14)
│ mouse_config: u8 │ Mouse additional config (-> mouse offset 0x1c)
└───────────────────────────────┘
Total: 6 bytes (client expectation).
Server-side discrepancy (Hermon/WPCM450 firmware): The server's GetKbdMouseInfo
handler at 0xeeac only sends 3 bytes total: [type=0x35] [info_byte_2] [info_byte_1].
It does NOT include the 3 mouse info bytes. The client expects 6 bytes but the server
sends only 3, which would cause a protocol desync if the client sends a type 0x35
request. In practice, this discrepancy may not manifest if the server's keyboard info
bytes happen to be parseable as valid mouse config, or if the client reads the mouse
bytes from a subsequent message. This appears to be a firmware bug specific to the
WPCM450/Hermon BMC. Other BMC firmware versions (AST, Pilot3) may implement the full
6-byte response correctly.
The ProcKeyboardInfo function (0x0011A720) reads:
- byte 1 (u8) -> stored at RFBKeyboard offset 0x18 (encryption_enabled)
- byte 2 (u8) -> stored at RFBKeyboard offset 0x14 (keyboard_type)
- Sets config_received flag at offset 0x10
The ProcMouseInfo function (0x001198C0) reads:
- byte 1 (u8) -> stored at RFBMouse offset 0x18 (encryption_enabled) Controls mouse event encryption in SendMouse (0=cleartext, non-zero=AES)
- byte 2 (u8) -> stored at RFBMouse offset 0x14 (mouse_mode)
- byte 3 (u8) -> stored at RFBMouse offset 0x1c (additional config)
- Sets config_received flag at offset 0x10
MouseInfo (type 0x37)
Type 0x37 is a mouse-only configuration update. It calls ProcMouseInfo()
without the keyboard info prefix.
┌───────────────────────────────┐
│ type: u8 = 0x37 │ (consumed by dispatcher)
│ mouse_encrypt:u8 │ Mouse encryption (0=clear, !0=AES) (-> offset 0x18)
│ mouse_mode: u8 │ Mouse mode (-> offset 0x14)
│ mouse_config: u8 │ Mouse additional config (-> offset 0x1c)
└───────────────────────────────┘
Total: 4 bytes. Uses the same ProcMouseInfo handler as type 0x35.
Confirmed from decompilation of RFBMouse::ProcMouseInfo at 0x001198c0:
StreamRead8()-> store at this+0x18 as uint (encryption_enabled)StreamRead8()-> store at this+0x14 as uint (mouse_mode)StreamRead8()-> store at this+0x1c as uint (additional config)- Set this[0x10] = 1 (config received flag)
Encryption control: The field at RFBMouse offset 0x18 is the encryption
enable flag. In SendMouse, the check is if (*(int*)(param_1 + 0x18) != 0)
-- when this value is non-zero, mouse events are AES-encrypted; when zero,
they are sent in cleartext. This is the first byte read by ProcMouseInfo,
which is the server telling the client whether to encrypt mouse events.
PrivilegeInfo (type 0x39)
┌─────────────────────────────────────┐
│ type: u8 = 0x39 │ (consumed by dispatcher)
│ session_word_lo: u32 │ Stored at RFBPrivilege+0x10
│ session_word_hi: u32 │ Stored at RFBPrivilege+0x14
│ aes_key_material: u8[256] │ Stored at RFBPrivilege+0x18..0x117
└─────────────────────────────────────┘
Total: 265 bytes. Format is identical whether received immediately after ServerInit or later during the session.
The handler (RFBPrivilege::ProcPrivilegeInfo at 0x0011A120):
StreamRead32()-> this+0x10 (session_word_lo)StreamRead32()-> this+0x14 (session_word_hi)- Compares the combined 64-bit value at this+0x10 against 0x400000001
(little-endian: lo=1, hi=4):
- If NOT equal: calls
SetThreadNormaleStart(1)(normal start) - If equal: calls
SetThreadNormaleStart(0)(not normal start)
- If NOT equal: calls
- Reads 256 bytes one-at-a-time via
StreamRead8()in a loop (0x100 iterations), storing each byte at this+0x18 through this+0x117 - Calls
ExePrivilegeCtrl()via vtable offset 0x18, which forwards (session_word_lo, session_word_hi, key_material_ptr) to Java via theprivilegeCtrl(int, int, byte[])JNI callback
The 256-byte key material is forwarded to Java for session/privilege management. The actual AES encryption key used in the native layer is hardcoded (see AES Encryption section).
GetScreenUILang (type 0x3C)
Server response to a client's ProcGetScreenUI request (client sends type 0x3C):
┌───────────────────────────────┐
│ type: u8 = 0x3C │ (consumed by dispatcher)
│ lang_id: u32 │ Language identifier
│ lang_sub: u32 │ Language sub-identifier
└───────────────────────────────┘
Total: 9 bytes.
The handler (RFBScreen::GetScreenUILang at 0x00118CB0) reads two u32 values
and passes them to Java via the getScreenUILangConf callback. Contains debug
printf statements ("doris getscreen ui lang", "doris %d %d").
Client-to-Server Messages
All 19 client-to-server message types, verified by searching every
StreamWrite8 call in the binary. Each function was decompiled from
/liblinux_x86_64__V1.0.5/libiKVM64.so-730126.
| Type | Hex | Name | Size | Sender Function (Address) | Description |
|---|---|---|---|---|---|
| 3 | 0x03 | FramebufferUpdateRequest | 10 | RFBScreen::ScreenUpdate (0x118850) | Request screen update |
| 4 | 0x04 | KeyEvent | 18 | RFBKeyboard::Sendkey (0x11a670) | Keyboard event (ATEN extended) |
| 5 | 0x05 | PointerEvent | 18 | RFBMouse::SendMouse (0x1194f0) | Mouse event (ATEN extended) |
| 7 | 0x07 | MouseSync | 3 | RFBMouse::MouseSync (0x119730) | Mouse synchronization |
| 8 | 0x08 | MouseReset | 2 | RFBMouse::MouseReset (0x119780) | Mouse device reset |
| 21 | 0x15 | ScreenSetPosition | 9 | RFBScreen::ScreenSetPosition (0x1189b0) | Set screen position |
| 22 | 0x16 | KeepAlive ACK | 2 | RFBProtocol::ProcAlive (0x118230) | Response to server keepalive |
| 23 | 0x17 | ScreenCalibration | 2 | RFBScreen::ScreenCalibration (0x118960) | Trigger screen recalibration |
| 25 | 0x19 | GetCursorPos | 1 | RFBScreen::GetCursorPos (0x118d60) | Request cursor position |
| 26 | 0x1A | SetPowerOnOff | 2 | RFBScreen::SetPowerOnOff (0x118900) | Power control |
| 50 | 0x32 | ScreenSetInfo | 5 | RFBScreen::ScreenSetInfo (0x118b80) | Set resolution |
| 53 | 0x35 | KeyboardUpdateInfo | 1 | RFBKeyboard::KeyboardUpdateInfo (0x11a610) | Request keyboard config refresh |
| 54 | 0x36 | MouseSetPT | 3 | RFBMouse::MouseSetPT (0x119820) | Set pointer type on server |
| 55 | 0x37 | MouseUpdateInfo | 1 | RFBMouse::MouseUpdateInfo (0x1197d0) | Request mouse config refresh |
| 56 | 0x38 | SendKickRequest | 73 | RFBPrivilege::SendKickRequest (0x11a1f0) | Privilege kick request |
| 58 | 0x3A | MouseHotPlug | 1 | RFBMouse::MouseHotPlug (0x119910) | Mouse hot-plug reset |
| 59 | 0x3B | QoS | 13 | RFBProtocol::ProcQos (0x118290) | Quality of Service params |
| 60 | 0x3C | GetScreenUI | 1 | RFBProtocol::ProcGetScreenUI (0x1183a0) | Request OSD language config |
| 61 | 0x3D | SetScreenUILang | 5-9 | RFBProtocol::ProcSetScreenUI (0x118320) | Set UI language |
FramebufferUpdateRequest (type 0x03)
Standard RFB FramebufferUpdateRequest:
┌─────────────────────────────────┐
│ type: u8 = 0x03 │
│ incremental: u8 │ 0 = full, 1 = incremental
│ x: u16 │
│ y: u16 │
│ width: u16 │
│ height: u16 │
└─────────────────────────────────┘
Total: 10 bytes.
KeyEvent (type 0x04) — ATEN Extended
IMPORTANT: Keyboard events are NEVER encrypted by the client. Unlike
PointerEvents, the Sendkey() function always sends keyboard events in
cleartext. The RFBKeyboard object does allocate an RFBKMCryto instance
(at this+0x20), but Sendkey() never calls it.
Server-side note: The server's KeyEvent handler at 0xf8c4 does check the
encryption flag byte and supports decrypting AES-encrypted keyboard events
(same AES-128-CBC scheme as mouse events). If the first byte after type is
non-zero, the server treats the remaining 16 bytes as AES ciphertext. However,
since the client never encrypts keyboard events, this code path is never
exercised in practice. A custom client could encrypt keyboard events for
additional security — the server will handle them correctly.
┌─────────────────────────────────┐
│ type: u8 = 0x04 │
│ padding: u8 = 0x00 │ Always zero (no encryption flag)
│ down_flag: u8 │ 0 = up, 1 = down
│ padding: 2 bytes (zero) │
│ keycode: u32 │ USB HID keycode
│ padding: 9 bytes (zero) │
└─────────────────────────────────┘
Total: 18 bytes. Always sent in cleartext.
The down_flag values:
0= key released (up)1= key pressed (down)
The keycode field contains USB HID keycodes. The Java client's
KeyMap.VKtoHID() maps Java VK codes to HID keycodes before passing them
to the native layer. The native processVK() tables documented below show
the intermediate X11 keysym representation used internally, but the BMC
firmware ultimately interprets the value field as a HID keycode.
Keyboard send flow (RFBKeyboard::Sendkey at 0x0011a670):
StreamWriteStart()- begin write batchStreamWrite8(0x04)- type byte- Clear the keyboard mode field at this+0x18 to 0
StreamWrite8(0x00)- padding byte (always 0, no encryption flag)StreamWrite8(down_flag)- key stateStreamWriteSkip(2)- 2 zero bytes paddingStreamWrite32(keycode)- the HID keycode valueStreamWriteSkip(9)- 9 zero bytes paddingStreamWriteFlush()- send
Lock key handling (in keyboardAction JNI at 0x00121000):
For CapsLock (VK 0x14), ScrollLock (VK 0x91), and NumLock (VK 0x90), the JNI
entry point reads the local X11 keyboard indicator state via XkbGetIndicatorState():
- Bit 0 (0x01) of indicator state = ScrollLock active
- Bit 1 (0x02) of indicator state = NumLock active
- Bit 2 (0x04) of indicator state = CapsLock active
(Verified from RemoteVideo_init at 0x0011fbd0: & 1 -> scrollLock,
& 2 -> numLock, & 4 -> capsLock)
If the corresponding lock is NOT currently active on the local machine, the VK code is modified: the low byte is OR'd with 0xFF to signal a toggle request. For example, CapsLock VK 0x14 becomes 0xFF14 when CapsLock is not active.
The three special VK codes checked are:
- VK 0x14 (CapsLock) -> checks
capsLock_status - VK 0x90 (NumLock) -> checks
numLock_status - VK 0x91 (ScrollLock) -> checks
scrollLock_status
After lock key processing, the modified VK code is passed through processVK()
which translates it to the final keysym sent on the wire.
Extended key handling (in keyboardAction JNI):
If the extendedKey flag is set in the KeyInfo_t struct, 0x100 is added to
both the VK code and the scan code fields. This distinguishes keys like
right-Ctrl (VK 0x111) from left-Ctrl (VK 0x11), or numpad-Enter (VK 0x10D)
from main Enter (VK 0x0D).
processVK Key Translation (RFBKeyboard::processVK at 0x0011a920)
The processVK() function uses three std::map<int, uint> lookup tables
populated in the RFBKeyboard constructor (0x0011abf0). It performs a
cascading lookup: Table 1, then Table 2 (with 0x100 toggle fallback), then
Table 3. The first non-zero result is used.
Lookup algorithm:
- If the
extendedKeyflag is set, add 0x100 to the VK code - Table 1 (this+0x28): Look up the VK code directly
- If found and value != 0, return value & 0xFFFF
- Table 2 (this+0x58): Look up the VK code (with extended flag)
- If not found, try with the 0x100 bit toggled (XOR 0x100)
- If found and value != 0, return value & 0xFFFF
- Table 3 (this+0x88): Look up the scan code
- If found and value != 0, return value & 0xFFFF
- If all lookups return 0, the key event is dropped (keysym = 0)
Special lock key handling in processVK:
Before the table lookups, if iStack0000000000000014 == 0 (action field is 0),
three special cases are checked:
- VK 0x14 (CapsLock): returns
~(-(capsLock_status == 0)) + 0x2FFE6- If CapsLock active: returns 0x2FFE5 (X11 Caps_Lock keysym)
- If not active: returns 0x2FFE6
- VK 0x90 (NumLock): returns
~(-(numLock_status == 0)) + 0x2FF80- If NumLock active: returns 0x2FF7F (X11 Num_Lock keysym)
- If not active: returns 0x2FF80
- VK 0x91 (ScrollLock): returns
~(-(scrollLock_status == 0)) + 0x2FF15- If ScrollLock active: returns 0x2FF14 (X11 Scroll_Lock keysym)
- If not active: returns 0x2FF15
Table 1 (this+0x28, 48 entries): Printable character VK codes
Direct mapping of Java VK codes to ASCII character codes:
| VK Code | Keysym | Key | VK Code | Keysym | Key |
|---|---|---|---|---|---|
| 0x20 | 0x20 | Space | 0x30-0x39 | 0x30-0x39 | 0-9 |
| 0x41-0x5A | 0x41-0x5A | A-Z | 0xBA | 0x3A | : (colon) |
| 0xBB | 0x2B | + (plus) | 0xBC | 0x2C | , (comma) |
| 0xBD | 0x5F | _ (underscore) | 0xBE | 0x2E | . (period) |
| 0xBF | 0x2F | / (slash) | 0xC0 | 0x60 | ` (backtick) |
| 0xDB | 0x5B | [ (bracket) | 0xDC | 0x5C | \ (backslash) |
| 0xDD | 0x5D | ] (bracket) | 0xDE | 0x22 | " (quote) |
For A-Z and 0-9, the VK code equals the keysym (identity mapping).
Table 2 (this+0x58, 89 entries): Function/control/navigation keys
Maps VK codes (with extended bit) to X11 keysyms:
| VK Code | Keysym | Key | VK Code | Keysym | Key |
|---|---|---|---|---|---|
| 0x08 | 0xFF08 | BackSpace | 0x09 | 0xFF09 | Tab |
| 0x0C | 0xFF0B | Clear | 0x0D | 0xFF0D | Return |
| 0x13 | 0xFF13 | Pause | 0x1B | 0xFF1B | Escape |
| 0x12E* | 0xFFFF | Delete | 0x124* | 0xFF50 | Home |
| 0x125* | 0xFF51 | Left | 0x126* | 0xFF52 | Up |
| 0x127* | 0xFF53 | Right | 0x128* | 0xFF54 | Down |
| 0x121* | 0xFF55 | Page_Up | 0x122* | 0xFF56 | Page_Down |
| 0x123* | 0xFF57 | End | 0x029 | 0xFF60 | Select |
| 0x02C | 0xFF61 | 0x02B | 0xFF62 | Execute | |
| 0x12D* | 0xFF63 | Insert | 0x02F | 0xFF6A | Help |
| 0x103* | 0xFF6B | Break | 0x70-0x87 | 0xFFBE-0xFFD5 | F1-F24 |
| 0x10D* | 0xFF8D | KP_Enter | 0x24 | 0xFF95 | KP_Home |
| 0x25 | 0xFF96 | KP_Left | 0x26 | 0xFF97 | KP_Up |
| 0x27 | 0xFF98 | KP_Right | 0x28 | 0xFF99 | KP_Down |
| 0x21 | 0xFF9A | KP_Page_Up | 0x22 | 0xFF9B | KP_Page_Down |
| 0x23 | 0xFF9C | KP_End | 0x0C | 0xFF9D | KP_Begin |
| 0x2D | 0xFF9E | KP_Insert | 0x2E | 0xFF9F | KP_Delete |
| 0x6A | 0xFFAA | KP_Multiply | 0x6B | 0xFFAB | KP_Add |
| 0x6C | 0xFFAC | KP_Separator | 0x6D | 0xFFAD | KP_Subtract |
| 0x6E | 0xFFAE | KP_Decimal | 0x16F* | 0xFFAF | KP_Divide |
| 0x60-0x69 | 0xFFB0-0xFFB9 | KP_0-KP_9 | |||
| 0x10 | 0xFFE1 | Shift_L | 0x10 | 0xFFE2 | Shift_R |
| 0x11 | 0xFFE3 | Control_L | 0x111* | 0xFFE4 | Control_R |
| 0x12 | 0xFFE9 | Alt_L | 0x112* | 0xFFEA | Alt_R |
| 0x5B | 0xFFEB | Super_L | 0x5C | 0xFFEC | Super_R |
| 0x5D | 0xFF67 | Menu | 0x19 | 0xFF21 | Kanji |
| 0x15 | 0xFF2E | Hangul |
Entries marked with * have the extended bit (0x100) set.
Note: Some VK codes appear twice with different keysyms (e.g., VK 0x10 maps
to both Shift_L 0xFFE1 and Shift_R 0xFFE2; VK 0x24 maps to both Home 0xFF95
and KP_Home). The std::map will keep only one entry per key, so the last
insert wins during constructor initialization. The non-extended variants
(0x24, 0x25, 0x26, 0x27, 0x28, etc.) thus map to the KP_* variants since
those are inserted after the navigation key versions.
Table 3 (this+0x88, 32 entries): Scan code based mapping
Maps raw scan codes to keysyms. This table handles multimedia keys, Japanese keyboard keys, and other special keys not covered by Tables 1-2:
| Scan Code | Keysym | Description | Scan Code | Keysym | Description |
|---|---|---|---|---|---|
| 0x15E* | 0x8181 | JP key | 0x15F* | 0x8282 | JP key |
| 0x163* | 0x8383 | JP key | 0x073 | 0x8787 | JP key |
| 0x070 | 0x8888 | JP Katakana/Hiragana | 0x07D | 0x8989 | JP Yen |
| 0x079 | 0x8A8A | JP Henkan | 0x07B | 0x8B8B | JP Muhenkan |
| 0x05C | 0x8C8C | JP key | 0x0F2 | 0x9090 | Hanja |
| 0x0F1 | 0x9191 | Hangeul | 0x078 | 0x9292 | JP key |
| 0x077 | 0x9393 | JP key | 0x076 | 0x9494 | JP key |
| 0x119* | 0x00B5 | Media Next Track | 0x110* | 0x00B6 | Media Prev Track |
| 0x124* | 0x00B7 | Media Stop | 0x122* | 0x00CD | Media Play/Pause |
| 0x120* | 0x00E2 | Volume Mute | 0x130* | 0x00E9 | Volume Up |
| 0x12E* | 0x00EA | Volume Down | 0x16D* | 0x0183 | Media Select |
| 0x16C* | 0x018A | 0x121* | 0x0192 | Calculator | |
| 0x16B* | 0x0194 | My Computer | 0x165* | 0x0221 | WWW Search |
| 0x132* | 0x0223 | WWW Home | 0x16A* | 0x0224 | WWW Back |
| 0x169* | 0x0225 | WWW Forward | 0x168* | 0x0226 | WWW Stop |
| 0x167* | 0x0227 | WWW Refresh | 0x166* | 0x022A | WWW Favorites |
Entries marked with * have the extended bit (0x100) set.
PointerEvent (type 0x05) — ATEN Extended
Two formats depending on whether AES encryption is enabled:
Unencrypted (when mouse.encryption_flag == 0):
┌─────────────────────────────────┐
│ type: u8 = 0x05 │
│ encrypt_flag: u8 = 0x00 │
│ button_mask: u8 │
│ x_position: u16 │
│ y_position: u16 │
│ padding: 11 bytes (zero) │
└─────────────────────────────────┘
AES-encrypted (when mouse.encryption_flag != 0):
┌─────────────────────────────────────┐
│ type: u8 = 0x05 │
│ encrypt_flag: u8 = 0x01 │
│ encrypted: u8[16] │ AES-128-CBC encrypted block
└─────────────────────────────────────┘
Total: 18 bytes in both cases.
Encrypted payload (16 bytes before encryption):
┌─────────────────────────────────────┐
│ button_mask: u8 │ byte 0: from param_3 low byte
│ x_position: u16 (big-endian, │ bytes 1-2: PsudoStreamSwap16(x)
│ byte-swapped) │
│ y_position: u16 (big-endian, │ bytes 3-4: PsudoStreamSwap16(y)
│ byte-swapped) │
│ random_pad: 11 bytes │ bytes 5-15: random fill (from 4x rand() calls)
└─────────────────────────────────────┘
The encrypted path (RFBMouse::SendMouse at 0x001194f0):
- Fills a 16-byte
local_48buffer with random data (4 calls torand(), each filling 4 bytes = 16 bytes total) - Overwrites byte 0 with button_mask (low byte of
param_3) - Overwrites bytes 1-2 with
PsudoStreamSwap16(x_position)(big-endian u16) - Overwrites bytes 3-4 with
PsudoStreamSwap16(y_position)(big-endian u16) - Bytes 5-15 remain as the random data (11 bytes of random padding)
- Calls
EnCryto(local_48, output, 0x10)through the RFBKMCryto vtable at*(this+0x20)offset 0x18 -- this is theRFB_AES128_EventCryto()method - Sends: type=0x05, encrypt_flag=0x01, followed by the 16 encrypted bytes
StreamWriteFlush()to send
The unencrypted path:
- Sends: type=0x05, encrypt_flag=0x00
StreamWrite8(button_mask)-- 1 byteStreamWrite16(x_position)-- 2 bytes big-endianStreamWrite16(y_position)-- 2 bytes big-endianStreamWriteSkip(11)-- 11 zero bytes paddingStreamWriteFlush()to send
Button mask bits:
- Bit 0 (0x01): Left button
- Bit 1 (0x02): Middle button
- Bit 2 (0x04): Right button
Mouse wheel (RFBMouse::MouseAction at 0x00119660):
The MouseAction function receives the full MouseInfo_t struct (button_mask,
x, y, wheel_rotation). The wheel rotation value is in the upper 32 bits of
param_3 (i.e., param_3 >> 32).
- If
wheel_rotation == 0: callsSendMouse()once (normal move/click). - If
wheel_rotation != 0: takes the absolute value ofwheel_rotation, then callsSendMouse()twice per unit of wheel rotation (press + release pair). The loop runsabs(wheel_rotation)iterations, each iteration callingSendMouse()two times. The sign ofwheel_rotationdetermines the scroll direction (the button_mask passed to SendMouse encodes scroll-up vs scroll-down as different button bits set by the Java caller).
Total mouse events per wheel tick: 2 (one press, one release). For a wheel rotation of N ticks, 2*|N| PointerEvent messages are sent.
MouseSync (type 0x07)
┌─────────────────────────────────┐
│ type: u8 = 0x07 │
│ value: u16 = 0x0780 │ Fixed value (1920 decimal)
└─────────────────────────────────┘
Total: 3 bytes. Confirmed from decompilation of RFBMouse::MouseSync at
0x00119730:
StreamWriteStart()StreamWrite8(0x07)-- type byteStreamWrite16(0x0780)-- hardcoded value 1920 (max framebuffer width)StreamWriteFlush()
Sent to synchronize mouse position with the remote host. The value 0x0780 (1920) represents the maximum horizontal resolution and is always sent unchanged regardless of the actual display resolution.
Server-side behavior (WPCM450/Hermon): The server handler at 0xf5dc reads
the u16 value and calls DeviceManager vtable+0x0c, which dispatches to
MouseDevice::ReSync at 0xe9b0. This issues ioctl(ATEN_MOUSE_IOCRESYNC) to
the kernel's usb_hid module, which queues a mouseSync work item that resets the
virtual mouse position tracking to origin. The u16 value is passed as the ioctl
parameter but is not used by the kernel driver.
MouseReset (type 0x08)
┌─────────────────────────────────┐
│ type: u8 = 0x08 │
│ mouse_type: u8 │ Current mouse type (from this+0x14)
└─────────────────────────────────┘
Total: 2 bytes. Confirmed from RFBMouse::MouseReset at 0x00119780. Sends
the current mouse type byte stored at RFBMouse offset 0x14 (received from
the server via ProcMouseInfo).
MouseUpdateInfo (type 0x37, client-to-server)
┌─────────────────────────────────┐
│ type: u8 = 0x37 │
└─────────────────────────────────┘
Total: 1 byte. Sent by RFBMouse::MouseUpdateInfo at 0x001197d0. Clears the
"config received" flag (offset 0x10 = 0) and sends type 0x37 to the server to
request a fresh MouseInfo update. The server responds with a type 0x37
server-to-client message containing the current mouse configuration.
KeyboardUpdateInfo (type 0x35, client-to-server)
┌─────────────────────────────────┐
│ type: u8 = 0x35 │
└─────────────────────────────────┘
Total: 1 byte. Sent by RFBKeyboard::KeyboardUpdateInfo at 0x0011a610. Clears
the "config received" flag (offset 0x10 = 0) and sends type 0x35 to the server
to request a fresh keyboard + mouse info update. The server responds with a
type 0x35 server-to-client message containing both keyboard and mouse config.
Note: This is the client-to-server counterpart of the server-to-client type 0x35 (KeyboardInfo + MouseInfo). The type 0x35 is dual-purpose: when sent by the client it is a 1-byte request; when sent by the server it is a 6-byte configuration message.
MouseSetPT (type 0x36)
┌─────────────────────────────────┐
│ type: u8 = 0x36 │
│ mouse_mode: u8 │ Mode value (from this+0x18)
│ mouse_type: u8 │ Type value (from this+0x14)
└─────────────────────────────────┘
Total: 3 bytes. Sent by RFBMouse::MouseSetPT at 0x00119820. This function
sets the pointer type on the server side. It first compares the requested
mode/type against the current values (offsets 0x18 and 0x14); if both match,
the message is suppressed (returns without sending). Otherwise, it stores the
new values and sends the type 0x36 message.
KeepAlive ACK (type 0x16)
┌─────────────────────────────────┐
│ type: u8 = 0x16 │
│ ack: u8 │ Boolean: always 1 in practice
└─────────────────────────────────┘
Total: 2 bytes. Confirmed from RFBProtocol::ProcAlive at 0x00118230:
StreamWriteStart(ntw);
StreamWrite8(ntw, 0x16); // type
StreamWrite8(ntw, param_1); // ack value (bool parameter)
StreamWriteFlush(ntw);
The param_1 parameter is typed as bool in the decompilation. The
sendKeepAliveAck JNI function at 0x00121a00 always passes 1 (true).
A client implementation should send 0x16 0x01 in response to every
server KeepAlive.
ScreenSetPosition (type 0x15)
┌─────────────────────────────────┐
│ type: u8 = 0x15 │
│ x_position: u32 │ Screen X offset
│ y_position: u32 │ Screen Y offset
└─────────────────────────────────┘
Total: 9 bytes. Sends a screen position/offset to the server. The x and y
values are packed as a 64-bit value split into two 32-bit words.
(RFBScreen::ScreenSetPosition at 0x001189b0)
Server-side note (WPCM450/Hermon): The server handler at 0xf7b0 reads the
two u32 values and discards them. This message has no effect on the Hermon
server — the VCD hardware captures the full video signal regardless of
position offset.
ScreenCalibration (type 0x17)
┌─────────────────────────────────┐
│ type: u8 = 0x17 │
│ flag: u8 = 0x01 │
└─────────────────────────────────┘
Triggers a screen recalibration on the BMC.
Server-side note (WPCM450/Hermon): Type 0x17 is NOT present in the server's ProtocolHandler dispatch switch. The server does not handle this message type. Sending it will trigger the default case and print "ProtocolHandler: Error message type !!". This message type may only be supported on other BMC chipsets (AST, Pilot3).
SetPowerOnOff (type 0x1A)
┌─────────────────────────────────┐
│ type: u8 = 0x1A │
│ action: u8 │ Power action code
└─────────────────────────────────┘
Total: 2 bytes.
Power action codes (confirmed from binary analysis of JNI functions):
| Code | Constant | JNI Function | Address | Description |
|---|---|---|---|---|
| 0 | POWER_OFF | setPowerOff (0x00121a70) |
Calls SetPowerOnOff(0) | Hard power off |
| 1 | POWER_ON | setPowerOn (0x00121a50) |
Calls SetPowerOnOff(1) | Power on |
| 2 | POWER_RESET | setPowerReset (0x00121ab0) |
Calls SetPowerOnOff(2) | Hard reset (reboot) |
| 3 | SOFT_OFF | setSoftPowerOff (0x00121a90) |
Calls SetPowerOnOff(3) | ACPI soft power off |
All four JNI functions call SetPowerOnOff(action) through the RFBScreen vtable
(vtable index 7, offset 0x38 from vtable base). RFBScreen::SetPowerOnOff at
0x00118900 simply writes: type 0x1A, then the action byte, then flushes.
Server-side note (WPCM450/Hermon): The server handler at 0xfae4 dispatches
to libutility.so functions: code 0 → UtilPowerDown(), 1 → UtilPowerUp(),
2 → UtilPowerReset(), 3 → UtilSoftPowerDown(). Any other code prints
"Error Power Control Request!!". No response is sent to the client. Power
control requires no special permission — it is in the "always allowed" group
in the server's permission filter (alongside KeepAlive, SetBandwidth, etc.).
ScreenSetInfo (type 0x32)
┌─────────────────────────────────┐
│ type: u8 = 0x32 │
│ width: u16 │
│ height: u16 │
└─────────────────────────────────┘
Total: 5 bytes.
Sent when the client detects a resolution change and informs the server.
The function (RFBScreen::ScreenSetInfo at 0x00118b80) only sends the message
if the resolution has actually changed (compares new height against stored
value at RFBScreen offset 0x18). If unchanged, the message is suppressed.
When the resolution changes, it also updates the internal ScreenInfo_t structure (offsets 0x10-0x30) with the new screen parameters before sending.
SendKickRequest (type 0x38)
┌─────────────────────────────────────┐
│ type: u8 = 0x38 │
│ action: u32 │ Kick action code (param_1)
│ reserved: u32 = 0 │ Always zero (hardcoded)
│ data: u8[64] │ Kick data payload (param_2)
└─────────────────────────────────────┘
Total: 73 bytes. Confirmed from decompilation of RFBPrivilege::SendKickRequest
at 0x0011a1f0:
StreamWriteStart()StreamWrite8(0x38)-- type byte (0x38 = '8')StreamWrite32(param_1)-- action code (from JavasendPrivilegeCtrl)StreamWrite32(0)-- hardcoded zero (reserved field)StreamWrite(param_2, 0x40)-- 64 bytes of kick dataStreamWriteFlush()
The sendPrivilegeCtrl JNI function at 0x00121970 extracts a 64-byte array
from Java (via GetByteArrayRegion) and passes it along with the action code
integer to SendKickRequest through the RFBPrivilege vtable (offset 0x20).
Used for privilege control -- kicking other users off the KVM session. The action code and 64-byte data payload are defined by the Java application layer.
MouseHotPlug (type 0x3A)
┌─────────────────────────────────┐
│ type: u8 = 0x3A │
└─────────────────────────────────┘
Total: 1 byte. Confirmed from decompilation of RFBMouse::MouseHotPlug at
0x00119910:
StreamWriteStart()StreamWrite8(0x3A)-- type byte (0x3A = ':')StreamWriteFlush()
Single byte message to trigger mouse device re-enumeration on the BMC. No
additional payload. Called from the hotPlug JNI method at 0x00121a30.
QoS (type 0x3B)
┌─────────────────────────────────┐
│ type: u8 = 0x3B │
│ param1: u32 │ QoS parameter 1 (video quality)
│ param2: u32 │ QoS parameter 2 (compression)
│ param3: u32 │ QoS parameter 3 (bandwidth)
└─────────────────────────────────┘
Total: 13 bytes. Confirmed from decompilation of RFBProtocol::ProcQos at
0x00118290:
StreamWriteStart()StreamWrite8(0x3B)-- type byte (0x3B = ';')StreamWrite32(param1)-- first QoS parameterStreamWrite32(param2)-- second QoS parameterStreamWrite32(param3)-- third QoS parameterStreamWriteFlush()
Quality-of-service parameters for video compression tuning. The three u32
parameters are passed directly from the Java setQosParameter JNI method
at 0x00121c00, which takes three int arguments from the Java side. The JNI
function calls through RMConnection::QosControl at 0x00112b40 (vtable
offset 0x40) which delegates to RFBProtocol::ProcQos.
The base class RMProtocol::ProcQos at 0x0011fa80 is a stub (returns 1
without sending anything), so only the RFBProtocol override actually sends
the message. The exact semantics of the three parameters are determined by
the Java application and BMC firmware (video quality level, compression ratio,
and bandwidth limit are the likely interpretations based on the Java UI
option frames).
Server-side note (WPCM450/Hermon): The server handler at 0xfa10 reads
all 3 u32 values and passes them to SetBandwidthLimits at 0x13128, which
configures the stream's read and write bandwidth counter objects. These counters
implement select()-based throttling — when bandwidth exceeds the configured
limit, the stream write functions sleep to enforce the cap. The 3 parameters
map to: limit1 (read bandwidth), limit2 (write bandwidth), period
(time window for measurement).
GetScreenUI (type 0x3C)
┌─────────────────────────────────┐
│ type: u8 = 0x3C │
└─────────────────────────────────┘
Single byte message requesting the server's current OSD language configuration.
The server responds with a type 0x3C server-to-client message containing the
lang_id and lang_sub values. (RFBProtocol::ProcGetScreenUI at 0x001183a0)
SetScreenUILang (type 0x3D)
┌─────────────────────────────────┐
│ type: u8 = 0x3D │
│ lang_id: u32 │ Language identifier (param_1)
│ lang_sub: u32 (conditional) │ Only if FW protocol flag != 0
└─────────────────────────────────┘
Total: 5 bytes (without lang_sub) or 9 bytes (with lang_sub).
Confirmed from decompilation of RFBProtocol::ProcSetScreenUI at 0x00118320:
StreamWriteStart()StreamWrite8(0x3D)-- type byte (0x3D = '=')StreamWrite32(param_1)-- language ID (always sent)- If
this[0x48] != 0(thefw_protocol_flagat RFBProtocol offset 0x48):StreamWrite32(param_2)-- language sub-ID (conditionally sent) StreamWriteFlush()
The FW protocol flag (this[0x48]) is a single byte set by
RFBProtocol::SetFWProtocol(uchar) at 0x00118530. This flag is set from the
Java side through the vtable. When the flag is zero (default), only 5 bytes
are sent (type + lang_id). When non-zero, 9 bytes are sent (type + lang_id +
lang_sub).
Server-side note (WPCM450/Hermon): The server handler at 0xfaa4 always
reads 2 u32 values (8 bytes of payload), regardless of any FW protocol flag.
A client sending only 5 bytes (1 u32) will cause the server to block waiting
for the second u32, potentially causing a protocol desync. For Hermon servers,
always send both u32 values (9 bytes total).
The setScreenUILang JNI method at 0x00121b00 takes two int parameters from
Java and calls ProcSetScreenUI(param_3, param_4) through the RFBProtocol
vtable (offset 0x30).
The corresponding getScreenUILang JNI method at 0x00121b50 calls
ProcGetScreenUI() through the vtable (offset 0x38), which sends a 1-byte
type 0x3C request. The server responds with a type 0x3C message.
Video Encoding
Encoding Types
The encoding_type field in FramebufferUpdate identifies the BMC chipset's
video compression format:
| ID | Hex | Decoder Class | BMC Chipset |
|---|---|---|---|
| 87 | 0x57 | ASTVideoDecoder | ASPEED AST2050 |
| 88 | 0x58 | ASTVideoDecoder | ASPEED AST2100/2300/2400/2500 |
| 89 | 0x59 | HermonVideoDecoder | Nuvoton WPCM450 (Hermon) |
| 96 | 0x60 | YarkonVideoDecoder | Yarkon |
| 97 | 0x61 | Pilot3VideoDecoder | ServerEngines Pilot III |
AST Video Compression (0x57, 0x58)
ASPEED BMCs use a proprietary JPEG-like compression scheme:
-
AST2050 (0x57): Uses the
ast2100decoder class with DCT, Huffman coding, and VQ (Vector Quantization) decompression. The decoder allocates ~1MB (0x1029E8 bytes) of state. -
AST2100+ (0x58): Uses the
ast_jpegdecoder class. Pure JPEG-variant with standard DCT/IDCT, Huffman decoding, and YUV→RGB color conversion. Allocates 0x588 bytes of state.
Both decoders:
- Accept the compressed data buffer and produce RGBA pixel output
- Support incremental updates (dirty rectangles via
UpdateBlocks_t) - Track resolution changes and report them back to the Java layer via
changeResolutioncallback
Decoder Selection (RMDecoder::GetDecoder at 0x00112560)
The GetDecoder function creates a decoder based on the encoding type from
the FramebufferUpdate. It is a singleton -- once created, it is reused:
| Encoding | Decoder Class | Alloc Size |
|---|---|---|
| 0x57 | ASTVideoDecoder | 0x58 |
| 0x58 | ASTVideoDecoder | 0x58 |
| 0x59 | HermonVideoDecoder | 0x58 |
| 0x60 | YarkonVideoDecoder | 0x60 |
| 0x61 | Pilot3VideoDecoder | 0x458 |
GetDecoder is a singleton factory -- once a decoder is created, it is cached
in the global DecoderHandle and reused for all subsequent frames. If the
encoding type does not match any of 0x57, 0x58, 0x59, 0x60, or 0x61, the
function returns NULL (there is no default/fallback decoder). A NULL decoder
pointer will cause ScreenDecode to skip decoding (it checks lVar9 != 0
before accessing the decoder). The switch statement has no default case.
ASTVideoDecoder::Decode (at 0x0010f8b0) then creates the inner decoder:
- Encoding 0x58: allocates
ast_jpeg(0x588 bytes) - Encoding 0x57: allocates
ast2100(0x1029E8 bytes)
AST JPEG Wire Format (encoding 0x58)
The compressed data blob within a FramebufferUpdate has a 16-byte header followed by a stream of tile command words.
Header (first 16 bytes of the data blob, read as big-endian u64 words):
Offset Size Description
0x00 8 Header word 0 (big-endian u64): total compressed size (bits [55:0])
0x08 8 Header word 1 (big-endian u64): parameters
The byte-swap in the decoder reads the first two 8-byte words from the buffer and reinterprets them as big-endian 64-bit integers.
Header word 1 bit fields:
| Bits | Field | Description |
|---|---|---|
| [31] | Advance mode | If set, negates the advance counter |
| [10:8] | Y quantization level | Luminance quant (3 bits) |
| [22:21] | CbCr quant level | Chrominance quant (3 bits) |
| [20:16] | CbCr tile size | Chrominance tile size (5 bits, 0 = 16) |
| [28:24] | Y tile size | Luminance tile size (5 bits, 0 = 16) |
If Y tile size == 0, defaults to 16x16 tiles. If CbCr tile size == 0,
defaults to 16x16 tiles. The tile size controls the IDCT block dimensions.
After the header, additional data at offset 0x10 provides initial tile positions and the bitstream data.
Tile command stream:
After the header, the data is processed as a stream of tile commands. Each command word is read from the bitstream with the top 2 bits indicating the command type:
| Bits [31:30] | Type | Description |
|---|---|---|
| 0 (0b00) | Same position, alt QT | Decode tile at current position with QT_TableSelection=2 |
| 1 (0b01) | Same position | Decode tile at current position with QT_TableSelection=0 |
| 2 (0b10) | New position, alt QT | Jump to new position, decode with QT_TableSelection=2 |
| 3 (0b11) | New position | Jump to new position, decode with QT_TableSelection=0 |
For types 2 and 3 (new position), the tile position is embedded in the command word:
| Bits | Field | Description |
|---|---|---|
| [29:23] | tile_x | X tile index (7 bits) |
| [22:16] | tile_y | Y tile index (7 bits) |
For types 0 and 1 (same position), the previous tile position is reused and only 2 bits are consumed from the bitstream for the command.
Tile decoding (per tile, 6 Huffman-coded DCT blocks):
Each tile is decoded as a 4:2:0 YCbCr macroblock:
-
4 luminance (Y) blocks (8x8 each, stored at byTileYuv+0x00, +0x40, +0x80, +0xC0):
process_Huffman_data_unit()->get_DCT()->IDCT_transform(QT=0)
-
1 Cb block (8x8, stored at byTileYuv+0x100):
process_Huffman_data_unit()->get_DCT()->IDCT_transform(QT=1)
-
1 Cr block (8x8, stored at byTileYuv+0x140):
process_Huffman_data_unit()->get_DCT()->IDCT_transform(QT=1)
Total: 6 blocks per tile = 4Y + 1Cb + 1Cr (standard 4:2:0 JPEG subsampling)
YUV to RGB conversion (ast_jpeg::YUVToRGB at 0x0011e4e0):
After IDCT, the 6 decoded blocks are in byTileYuv buffer (0x180 = 384 bytes).
The YUVToRGB() function converts the tile from YCbCr to 32bpp RGBA and writes
it into the framebuffer at the tile position.
Screen output (ast_jpeg::ScreenResolution at 0x0011e770):
Two modes based on the tile_size field at this+0x564:
- 16x16 tile mode: 256 pixels per tile, written as 16bpp to the output buffer with color scaling using the pixel format masks
- Non-16 mode: 256 pixels per tile, written as 32bpp (BGRX) to the output buffer with a boundary check against 0x7E9000 bytes (framebuffer limit)
Tile index advancement (ast_jpeg::MoveBlockIndex at 0x0011e700):
After decoding each tile:
- Increment X tile index by 1
- If X >= (aligned_width / tile_size): wrap to X=0, increment Y tile index
- If Y >= (aligned_height / tile_size): reset to Y=0
- Track total decoded pixels in a counter (incremented by 0x100 = 256 per tile)
- Stop when counter >= total pixels (header word 0 << 8)
Huffman tables:
The decoder uses fixed Huffman tables (not embedded in the stream):
- DC luminance table:
DC_LUMINANCE_HUFFMANCODEat 0x001304C0 - DC chrominance table:
DC_CHROMINANCE_HUFFMANCODEat 0x00130480 - AC luminance table:
AC_LUMINANCE_HUFFMANCODEat 0x001303E0 - AC chrominance table:
AC_CHROMINANCE_HUFFMANCODEat 0x00130320 - General Huffman constants:
HUFFMANCODEat 0x0012FAA0
These are standard JPEG Huffman tables stored in a pre-computed format.
Quantization:
Quantization levels are set per-frame from header word 1 bits. The
decoder_fun::set_QuantLevel() function configures three QT table selections
(0, 1, 2) based on these parameters. QT_TableSelection=0 is used for standard
luminance, QT_TableSelection=1 for chrominance, and QT_TableSelection=2 for
an alternate quality level (used in command types 0 and 2).
AST2050 Video Compression (encoding 0x57)
The AST2050 uses the ast2100 class (~1 MB state, 0x1029E8 bytes) with a hybrid
VQ (Vector Quantization) + Huffman + IDCT decoder, optionally preceded by RC4
decryption.
Data blob layout (within FramebufferUpdate):
The first 4 bytes of the compressed data blob are parsed by SetOptions():
Offset Size Description
0x00 1 Mode byte 0 (param_3, maps to YUV/grayscale mode)
0x01 1 Mode byte 1 (param_4, maps to quality level)
0x02 2 Chip revision (big-endian u16, byte-swapped via PsudoStreamSwap16)
0x04 ... Compressed tile bitstream data
Chip revision (2 bytes at offset 0x02):
0x01A6: AST2050 mode - tile size 16x16, YUV 4:2:0 mode (mode_flag=1)0x01BC: AST2050 alternate - tile size 8x8, grayscale mode (mode_flag=0)
After the 4-byte header, the remainder is the compressed bitstream.
RC4 decryption (conditional):
If the encrypt_flag (this+0x101f6d) is set to 1, the entire data payload
(after the 4-byte header) is decrypted with RC4 before tile decoding.
- RC4 key: The hardcoded string
"fedcba9876543210"(at address 0x00230500), expanded to 256 bytes byKeys_Expansion()which repeats the key cyclically - RC4 state: Standard RC4 KSA (Key Scheduling Algorithm) implemented in
DecodeRC4_setup(), producing a 256-entry S-box - Decryption:
RC4_crypt()XORs the data with the RC4 PRGA stream
The RC4 encryption appears to be optional obfuscation of the video stream data to prevent casual interception. The key is hardcoded and not derived from any session material.
Tile bitstream format:
The bitstream is processed as a sequence of 32-bit command words. The top 4 bits
(uVar6 >> 28, called "nibble code") select the command type:
| Nibble | Description |
|---|---|
| 0x0 | Huffman+IDCT decode at current position (no QT change) |
| 0x4 | Huffman+IDCT decode with alternate QT (param_4=2) |
| 0x5 | VQ decode, 1 color cache entry (no position update) |
| 0x6 | VQ decode, 2 color cache entries (no position update) |
| 0x7 | VQ decode, 4 color cache entries (no position update) |
| 0x8 | Set new tile position + Huffman+IDCT decode |
| 0x9 | End of frame (return immediately) |
| 0xC | Set new tile position + Huffman+IDCT alternate QT |
| 0xD | Set new tile position + VQ 1-entry decode |
| 0xE | Set new tile position + VQ 2-entry decode |
| 0xF | Set new tile position + VQ 4-entry decode |
For position-setting commands (0x8, 0xC, 0xD, 0xE, 0xF), the position is encoded in the command word:
Bits [27:20] = Y block coordinate (row)
Bits [19:12] = X block coordinate (column)
Then 20 bits are consumed from the bitstream (4 nibble + 8+8 position bits).
For non-position commands (0x0, 0x4, 0x5, 0x6, 0x7), only 4 bits are consumed for the command nibble.
VQ (Vector Quantization) decompression (VQ_Decompress at 0x0011d490):
The COLOR_CACHE is a 4-entry palette (each entry is a YCbCr triple: Y, Cb, Cr).
Initialized by VQ_Initialize() with default values:
- Entry 0: Y=0x00, Cb=0x80, Cr=0x80 (black)
- Entry 1: Y=0xFF, Cb=0x80, Cr=0x80 (white)
- Entry 2: Y=0x80, Cb=0x80, Cr=0x80 (gray)
- Entry 3: Y=0xC0, Cb=0x80, Cr=0x80 (light gray)
For VQ commands, additional bits from the bitstream update the color cache:
- Bit 31 set: A new 24-bit color (YCbCr) is read from the bitstream and stored into the cache slot indicated by bits [30:29] (2-bit index)
- Bit 31 clear: Only a 3-bit cache index is consumed, selecting an existing color from the cache
VQ fills 64 pixel values (an 8x8 block) by either:
- Flat fill (cache entry 0x24 == 0): fill all 64 pixels with one color
- Per-pixel indexing (cache entry 0x24 != 0): read cache indices from bitstream
Huffman+IDCT decompression (Decompress at 0x0011d150):
Two modes based on mode_flag (this+0x101f6f):
- Mode 0 (8x8 tiles): 1 Y block + 1 Cb block + 1 Cr block (YCbCr 4:4:4)
- Mode 1 (16x16 tiles): 4 Y blocks + 1 Cb block + 1 Cr block (YCbCr 4:2:0)
Each block is Huffman-decoded via process_Huffman_data_unit() and then IDCT
transformed via IDCT_transform(). The IDCT output is 64 coefficients per
8x8 block (standard JPEG-like DCT).
YUV to RGB conversion (ast2100::YUVToRGB at 0x0011cec0):
Uses pre-computed lookup tables stored in the ast2100 object:
Cr_tab(offset 0x044): Cr to R contributionCb_tab(offset 0x444): Cb to G contributionCr_Cb_green_tab(offset 0x844): combined CrCb to G contributionY_tab(offset 0x1044): Y to luminancerlimit_table(offset 0x101a60): clamping/range-limit table
Output format: 32bpp BGRX (byte order: [0]=unused, [1]=B, [2]=G, [3]=R)
Tile advancement (MoveBlockIndex at 0x0011d590):
After each tile is decoded, the block position advances:
- Mode 0 (8x8): X increments by 1; wraps at (width / 8), then Y increments
- Mode 1 (16x16): X increments by 1; wraps at (width / 16), then Y increments
Dirty rectangles (x, y, w, h) are recorded into the UpdateBlocks_t array:
- Mode 0: each block is 8x8 pixels
- Mode 1: each block is 16x16 pixels
Frame termination: The loop terminates when either:
- Nibble code 0x9 is encountered (explicit end marker)
- The iteration counter exceeds
(width * height) / 64(minimum 0x1000)
Hermon Video Compression (0x59)
Nuvoton WPCM450 BMCs use the HermonVideoDecoder with raw pixel tiles and
simple pixel format conversion.
Multi-session check: If encoding_type == 0x10001 (multi-session flag), the decoder throws an exception: "Not Support Multi-Session!"
Data blob wire format (HermonVideoDecoder::Decode at 0x0010fee0):
Offset Size Description
0x00 1 Frame type: 0x01 = full frame, other = incremental
0x01 1 Color depth flag (capture slot mode): 0x00 = 16bpp RGB555, non-zero = 8bpp palette
0x02 4 Full: magic 0x12345678; Incremental: tile count (big-endian u32)
0x06 4 Total data length (big-endian u32, bytes [6..9])
0x0A ... Tile data (format depends on frame_type)
Server-side detail: Bytes 2-5 have different semantics for full vs incremental:
- Full frame: always the magic bytes
0x12, 0x34, 0x56, 0x78(constant sentinel) - Incremental: tile count as a big-endian u32
Fallback behavior: When the server determines that incremental frame data would
be larger than a full frame (diff_size >= bpp * width * height + 10), it discards
the incremental result and sends a full frame instead.
Full frame (frame_type == 0x01):
- The entire tile data starting at offset 10 contains raw pixel data for the complete screen
ConvertVierwerPixelFormat()converts from source bpp to 32bpp BGRX- The converted data is copied to both the working and display framebuffers
- UpdateBlocks reports a single full-screen rectangle (0xFFFF, 0xFFFF, 16, 16)
Incremental update (frame_type != 0x01): Each tile in the stream has a 6-byte header followed by raw pixel data:
Per-tile format:
Offset Size Description
0x00 4 Padding (uninitialized stack data from server's CopyTileData, not meaningful)
0x04 1 Tile Y position (in units of 16 pixels)
0x05 1 Tile X position (in units of 16 pixels)
0x06 N Raw pixel data (N = 0x200 for 16bpp or 0x100 for 8bpp)
N depends on the color depth flag:
- 16bpp (flag == 0): N = 512 bytes (16x16 pixels x 2 bytes each)
- 8bpp (flag != 0): N = 256 bytes (16x16 pixels x 1 byte each)
Each tile is passed through ConvertVierwerPixelFormat() and then SetRect()
copies the converted 16x16 tile into the framebuffer at (x16, y16).
Pixel format conversion (ConvertVierwerPixelFormat at 0x0010fd60):
Two modes based on the color_depth_flag parameter:
16bpp mode (RGB555, flag == 0): Input is 2 bytes per pixel
Input: byte[0] = GGGBBBBB, byte[1] = xRRRRRGG
Output: [0]=0x00, [1]=B<<3, [2]=(byte[1]<<6)|(byte[0]&0xE0)>>2, [3]=(byte[1]&0x7C)<<1
This is RGB555 -> BGRX32 conversion.
8bpp mode (2-bit-per-channel palette, flag != 0): Input is 1 byte per pixel
Input: byte = xxRRGGBB
Output: [0]=0x00, [1]=BB<<6, [2]=GG<<4, [3]=RR<<2
This is a simple 2-bit-per-channel to 8-bit expansion.
SetRect (HermonVideoDecoder::SetRect at 0x0010fe30):
Copies a decoded 16x16 tile (64 bytes per row, 16 rows) into the framebuffer at position (x * 64_bytes, y * pitch * 16). The pitch is clamped to min(screen_width, 1280).
Yarkon Video Compression (0x60)
The Yarkon decoder uses a combination of Huffman compression, RLE compression, and RFB Hextile-style encoding.
Multi-session check: Same as Hermon -- throws "Not Support Multi-Session!"
Data blob wire format (YarkonVideoDecoder::Decode at 0x00110630):
The data blob starts with a 1-byte compression indicator:
Offset Size Description
0x00 1 Compression type: 0xFF = Huffman, 0xFE = RLE, other = raw
Huffman-compressed (byte[0] == 0xFF):
0x00 1 0xFF marker
0x01 4 Compressed size (big-endian u32)
0x05 4 Decompressed size (big-endian u32)
0x09 ... Huffman-compressed payload
The payload is decompressed via Huffman_Uncompress() into a working buffer
before tile parsing.
RLE-compressed (byte[0] == 0xFE):
0x00 1 0xFE marker
0x01 4 Compressed size (big-endian u32)
0x05 4 (unused for RLE, but present for alignment)
0x09 ... RLE-compressed payload
Decompressed via RLE_Uncompress().
Raw/uncompressed (other values): Data is used directly.
Decompressed tile stream format:
After decompression, the tile stream has:
Offset Size Description
0x00 1 Frame type (0x00 = full frame, other = incremental)
0x01 2 Tile count (big-endian u16, bytes [1..2])
0x03 4 Data length (big-endian u32, bytes [3..6])
0x07 ... Per-tile records
Per-tile record:
0x00 1 Tile X position (in tiles)
0x01 1 Tile Y position (in tiles)
0x02 2 Tile pixel data length (big-endian u16)
0x04 N Hextile-encoded tile data
Each tile is decoded via HextileDecoder() which implements the RFB Hextile
encoding:
Hextile sub-encoding flags (first byte of tile data):
| Bit | Flag | Description |
|---|---|---|
| 0 | Raw | Tile is raw pixel data (no hextile encoding) |
| 1 | BackgroundSpecified | 2-byte background color follows |
| 2 | ForegroundSpecified | 2-byte foreground color follows |
| 3 | AnySubrects | Subrectangle count byte follows |
| 4 | SubrectsColoured | Each subrect has its own 2-byte color |
When bit 0 (Raw) is set, the tile is raw 16bpp pixel data, converted via
ConvertVierwerPixelFormat() and placed via SetRect().
Otherwise, the tile is filled with the background color, then subrectangles (if present) are drawn on top. Each subrect is encoded as:
[optional 2-byte color] -- if SubrectsColoured flag set
1 byte: (x << 4) | y -- position within tile (4 bits each)
1 byte: (w << 4) | h -- size within tile (4 bits each, value+1)
Yarkon pixel format conversion is identical to Hermon's 16bpp mode (RGB555): input is 2 bytes per pixel, output is 4 bytes BGRX.
Full frame indicator: When frame_type == 0x00 (first byte of decompressed stream), the UpdateBlocks reports a single full-screen update (0xFFFF, 0xFFFF).
RLE_Uncompress (RLE_Uncompress at 0x0011a520):
Simple RLE with an escape character (first byte of input):
- Literal bytes (not equal to escape): copied directly
- Escape + count (0..2): output
count+1copies of the escape character itself - Escape + count (3..127) + value: output
count+1copies of value - Escape + count (128..255) + value: count is extended to 16 bits
(
(count & 0x7F) << 8 | next_byte), then output count+1 copies of value
Pilot3 Video Compression (0x61)
ServerEngines Pilot III BMCs use the Pilot3VideoDecoder with a complex
multi-mode compression scheme supporting 1bpp, 2bpp, 4bpp, 8bpp, 16bpp,
and 32bpp color depths, and both full-frame and incremental tile updates.
Data blob wire format (Pilot3VideoDecoder::Decode at 0x00110d20):
Offset Size Description
0x00 1 Frame type: 0x01 = full, other = incremental
0x01 1 Color depth code (bpp_code): 1,2,3,4,5 (see below)
0x02 1 Sub-mode: 0x00 = standard, 0x02 = planar, other = uncompressed
0x06 4 Total data length (big-endian u32, bytes [6..9])
Color depth codes (byte at offset 0x01):
| Code | Bits per pixel | Pixel format |
|---|---|---|
| 1 | 8bpp | Palette-indexed (uses 1024-byte palette) |
| 2 | 16bpp | RGB555 |
| 3 | Uncompressed | Raw pixel data |
| 4 | 32bpp | BGRX (4 bytes per pixel) |
| 5 | 8bpp variant | 8bpp with palette at offset 0x0A |
Full frame (frame_type == 0x01):
For bpp_code==1 or bpp_code==5: A 1024-byte (256-entry x 4 bytes) color
palette is read at offset 0x0A, stored at this+0x30.
The pixel data follows and is decompressed through a custom RLE scheme with escape markers:
- Byte
0x55is the primary escape:0x55 0x00: literal 0x550x55 0x01: literal 0xAA0x55 count value: repeat valuecount+1times
- Byte
0xAAis the secondary escape:0xAA value: repeat value 3 times
The decompressed data is further processed through planar-to-chunky conversion for 32bpp and 16bpp modes, reconstructing interleaved pixel data from separate bitplanes.
For bpp_code==4 (32bpp), each pixel has 4 planes; for bpp_code==2 (16bpp),
each pixel has 2 planes.
Incremental update (frame_type != 0x01):
Tile records start after a tile-count header:
Offset Size Description
0x0A 2 Tile count (big-endian u16)
0x0C+N*4 ... Per-tile headers (4 bytes each)
Per-tile header (for sub-modes 2 and 4):
0x00 1 Tile row (Y)
0x01 1 Tile column (X)
0x02 1 Tile width (in pixels)
0x03 1 Tile height (in pixels)
Each tile's compressed data follows, using the same RLE scheme (0x55/0xAA escapes) with 4-byte-aligned chunk boundaries. A 4-byte "chunk length" header precedes each compressed segment.
Planar mode (sub-mode == 0x00 with bpp_code 2 or 4):
In this mode, the data is stored as bitplanes. After RLE decompression, the decoder reconstructs chunky pixel data from the planes using bit manipulation. For 32bpp, 4 planes are combined into BGRA; for 16bpp, 2 planes are combined into RGB555.
Sub-mode 0x02 (planar with change-list):
A special full-frame mode where the data contains a change-list. Each entry specifies an 8x16 character cell position and a bitmask indicating which of the 16 rows have changed. Each bit in the bitmask corresponds to a row; if set, that row's pixels (8 pixels wide) are set to white (0xFF,0xFF,0xFF,0x00).
Pilot3 pixel format conversion (ConvertVierwerPixelFormat at 0x00110a30):
Five modes based on the format byte:
| Format | Input | Conversion |
|---|---|---|
| 0x04 | 4bpp | Each byte = 2 pixels (low nibble, high nibble) |
| Palette lookup through 1024-byte table at this+0x30 | ||
| 0x08 | 8bpp | Each byte = 1 pixel, palette lookup at this+0x30 |
| 0x10 | 16bpp | RGB565 -> BGRX32 (B<<3, G<<5 |
| 0x20 | 32bpp | ARGB -> BGRX (byte swap: [3]->[0], [0]->[1], etc) |
For palette modes (0x04, 0x08), the output pixel is constructed from 4
consecutive bytes in the palette table at this+0x30 + index*4.
SetRect (Pilot3VideoDecoder::SetRect at 0x00110c20):
Copies a decoded 32x32 tile (128 bytes per row, 32 rows) into the framebuffer. Tile size is larger than Hermon/Yarkon (32x32 vs 16x16).
Framebuffer Format
The decoded framebuffer uses:
- 32 bits per pixel (BGRX in memory)
- Color masks: R=0x000000FF, G=0x0000FF00, B=0x00FF0000
- Max resolution: 1920 × 1200
- Default resolution: 640 × 480
The framebuffer is allocated as 0x600000 bytes (6,291,456 = 1920 × 1200 × ~3.4, rounded up, or simply a large fixed buffer).
The Java side wraps this in a DataBufferInt → WritableRaster → BufferedImage
for rendering.
AES-128-CBC Encryption
Mouse events can be optionally encrypted using AES-128-CBC when enabled by
the server. Keyboard events are never encrypted -- they are always sent
in cleartext despite the protocol having an RFBKMCryto instance allocated
for the keyboard object.
The encryption is controlled per-input-type:
- Mouse: Encrypted when
*(int*)(RFBMouse + 0x18)!= 0. This field isencryption_enabled(set from the first byte read by ProcMouseInfo -- the server tells the client whether to encrypt). InSendMouseat 0x001194f0, the check isif (*(int*)(param_1 + 0x18) != 0)-- when non-zero, encryption is enabled; when zero, events are sent in cleartext. The RFBMouse constructor initializes offset 0x18 to 0 (unencrypted by default). - Keyboard: Never encrypted.
RFBKeyboard::Sendkey()does not check any encryption flag and always sends in cleartext.
Hardcoded Key
Key: 2B 7E 15 16 28 AE D2 A6 AB F7 15 88 09 CF 4F 3C
Initialization Vector
IV: 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
Additional Key Schedule Data
The RFBKMCryto class contains additional hardcoded key schedule material:
Block 1: 8E 73 B0 F7 DA 0E 64 52 C8 10 F3 2B 80 90 79 E5
Block 2: 62 F8 EA D2 52 2C 6B 7B
Block 3: 60 3D EB 10 15 CA 71 BE 2B 73 AE F0 85 7D 77 81
Block 4: 1F 35 2C 07 3B 61 08 D7 2D 98 10 A3 09 14 DF F4
These are AES-128 expanded round key constants. The encryption uses
SW_AES_CBC() — a software AES-CBC implementation.
Encryption Flow
Mouse events only (RFBMouse::SendMouse at 0x001194f0):
When encryption_enabled (RFBMouse offset 0x18, set by server via
ProcMouseInfo first byte) is non-zero:
- Fill a 16-byte buffer with random data using
rand()(4 calls torand(), filling 4 x 4-byte words) - Place
button_mask(1 byte) at offset 0 - Place byte-swapped
x_position(2 bytes) at offset 1-2 viaPsudoStreamSwap16() - Place byte-swapped
y_position(2 bytes) at offset 3-4 viaPsudoStreamSwap16() - Bytes 5-15 remain as random data (11 bytes of random padding)
- Call
EnCryto(plaintext_16, ciphertext_16, 0x10)through the RFBKMCryto vtable- This calls
RFB_AES128_EventCryto()which: a. Sets up the hardcoded AES key and IV on the stack (see above) b. CallsSW_AES_CBC(mode=0, keysize=0, input, blocks=1, key, output, iv)c.mode=0= encryption,blocks=1= one 16-byte block
- This calls
- Send: type=0x05, encrypt_flag=0x01, followed by the 16 encrypted bytes
StreamWriteFlush()to send
When encryption_enabled is zero (unencrypted):
- Send: type=0x05, encrypt_flag=0x00, button_mask, x_position (u16), y_position (u16)
StreamWriteSkip(11)- 11 zero bytes paddingStreamWriteFlush()to send
SW_AES_CBC implementation (RFBKMCryto::SW_AES_CBC at 0x0010e2e8):
Parameters: (this, mode, keysize, input, num_blocks, key, output, iv)
mode=0: Encrypt. XOR each block with IV (first block) or previous ciphertext, thenaes_encrypt(). Standard CBC encrypt.mode=1: Decrypt.aes_decrypt()each block, then XOR with IV (first block) or previous ciphertext. Standard CBC decrypt.keysize: Passed askeysize * 64 + 128toaes_set_key()(for keysize=0, this is 128 bits).
Keyboard events (RFBKeyboard::Sendkey at 0x0011a670):
Keyboard events are always sent unencrypted. Despite having an RFBKMCryto* at
RFBKeyboard offset 0x20, the Sendkey() function never references it. The
message is simply: type 0x04, padding 0x00, down_flag, 2 zero bytes, keycode (u32),
9 zero bytes.
Connection Flow (Java Side)
Startup
KVMMain.main()parses arguments: IP, username, password, KVM_port, hostname, VM_port (default 623), company_id, board_id, language- Static initializer loads native libraries:
System.loadLibrary("iKVM64")andSystem.loadLibrary("SharedLibrary64") RMConnection.init(ConnInfo.class, UserInfo.class)initializes JNI field IDsRMConnection.keepActive(connInfo)establishes TCP connection:- Creates
RMConnectionobject - Calls
RFBProtocol::InitHandShake()which: a. Creates NtwStream and establishes TCP connection viaNtwStream::Connect()b. RunsProcVersion()(version exchange) c. RunsProcSecurity()(security type negotiation)
- Creates
RMConnection.checkValidUser(userInfo)authenticates:- JNI layer (
checkValidUserat 0x00119C60) extracts username (max 24 chars) and password (max 96 chars, but only 24 sent) from JavaUserInfoobject - Calls
RMConnection::CheckVaildUser()which delegates toRFBProtocol::Authenticate(): a. Reads 24-byte challenge from server (appears to be discarded) b. Sends 24-byte username (null-padded) c. Sends 24-byte password (null-padded) d. Reads auth result (u32): 0 = success, else reads error message e. On success: runsProcClientInit()thenProcServerInit()
- JNI layer (
- On success, creates
Viewerwhich initializes the GUI
Video Loop
The Viewer creates a RemoteVideo panel, which starts three threads:
-
CatchThread: Calls
catchLoop()→ nativeRFBProtocol::ProtocolHandler()in a loop, processing all incoming server messages -
DecodeThread: Calls
runImage()in a loop:- Calls
RFBProtocol::ProtocolHandler()to receive next frame - Calls
ScreenDecode()to decompress the video data - Calls
ScreenGetFreame()to get decoded pixels - Triggers
changeResolutioncallback on resolution changes - Reports dirty rectangles via
addClipBounds
- Calls
-
LazyWorker: Periodic tasks including:
refresh()→ sends FramebufferUpdateRequestupdateImage()→ copies framebuffer to Java BufferedImage- FPS calculation and display
Input Flow
Keyboard: processKeyEvent() ->
KeyMap.VKtoHID(vkCode, location) ->
keyboardAction(virtualKey, scanKey, extendedKey, action, 0, 0, 0) ->
native JNI at 0x00121000 (lock key check via X11 LED query) ->
RFBKeyboard::KeyboardAction(KeyInfo_t) ->
RFBKeyboard::processVK(KeyInfo_t) (VK -> X11 keysym via 3 lookup maps) ->
RFBKeyboard::Sendkey(keysym, downFlag) (always cleartext, type 0x04)
Mouse: mousePressed/Released/Moved/Dragged() ->
doMouseAction() ->
mouseAction(x, y, buttonMask, wheelRotation) ->
JNI Java_..._mouseAction at 0x001212d0 (null-checks desktop pointer) ->
RFBMouse::MouseAction() at 0x00119660 via vtable (offset 0x10 of desktop+8):
- If wheelRotation == 0: single
SendMouse()call - If wheelRotation != 0:
abs(wheelRotation)iterations, each callingSendMouse()twice (press + release pair) ->RFBMouse::SendMouse()at 0x001194f0 (AES-encrypted if mouse_mode != 0, cleartext if mouse_mode == 0, type 0x05)
Mouse Modes
Three mouse modes controlled by the server:
- Normal (mode 0): Sends absolute coordinates, client syncs mouse position
before each action with
MouseSync - Absolute (mode 1): Sends absolute coordinates directly
- Single (mode 2): Relative mode — sends delta from center of screen, Java side warps mouse cursor back to center after each event
Class Hierarchy (Native)
TcpSocket
└─ NtwStream (buffered I/O over TCP)
RMProtocol (virtual base)
└─ RFBProtocol
├─ InitHandShake() → TCP connect + ProcVersion + ProcSecurity
├─ Authenticate() → ATEN auth + ProcClientInit + ProcServerInit
├─ ProcVersion()
├─ ProcSecurity()
├─ ProcClientInit()
├─ ProcServerInit()
├─ ProtocolHandler() ← main message dispatch (jump table 0x00-0x3C)
├─ ProcAlive()
├─ ProcQos()
├─ ProcSetScreenUI()
├─ ProcGetScreenUI()
└─ SetFWProtocol()
RMScreen → RFBScreen
├─ ScreenUpdate() → send FramebufferUpdateRequest
├─ ScreenDecode() → receive and decode FramebufferUpdate (type 0x00)
├─ ScreenCursorPosProc() → handle cursor position (type 0x04)
├─ ScreenGetRect() → copy rect from framebuffer
├─ ScreenSetInfo() → send resolution (client type 0x32)
├─ ScreenSetPosition() → send position (client type 0x15)
├─ ScreenCalibration() → trigger recalibration
├─ SetPowerOnOff() → power control
├─ GetScreenUILang() → handle OSD lang response (type 0x3C)
└─ GeFrontGround() → get foreground/cursor compositing
RMKeyboard → RFBKeyboard
├─ Sendkey() → send KeyEvent message
├─ processVK() → translate VK/HID to keysym
├─ ProcKeyboardInfo() → handle server keyboard config
└─ KeyboardSetType() → set keyboard type
RMMouse → RFBMouse
├─ SendMouse() → send PointerEvent message (type 0x05)
├─ MouseAction() → process mouse action (+ wheel repeat)
├─ MouseSync() → send MouseSync message (type 0x07, value 0x0780)
├─ MouseReset() → send MouseReset (type 0x08, mouse_type)
├─ MouseUpdateInfo() → request mouse config (client type 0x37)
├─ MouseGetPT() → get pointer type info (16 bytes from this+0x10)
├─ MouseSetPT() → set pointer type (type 0x36, mode+type)
├─ MouseHotPlug() → send hot-plug reset (type 0x3A)
└─ ProcMouseInfo() → handle server mouse config (3 bytes)
RMPrivilege → RFBPrivilege
├─ ProcPrivilegeInfo() → handle privilege/AES key data (type 0x39)
├─ ExePrivilegeCtrl() → forward privilege data to Java
├─ ViewerConfig() → store session_id + config
└─ SendKickRequest() → kick user (type 0x38)
RMDecoder
├─ ASTVideoDecoder (0x57, 0x58)
│ ├─ ast2100 (AST2050 — VQ + Huffman + IDCT)
│ └─ ast_jpeg (AST2100+ — JPEG variant + YUV→RGB)
├─ HermonVideoDecoder (0x59)
├─ YarkonVideoDecoder (0x60)
└─ Pilot3VideoDecoder (0x61)
RFBKMCryto
└─ AES-128-CBC encryption for mouse events only (keyboard is always cleartext)
RMConnection
├─ ConnKeepActive() → establish connection
├─ CheckVaildUser() → authenticate
└─ QosControl() → QoS settings
RMDesktop (composition)
├─ RFBKeyboard (offset 0x00)
├─ RFBMouse (offset 0x08)
├─ RFBScreen (offset 0x10)
└─ RFBPrivilege (offset 0x18)
Wire Format Summary
All multi-byte values are big-endian (network byte order).
Client-to-Server Quick Reference
KeyEvent (18 bytes, always cleartext):
04 00 [down] 00 00 [keycode:4] 00 00 00 00 00 00 00 00 00
PointerEvent unencrypted (18 bytes):
05 00 [buttons] [x:2] [y:2] 00 00 00 00 00 00 00 00 00 00 00
PointerEvent encrypted (18 bytes):
05 01 [aes_encrypted:16]
FramebufferUpdateRequest (10 bytes):
03 [incremental] [x:2] [y:2] [w:2] [h:2]
MouseSync (3 bytes):
07 07 80 (hardcoded 0x0780 = 1920)
MouseReset (2 bytes):
08 [mouse_type]
ScreenSetPosition (9 bytes):
15 [x_pos:4] [y_pos:4]
KeepAlive ACK (2 bytes):
16 01 (always ack=1 in ATEN implementation)
ScreenCalibration (2 bytes):
17 01 (flag always 0x01)
GetCursorPos (1 byte):
19
SetPowerOnOff (2 bytes):
1A [action] (0=off, 1=on, 2=reset, 3=soft-off)
ScreenSetInfo (5 bytes):
32 [width:2] [height:2]
MouseSetPT (3 bytes):
36 [mouse_mode] [mouse_type]
KeyboardUpdateInfo (1 byte):
35 (request keyboard+mouse config from server)
MouseUpdateInfo (1 byte):
37 (request fresh mouse config from server)
SendKickRequest (73 bytes):
38 [action:4] [reserved:4=0] [data:64]
MouseHotPlug (1 byte):
3A
QoS (13 bytes):
3B [param1:4] [param2:4] [param3:4]
GetScreenUI (1 byte):
3C
SetScreenUILang (5-9 bytes):
3D [lang_id:4] [lang_sub:4 if FW flag]
Server-to-Client Quick Reference
FramebufferUpdate (type 0x00):
00 [pad:3] [x:2] [y:2] [w:2] [h:2] [encoding:4] [frame:4] [len:4] [data:len]
CursorPosition (type 0x04):
04 [cursor_x:4] [cursor_y:4] [width:4] [height:4] [cursor_type:4]
If cursor_type == 1: [extra:4] [pixels:width*height*2]
KeepAlive (type 0x16):
16 [status:1]
KeyboardInfo + MouseInfo (type 0x35):
35 [kb_encrypt] [kb_type] [mouse_encrypt] [mouse_mode] [mouse_config]
MouseInfo (type 0x37):
37 [mouse_encrypt] [mouse_mode] [mouse_config]
PrivilegeInfo (type 0x39):
39 [session_lo:4] [session_hi:4] [key_data:256]
GetScreenUILang (type 0x3C):
3C [lang_id:4] [lang_sub:4]
RFBScreen Internals
RFBScreen Object Layout
The RFBScreen object is large (~0x2080 bytes) and contains the framebuffer management state, cursor data, and decoder references:
Offset Size Type Field Notes
0x00 8 vtable* vtable ptr Points to 0x0022f8d0
0x08 8 RMProtocol* protocol Network stream access
0x10 4 int screen_width Current width in pixels
0x14 4 int screen_height Current height in pixels
0x18 4 int stored_width Width for change detection
0x1C 4 int stored_height Height for change detection
0x20 4 uint encoding_type Current encoding (0x57-0x61)
0x28 8 uint64 update_blocks Dirty rect tracking
0x30-0x37 ... (screen info fields)
0x38 8 uchar* framebuffer Main decoded pixel buffer
0x40 8 uchar* compressed_data_buf Receive buffer for encoded data
0x48 8 uchar* cursor_buffer Cursor pixel data (Java ByteBuffer)
0x50 4 int frame_number Frame sequence counter
0x54 1 byte resolution_changed Flag: need to notify Java
0x55 1 byte cursor_data_valid Flag: cursor data available
0x56 varies uchar[] cursor_pixel_data Inline cursor image storage
0x2058 4 uint cursor_x Hardware cursor X position
0x205C 4 uint cursor_y Hardware cursor Y position
0x2060 4 int cursor_width Cursor image width
0x2064 4 int cursor_height Cursor image height
0x2070 4 uint cursor_extra Additional cursor info
0x2078 8 RMDecoder* current_decoder Active video decoder instance
ScreenDecode (0x00119090) -- Video Frame Receive
This function handles incoming FramebufferUpdate (type 0x00) messages. The ProtocolHandler reads the type byte; ScreenDecode reads the rest:
- Skip 3 bytes (standard RFB padding + num_rects, always 1)
- Read x_position (u16), y_position (u16)
- Read width (i16) -> stored at this+0x10; height (i16) -> stored at this+0x14
- Read encoding_type (u32) -> stored at this+0x20
- Call
RMDecoder::GetDecoder(compressed_data_buf, framebuffer)to get/create the appropriate decoder. This is a singleton -- only one decoder is ever created. If encoding_type is unknown (not 0x57-0x61), returns NULL and no decoding occurs. - Store x, y, width, height, encoding into the decoder's info struct (offsets 0x08-0x18)
- Read frame_number (u32):
- If frame_number == 1 and this is the first frame (this+0x50 == 0): set resolution_changed flag (this+0x54 = 1), clear cursor flag (this+0x55 = 0)
- Store frame_number at this+0x50
- Read data_length (u32)
- If data_length > 0:
- Make width/height absolute (negate if negative)
- Read data_length bytes into the compressed_data_buf (this+0x40)
- Clear update_blocks (this+0x28 = 0)
- If frame_number == 0 (not first frame): call decoder->Decode(UpdateBlocks_t) to decompress update stored_width/height from decoder's output dimensions
- If frame_number != 0 (first frame): set decoder's "deferred decode" flag (decoder offset 0x1c = 1)
ScreenUpdate (0x00118850) -- Send FramebufferUpdateRequest
Sends a standard RFB FramebufferUpdateRequest (type 0x03). The
ScreenReqInfo_t parameter struct is passed by value on the stack:
ScreenReqInfo_t layout (on stack):
offset 0x00: u16 x_position
offset 0x04: u16 y_position
offset 0x08: u16 width
offset 0x0C: u16 height
offset 0x10: u8 incremental_flag
Wire bytes:
- Write type 0x03
- Write incremental flag (u8) -- 0=full update, 1=incremental
- Write x (u16), y (u16), width (u16), height (u16)
- Flush
- If resolution_changed flag (this+0x54) is set:
- Call
MixedCursor()through vtable[0x60] to composite cursor - Clear the resolution_changed flag (this+0x54 = 0)
- Call
ScreenGetFreame (0x00118d40) -- Get Framebuffer Pointer
Simply returns the framebuffer pointer at this+0x38. This is the decoded
pixel data that the Java side maps as a DirectByteBuffer.
ScreenGetRect (0x00118a20) -- Copy Rectangle from Framebuffer
Copies a rectangular region from the framebuffer (this+0x38) to an output
buffer. Performs bounds checking against screen_width and screen_height.
Copies row by row using memcpy, with stride = screen_width * 4 (32bpp).
ScreenSetFrameBuff (0x00118d50) -- Set Framebuffer Pointers
Sets the framebuffer pointer (this+0x38) only if it hasn't been set yet (is NULL). Always sets the cursor buffer pointer (this+0x48). Called during initialization to establish the shared memory between Java ByteBuffers and the native decoder.
ScreenCursorPosProc (0x00118fb0) -- Cursor Position Update
Handles server message type 0x04. Reads:
- cursor_x (u32) -> this+0x2058
- cursor_y (u32) -> this+0x205C
- cursor_width (u32) -> this+0x2060
- cursor_height (u32) -> this+0x2064
- cursor_type (u32):
- If cursor_type == 1: reads cursor_extra (u32), then reads
width * height * 2bytes of 16bpp cursor pixel data into this+0x56 Sets cursor_data_valid flag
- If cursor_type == 1: reads cursor_extra (u32), then reads
- If frame_number indicates first frame received (this+0x50 != 1):
- Calls MixedCursor() through vtable to composite cursor onto framebuffer
- Calls getQuickCursor() JNI callback to notify Java
GeFrontGround (0x00118da0) -- Get Cursor Composite Buffer
Returns the cursor buffer (this+0x48) with metadata prepended:
- Writes frame_number, cursor_x, cursor_y, cursor_width, cursor_height (byte-swapped to big-endian) into the first 20 bytes of the cursor buffer
- If cursor data is valid and frame_number != 0: allocates a temporary buffer, copies the cursor rect via ScreenGetRect, calls decoder's MixedCursor(), then frees the temp buffer
- Returns pointer to the cursor buffer (or 0 if no framebuffer)
Network I/O Layer
NtwStream Object Layout (0x628 bytes)
Confirmed from constructor at 0x0010efb0:
Offset Size Field Description
0x000 0x28 pthread_mutex_t mutex Write mutex (pthread_mutex_init)
0x028 0x5F0 uint8_t write_buf[1520] Internal write buffer
0x618 0x08 uint8_t *write_cursor Current write position (init: this+0x28)
0x620 0x08 TcpSocket *socket Underlying socket object
Reads are NOT buffered -- every StreamRead call goes directly to
TcpSocket::read() via virtual dispatch (vtable +0x18). The 1520-byte buffer
is exclusively for writes.
Write batching: StreamWriteStart() locks the mutex, subsequent
StreamWrite*() calls accumulate into the buffer via memcpy, and
StreamWriteFlush() sends all buffered data in a single write() call and
unlocks the mutex. This ensures protocol messages are sent atomically.
If a StreamWrite() would overflow the buffer, the existing buffer is flushed
first, then the data is sent directly to the socket.
StreamReadSkip/StreamWriteSkip: Both allocate a temporary heap buffer, zero-fill it, read/write through it, then free it. Wasteful but functional.
Endianness: All StreamRead16/32 and StreamWrite16/32 functions use
big-endian (network byte order). Confirmed from decompilation:
StreamRead32returns(b0 << 24) | (b1 << 16) | (b2 << 8) | b3StreamWrite32writesval>>24, val>>16, val>>8, val
TcpSocket Object Layout (0x60 bytes)
Confirmed from constructor at 0x00117290:
Offset Size Field Description
0x00 0x08 void **vtable Virtual function table
0x08 0x34 char hostname[52] Hostname (copied via strcpy)
0x3C 0x04 int port Port number
0x40 0x04 int record_flag Debug: write recv'd data to file
0x44 0x04 int playback_flag Debug: read from file instead of socket
0x48 0x08 FILE *record_file Recording file handle
0x50 0x08 FILE *playback_file Playback file handle
0x58 0x04 int socket_fd Connected socket file descriptor
0x5C 0x04 int is_server 1=server (listen/accept), 0=client
Socket options set during connect (CreateScok at 0x001179c0):
SO_SNDBUF = 0x40000(256 KB)SO_RCVBUF = 0x40000(256 KB)- No TCP_NODELAY --
enableNagles()is a no-op stub (returns 1) - No SO_KEEPALIVE set
Connection uses getaddrinfo() + socket() + connect() with
ai_socktype = SOCK_STREAM, ai_flags = AI_PASSIVE.
Network error handling: Both TcpSocket::read() and write() use
select() with a 30-second timeout (tv_sec = 0x1e). On error or
timeout, they throw C++ exceptions (ErrMsg, 0x44 bytes: int error_code +
char[64] message):
| Error Code | Message | Condition |
|---|---|---|
| -1 | "Read: Socket closed" | recv() returns 0 or negative |
| -1 | "Socket Read Failed" | select() returns negative |
| -2 | "Socket Read Timeout" | select() timeout (30s) |
| -1 | "Write: Socket closed" | send() returns negative |
| -2 | "Socket Write Timeout" | select() timeout (30s) |
SIGPIPE (signal 13) is handled by BrokenPipe_handle which prints a message
and returns (prevents JVM crash on broken pipe).
Connection Teardown
There is NO graceful disconnect message. The connection is torn down by:
Java_..._RemoteVideo_destory()at 0x0011ffd0 (note: typo "destory")- Nulls the global
desktopandconnectionpointers - Destroys
RMDesktop-> destroys sub-objects (keyboard, mouse, screen, privilege) - Destroys
RMConnection-> destroysRFBProtocol-> destroysNtwStream NtwStream::~NtwStream-> destroysTcpSocketTcpSocket::EndSock()at 0x00117aa0:shutdown(socket_fd, SHUT_RDWR)(2 = both directions)close(socket_fd)
A client implementation can simply close the TCP socket. No protocol-level goodbye message is needed.
Notable Implementation Details
PsudoStreamSwap16()performs byte-swapping for 16-bit values (host→network or network→host order conversion)- The native code uses
XkbGetIndicatorState()to read X11 keyboard LED state for CapsLock/NumLock/ScrollLock synchronization - The
processVK()function uses threestd::map<int, uint>lookup tables to translate VK codes to X11 keysyms, falling through from table 1 → 2 → 3 - Video frame decode is done in-place into the shared framebuffer memory mapped
by the Java
ByteBuffer - The
sendViewerConfig()function sends a 4-byteg_configvalue and ag_session_idto the Java side viasetViewerConfigcallback - Keyboard events are always cleartext -- only mouse events support AES encryption
- The
GetCursorPos()function at 0x00118d60 sends type 0x19 to request cursor position from the server (an undocumented client-to-server message type) - The
setMouseModeJNI method at 0x001214a0 callsMouseSetPT(vtable offset 0x30) followed byMouseUpdateInfo(vtable offset 0x38) to configure the mouse mode on the server - The
changeLEDstateJNI method at 0x00120cd0 does NOT send any network message. It is purely local X11 manipulation: it opens an X11 display, simulates a key press+release for CapsLock (VK 0x39 -> keysym 0xFFE5), ScrollLock (VK 0x47 -> keysym 0xFF14), or NumLock (VK 0x53 -> keysym 0xFF7F) usingXTestFakeKeyEvent, then closes the display. This synchronizes the client's local keyboard LED state with the remote server's state. - The
updateInfoJNI method at 0x00120be0 callsMouseUpdateInfo()via vtable on the RFBMouse object (desktop+0x08, vtable offset 0x28), which sends a type 0x37 MouseUpdateInfo message to request fresh mouse configuration. - The
changeScreenInfoJNI method at 0x00120c00 callsScreenGetRect()thenScreenSetInfo()via vtable on the RFBScreen object (desktop+0x10). If the resolution has changed,ScreenSetInfosends a type 0x32 message with the new width/height. It also refreshes the JNI frame buffer references. - The
doCatchJNI method at 0x00121570 simply sets the globalfilterFlagvariable and returns. It does NOT process any messages. - The
catchLoopJNI method at 0x001216c0 is a no-op (empty function body). The actual message receive loop is driven byrunImage, notcatchLoop. - The
refreshJNI method at 0x00121510 does JNI buffer management (MonitorEnter/MonitorExit on frameObj) and resets theflagglobal. It does NOT send a FramebufferUpdateRequest on the wire -- that is done by the Java side callingScreenUpdate()through the vtable.
Error Handling
ErrMsg Exception Class
The native library uses C++ exceptions (__cxa_throw) for error propagation.
The ErrMsg class (RTTI at 0x0022df70, typeinfo-name at 0x00122415) is the
single exception type used throughout libiKVM64.so.
ErrMsg structure (0x44 bytes = 68 bytes):
Offset Size Type Field
0x00 4 int error_code
0x04 64 char[] error_message (null-terminated, strncpy'd with max 0x40)
Error Codes
Error codes are negative integers (stored as the first 4 bytes of the ErrMsg):
| Code | Decimal | Source | Message |
|---|---|---|---|
| 0xFFFFFFFF | -1 | TcpSocket::read | "Read: Socket closed" |
| 0xFFFFFFFF | -1 | TcpSocket::read | "Socket Read Failed" |
| 0xFFFFFFFF | -1 | TcpSocket::write | "Write: Socket closed" |
| 0xFFFFFFFF | -1 | TcpSocket::write | "Socket Write Failed" |
| 0xFFFFFFFE | -2 | TcpSocket::read | "Socket Read Timeout" |
| 0xFFFFFFFE | -2 | TcpSocket::write | "Socket Write Timeout" |
| 0xFFFFFFFD | -3 | HermonVideoDecoder::Decode | "Not Support Multi-Session!" |
| 0xFFFFFFFD | -3 | YarkonVideoDecoder::Decode | "Not Support Multi-Session!" |
| 0xFFFFFFFC | -4 | TcpSocket::read | "Read: File closed" |
| 0xFFFFFFFB | -5 | TcpSocket::read | "Read: File maybe curropted" [sic] |
Error Propagation to Java
Errors are propagated from native code to Java via the errorHandler JNI
callback. The callback is registered during RemoteVideo.init with signature
(I)V -- it takes a single integer parameter (the error code).
Registration (in Java_tw_com_aten_ikvm_ui_RemoteVideo_init at 0x0011fbd0):
errorHandlerMid = GetMethodID(class, "errorHandler", "(I)V")
The errorHandlerMid global (at BSS 0x04923730) stores the Java method ID.
If registration fails (returns 0), the init function prints
"Java_tw_com_aten_ikvm_RemoteVideo_init failed" and continues.
Exception flow:
- Socket I/O errors in
TcpSocket::read/writethrowErrMsgexceptions via__cxa_throw(exception, &ErrMsg::typeinfo, 0) - Video decoder errors (e.g., multi-session not supported) also throw
ErrMsg - The
runImageJNI function (0x001201e0) has try/catch blocks (CatchHandler at 0x00120515) that catch these exceptions - The catch handler extracts the error code and calls
errorHandler(code)on the Java RemoteVideo object via JNI CallVoidMethod - The Java
errorHandlermethod handles cleanup (closing connections, showing error dialogs, etc.)
Note: The checkValidUser and keepActive JNI functions also have
try/catch blocks for ErrMsg exceptions at their respective catch handler
addresses (0x00119e36 and 0x00119be8), but their catch handlers return false
rather than calling errorHandler -- the error is indicated by the boolean
return value to the Java caller.
Socket Timeout
The socket timeout for both reads and writes is 30 seconds (tv_sec = 0x1e).
TcpSocket::read and TcpSocket::write use select() with a 30-second
timeout. If select() returns 0 (timeout), an ErrMsg with code -2 is thrown.
If select() returns negative (error), an ErrMsg with code -1 is thrown.
Minimal Client Implementation Guide
This section summarizes the minimum protocol interactions needed for a working KVM client. All multi-byte integers are big-endian (network byte order).
(a) Establishing a KVM Session
1. TCP connect to BMC KVM port (e.g., 5900)
2. Version exchange:
recv(12) -> "RFB 003.008\n"
send(12) -> "RFB 003.008\n"
3. Security negotiation:
recv(1) -> count
recv(count) -> security_type bytes
send(1) -> last security_type received (pick any; ATEN ignores it)
4. Authentication:
recv(24) -> challenge (read and discard)
send(24) -> username (null-padded to 24 bytes)
send(24) -> password (null-padded to 24 bytes)
recv(4) -> auth_result (u32, 0 = success)
If auth_result != 0:
recv(4) -> error_len
recv(error_len) -> error message string
ABORT
5. ClientInit:
send(1) -> 0x00 (shared_flag = exclusive)
6. ServerInit (read and discard):
recv(2) -> fb_width (u16, discard)
recv(2) -> fb_height (u16, discard)
recv(16) -> pixel_format (discard all)
recv(4) -> name_length (u32)
recv(name_length) -> name (discard)
7. ATEN ServerInit extensions:
recv(4) -> skip (padding)
recv(4) -> session_id (u32, store for reference)
recv(1) -> aes_key_seed[0]
recv(1) -> aes_key_seed[1]
recv(1) -> aes_key_seed[2]
recv(1) -> aes_key_seed[3]
8. Server sends PrivilegeInfo (type 0x39):
recv(1) -> 0x39
recv(4) -> session_word_lo (u32)
recv(4) -> session_word_hi (u32)
recv(256) -> aes_key_material (read byte-by-byte or bulk)
(Store these for session management; native AES key is hardcoded)
9. Session is now established. Begin main loop.
(b) Receiving and Decoding Video
The main loop receives server messages and processes them:
LOOP:
recv(1) -> type_byte
SWITCH type_byte:
0x00 (FramebufferUpdate):
recv(3) -> skip (padding + num_rects)
recv(2) -> x (u16)
recv(2) -> y (u16)
recv(2) -> width (i16, take abs)
recv(2) -> height (i16, take abs)
recv(4) -> encoding_type (u32)
recv(4) -> frame_number (u32)
recv(4) -> data_length (u32)
if data_length > 0:
recv(data_length) -> compressed_data
if frame_number == 0:
decode(compressed_data, encoding_type, x, y, width, height)
else:
// First frame: store resolution, defer decode to next frame
// After processing, send FramebufferUpdateRequest for next frame:
send: 03 01 00 00 00 00 [width:2] [height:2]
0x04 (CursorPosition):
recv(4) -> cursor_x (u32)
recv(4) -> cursor_y (u32)
recv(4) -> cursor_width (u32)
recv(4) -> cursor_height (u32)
recv(4) -> cursor_type (u32)
if cursor_type == 1:
recv(4) -> cursor_extra (u32)
recv(cursor_width * cursor_height * 2) -> cursor_pixels (16bpp)
0x16 (KeepAlive):
recv(1) -> status (discard)
// Send ACK:
send: 16 01
0x35 (KeyboardInfo + MouseInfo):
recv(1) -> kb_mode
recv(1) -> kb_type
recv(1) -> mouse_mode (0=normal, 1=absolute, 2=single)
recv(1) -> mouse_type
recv(1) -> mouse_extra (encryption flag)
0x37 (MouseInfo):
recv(1) -> mouse_mode
recv(1) -> mouse_type
recv(1) -> mouse_extra
0x39 (PrivilegeInfo):
recv(4) -> session_word_lo
recv(4) -> session_word_hi
recv(256) -> aes_key_material
0x3C (GetScreenUILang):
recv(4) -> lang_id
recv(4) -> lang_sub
DEFAULT (types > 0x3C or unhandled):
// Unknown type: the native code returns the type byte to the
// caller without consuming further data. This is DANGEROUS --
// if the server sends an unexpected type, the stream will
// desynchronize. A robust client should disconnect on unknown types.
Encoding types (the encoding_type field in FramebufferUpdate):
| Encoding | Decoder | BMC Chipset | Notes |
|---|---|---|---|
| 0x57 | AST2050 (VQ+DCT) | ASPEED AST2050 | Optional RC4 decryption layer |
| 0x58 | AST JPEG | ASPEED AST2100+ | Most common; JPEG-like 4:2:0 |
| 0x59 | Hermon | Nuvoton WPCM450 | Raw 16bpp/8bpp tiles |
| 0x60 | Yarkon | Yarkon | Huffman/RLE + Hextile |
| 0x61 | Pilot3 | Pilot III | Multi-mode RLE + planar |
For most modern ATEN/ASPEED BMCs, expect encoding 0x58. See the "AST JPEG Wire Format" section for full decoder details.
(c) Sending Keyboard Input
Keyboard events are always sent in cleartext (never encrypted):
send: 04 00 [down_flag:1] 00 00 [keycode:4] 00 00 00 00 00 00 00 00 00
Total: 18 bytes. The keycode is a USB HID keycode (e.g., 0x28 for Enter,
0x04 for 'A'). The down_flag is 0 for key-up and 1 for key-down.
Example -- pressing and releasing the 'a' key (HID keycode 0x04):
send: 04 00 01 00 00 00 00 00 04 00 00 00 00 00 00 00 00 00 (key down, keycode=0x04)
send: 04 00 00 00 00 00 00 00 04 00 00 00 00 00 00 00 00 00 (key up, keycode=0x04)
(d) Sending Mouse Input
Mouse events support two formats. In normal mode (mouse_mode == 0), events are unencrypted:
send: 05 00 [button_mask:1] [x:2] [y:2] 00 00 00 00 00 00 00 00 00 00 00
Total: 18 bytes. Button mask bits: 0x01=left, 0x02=middle, 0x04=right.
In absolute/single mode (mouse_mode != 0), events are AES-encrypted:
Plaintext (16 bytes): [button_mask:1] [x:2 big-endian] [y:2 big-endian] [random:11]
Encrypt with AES-128-CBC:
Key: 2B 7E 15 16 28 AE D2 A6 AB F7 15 88 09 CF 4F 3C
IV: 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
send: 05 01 [ciphertext:16]
Total: 18 bytes either way.
Mouse wheel: For each wheel tick, send TWO PointerEvents (press + release) with the appropriate scroll button bit set in button_mask.
Object Allocation Sizes (for Implementation Reference)
All sizes verified from operator_new calls in constructors:
| Class | Alloc Size | Hex | Notes |
|---|---|---|---|
| TcpSocket | 96 | 0x60 | Underlying TCP socket |
| NtwStream | 1,576 | 0x628 | Buffered network I/O (includes mutex, 1520-byte write buffer) |
| RFBProtocol | 112 | 0x70 | Protocol state, sub-object pointers |
| RMConnection | 16 | 0x10 | Vtable + RFBProtocol pointer |
| RMDesktop | 48 | 0x30 | Composition of keyboard+mouse+screen+privilege |
| RFBKeyboard | 184 | 0xB8 | Includes 3 std::map lookup tables |
| RFBMouse | 40 | 0x28 | Includes RFBKMCryto pointer |
| RFBScreen | 8,320 | 0x2080 | Large: cursor data, framebuffer refs |
| RFBPrivilege | 288 | 0x120 | 256-byte key material + RFBKMCryto pointer |
| RFBKMCryto | 8,464 | 0x2110 | AES state: key schedule, S-boxes |
| ErrMsg | 68 | 0x44 | Exception: int error_code + char[64] |
Additionally, the RFBScreen constructor allocates a 0x600000-byte (6 MB) compressed data buffer for incoming encoded video frames.
Disconnection
There is no graceful disconnect message in this protocol. To disconnect:
- Simply close the TCP socket (shutdown + close)
- The server detects the disconnect via TCP RST/FIN
The server will also disconnect the client if KeepAlive ACKs stop arriving (timeout behavior depends on BMC firmware implementation).
Server-Side Findings (from BMC ikvmserver binary)
The following information was reverse-engineered from the ikvmserver binary
extracted from Supermicro X9 BMC firmware (SMT_X9_339.bin). The server is an
ARM 32-bit LE ELF binary compiled with GCC 3.4.4, running on a Nuvoton
WPCM450 ("Hermon") BMC SoC.
Server Architecture
┌──────────────────────────────────────────────────────┐
│ ikvmserver (userspace daemon, port 5900 default) │
│ - C++ with virtual dispatch (Stream, Protocol, Auth)│
│ - Multi-session: linked list at DAT_0001e58c │
│ - Per-session thread with POSIX message queues │
│ - Buffered I/O: 0x5F0 byte write buffer per stream │
├──────────────────────────────────────────────────────┤
│ Shared Libraries │
│ - libipmicrypt.so: Custom AES (not OpenSSL) │
│ - libutility.so: Auth, power control, IPMI ops │
│ - libldap_client.so / libradius_client.so: Auth │
│ - libsys.so: Shared memory (GlobalVar, NVRAM, PS) │
├──────────────────────────────────────────────────────┤
│ Kernel Modules │
│ - vcddev.ko: Video Capture Device (/dev/vcd) │
│ - usb_hid.ko: Virtual USB HID (/dev/keyboard, │
│ /dev/mouse) by "ATEN, Bobby" │
│ - ikvm_vmass.ko: Virtual USB mass storage │
├──────────────────────────────────────────────────────┤
│ Hardware: Nuvoton WPCM450 SoC │
│ - VCD (Video Capture/Differentiation) engine │
│ - USB 2.0 Device Controller (gadget mode) │
└──────────────────────────────────────────────────────┘
Server Session Lifecycle
-
Startup: Reads port from persistent storage (
at_p_St_PS + 0x1b42), defaults to 5900 if empty. Registers SIGUSR1 handler for session kill (reads/tmp/killsess), SIGUSR2 for preview capture. -
Accept loop (
FUN_0000acbc): Spawns a worker thread per connection. Maintains a linked list of active sessions. -
Per-session thread (
FUN_0000d334):- Creates Stream object (0x610 bytes, buffered I/O wrapper)
- Creates RFBProtocol object (0x28 bytes, virtual dispatch)
- Calls
InitHandShake()→Authenticate()→ mainProtocolHandler()loop - On disconnect: unregisters session, notifies peers
Server Message Dispatch (Client-to-Server)
The server's ProtocolHandler (FUN_000108b8) reads a 1-byte type, checks
permissions, and dispatches. Messages that fail permission checks are silently
consumed (skipped by reading the known payload length).
Complete server dispatch table (types handled by the server):
| Type | Hex | Payload | Server Handler | Description |
|---|---|---|---|---|
| 3 | 0x03 | 9 | FUN_0000f610 | FramebufferUpdateRequest |
| 4 | 0x04 | 17 | FUN_0000f8c4 | KeyEvent (supports AES decrypt) |
| 5 | 0x05 | 17 | FUN_0000f7d0 | PointerEvent (supports AES decrypt) |
| 7 | 0x07 | 2 | FUN_0000f5dc | ClientCutText |
| 8 | 0x08 | 1 | FUN_0000eb3c | SetEncoding |
| 20 | 0x14 | 2 | FUN_0000f55c | VideoControl (set FPS) |
| 21 | 0x15 | 8 | FUN_0000f7b0 | 2×u32 (read and discarded) |
| 22 | 0x16 | 1 | FUN_0000ef10 | KeepAlive ACK (echoed back) |
| 24 | 0x18 | 2 | FUN_0000f4dc | VideoControl2 |
| 25 | 0x19 | 0 | FUN_0000ef58 | GetCursorPosition → sends 0x04 |
| 26 | 0x1a | 1 | FUN_0000fae4 | PowerControl |
| 50 | 0x32 | 4 | FUN_0000f44c | VideoControl3 (set both FPS) |
| 51 | 0x33 | 0 | FUN_0000f0cc | GetVideoInfo → sends 0x33 |
| 52 | 0x34 | 2 | FUN_0000eb78 | SetEncoding2 |
| 53 | 0x35 | 0 | FUN_0000eeac | GetKbdMouseInfo → sends 0x35 |
| 54 | 0x36 | 2 | FUN_0000ebcc | SetEncoding3 |
| 55 | 0x37 | 0 | FUN_0000f994 | GetSessionInfo → sends 0x37 |
| 56 | 0x38 | 72 | FUN_0000fa54 | SendKickRequest |
| 58 | 0x3a | 0 | FUN_0000ec20 | RequestRefresh |
| 59 | 0x3b | 12 | FUN_0000fa10 | SetBandwidth (3×u32) |
| 60 | 0x3c | 0 | FUN_0000ee5c | GetViewerLang → sends 0x3c |
| 61 | 0x3d | 8 | FUN_0000faa4 | SetViewerLang (2×u32, saves to /nv/IKVMViewerLang) |
| 62 | 0x3e | 0 | FUN_0000ee0c | GetSessionStatus → sends 0x3e |
Unknown types print "ProtocolHandler: Error message type !!" and continue.
Permission filter (FUN_0000f130): Each message type is checked against one
of four privilege bytes (from ServerInit, stored at session offsets 0x155-0x158):
- Privilege[0] (video): types 3, 0x14, 0x15, 0x18, 0x19, 0x32, 0x33
- Privilege[1] (keyboard/mouse): types 4, 5, 7, 8, 0x34, 0x35, 0x36, 0x37, 0x3a
- Privilege[2] (kick/admin): types 0x38, 0x39
- Privilege[3] (power): type 0x1a (implicit, handler checks directly)
When a message is filtered, the server reads and discards the known payload
length from FUN_0000ec40 (the "Payload" column above).
Server-to-Client Message Formats (Server-Verified)
Type 0x00: FramebufferUpdate (from FUN_0000f610):
u8 type = 0x00
u8 padding = 0x00
u16 num_rects = 1 (always 1, big-endian)
u16 x = 0 (always 0)
u16 y = 0 (always 0)
u16 width (from video capture)
u16 height (from video capture)
u32 encoding_type (ATEN proprietary: 0x57-0x61)
u32 sub_encoding (client interprets as frame_number: 1=first, 0=subsequent)
u32 data_length
u8[data_length] compressed pixel data
Note: The server explicitly writes num_rects = 1 as a u16 (not just padding).
The client's StreamReadSkip(3) consumes the padding byte + num_rects together.
Type 0x04: CursorPosition (from FUN_0000ef58):
u8 type = 0x04
u32 cursor_field_1 (cursor type/visibility)
u32 cursor_field_2 (x position or hotspot)
u32 cursor_field_3 (y position or hotspot)
u32 cursor_field_4 (width or shape)
u32 cursor_field_5 (0=position only, 1=has cursor bitmap data)
[variable cursor bitmap data if cursor_field_5 == 1]
Type 0x16: KeepAlive Echo (from FUN_0000ef10):
The server handler for client type 0x16 ECHOES it back:
u8 type = 0x16
u8 status (echoed from client's KeepAlive ACK)
After echoing, the server broadcasts the status to all other connected
clients via POSIX message queues (FUN_0000fb70).
Type 0x33: VideoInfo (from FUN_0000f0cc, response to client 0x33):
u8 type = 0x33
u16 video_info_1 (current FPS or resolution width)
u16 video_info_2 (resolution height or video mode)
Type 0x35: KbdMouseInfo (from FUN_0000eeac, response to client 0x35):
u8 type = 0x35
u8 mouse_mode
u8 keyboard_mode
Note: The server sends only 2 bytes after the type for this response
(3 bytes total). This is fewer than the 5-byte payload described in the
client-side analysis. The client's ProcKeyboardInfo reads 2 bytes and
ProcMouseInfo reads 3 bytes (total 5 bytes after type). This discrepancy
may indicate the client expects a different format than what this server
version sends, or the additional bytes come from a different code path.
Type 0x37: SessionInfo (from FUN_0000f994, response to client 0x37):
u8 type = 0x37
u8 session_field_1
u8 session_field_2
u8 power_status (from UtilGetNowPowerStatus())
Type 0x39: PrivilegeInfo (from FUN_0001069c, sent proactively each loop):
u8 type = 0x39
u32 sequence_count (number of remaining privilege updates in queue)
u32 command_code
u8[256] command_data
Command codes: 0 = disconnect (no message, immediate), 4 = kick (sends PrivilegeInfo then disconnects), others = session events (user join/leave, etc.)
Type 0x3C: ViewerLang (from FUN_0000ee5c):
u8 type = 0x3C
u32 lang_id
u32 lang_sub
Type 0x3E: SessionStatus (from FUN_0000ee0c):
u8 type = 0x3E
u8 status
Server AES Implementation (Verified)
The server imports aes_set_key, aes_encrypt, aes_decrypt from
libipmicrypt.so — a custom AES implementation (matches XySSL/PolarSSL
source), NOT OpenSSL.
Key/IV from server binary (at DAT_00014ce0 and DAT_00014cf0):
Identical to the client-side hardcoded values:
- Key:
2b 7e 15 16 28 ae d2 a6 ab f7 15 88 09 cf 4f 3c(NIST FIPS-197 Appendix B) - IV:
00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f - Mode: AES-128-CBC, single block (16 bytes), fresh IV copy each call
The server decrypts both KeyEvent (type 4) and PointerEvent (type 5) when the encryption flag byte is non-zero. This contradicts the client-side finding that keyboard events are "never encrypted" — the server is prepared to decrypt encrypted keyboard events, even though the client never sends them.
AES decrypt wrapper (FUN_00011268):
memcpy(iv, 0x14ce0, 16); // static IV
memcpy(key, 0x14cf0, 96); // NIST key material (only first 16 bytes used)
aes_cbc(1/*decrypt*/, 0/*AES-128*/, ciphertext, 1/*block*/, key, output, iv);
AES encrypt wrapper (FUN_000112e8): Same key/IV, direction=0.
Key size formula: key_size_param * 64 + 128 bits. The wrappers pass
key_size_param=0 → 128-bit AES.
Server KeepAlive Model
The KeepAlive mechanism is bidirectional:
- Client sends type 0x16 with a status byte
- Server echoes type 0x16 + same status byte back to the sending client
- Server broadcasts the status to all other connected clients via message queues
This differs from the pure "server pings, client ACKs" model described in the client-side analysis. Both directions appear to be valid.
Server Key File Paths
| Path | Purpose |
|---|---|
/dev/vcd |
Video capture device (Nuvoton VCD engine) |
/dev/keyboard |
Virtual USB HID keyboard |
/dev/mouse |
Virtual USB HID mouse |
/tmp/killsess |
Session kill signal (-1=all, -2=port change, N=thread ID) |
/tmp/ikvmMaxSess |
Maximum session tracking |
/tmp/sess_<username> |
Web session authentication files |
/nv/MouseMode |
Persistent mouse mode setting |
/nv/IKVMViewerLang |
Viewer language preference |
/tmp/Snapshot.bmp |
Video preview snapshot |
Server Session Kill Mechanism
The server handles session termination via SIGUSR1 → reads /tmp/killsess:
- Value
-1: Kill all sessions - Value
-2: Kill due to port change - Other: Match thread ID, send command=3 ("Kill Session Due to Web Logout")
The init script (/etc/init.d/ikvmd) stops the server by writing -2 to
/tmp/killsess and then killall -9 ikvmserver.
Discrepancies Between Client and Server
| Area | Client-Side Doc | Server Binary | Notes |
|---|---|---|---|
| Security type | Example shows 0x02 | Always 0x10 (16) | Server is authoritative |
| Keyboard AES | "Never encrypted" | Decrypt path exists for type 4 | Server supports it; client doesn't use it |
| Type 0x35 S2C size | 6 bytes (type + 5) | 3 bytes (type + 2) | Possible version mismatch or different code path |
| Type 0x37 S2C | MouseInfo (3 data bytes) | SessionInfo (3 data bytes + power) | Different interpretation of same-length message |
| KeepAlive direction | Server initiates | Bidirectional (echo) | Both models may coexist |
| PrivilegeInfo seq field | session_word_lo | sequence_count | Server sends remaining queue count |
| FramebufferUpdate "frame_number" | 1=first, 0=subsequent | "sub_encoding" from video vtable | Same field, different semantic name |