aten-ipmi-tools/KVM_PROTOCOL.md

ATEN iKVM RFB Protocol Specification

Reverse-engineered from iKVM__V1.69.21.0x0 (Java) and libiKVM64.so (native x86-64).

Overview

The ATEN iKVM uses a modified RFB (Remote Framebuffer) protocol, version 3.8, for KVM (Keyboard, Video, Mouse) console access over TCP. It extends standard VNC/RFB with:

• ATEN-proprietary video encodings for BMC chipset-specific JPEG-like compression
• Extended keyboard/mouse messages (18 bytes instead of standard 8/6)
• Optional AES-128-CBC encryption for mouse events (keyboard is always cleartext)
• Privilege control, power management, and QoS extensions
• Hardware cursor position tracking

The protocol runs over a single TCP connection to the BMC's KVM port (configurable, typically set via the IPMI web interface).

Architecture

┌──────────────────────────────────────────────────────┐
│  Java GUI (RemoteVideo, Viewer)                      │
│  - AWT/Swing keyboard/mouse capture                  │
│  - BufferedImage framebuffer (32bpp, max 1920x1200)  │
│  - HID keycode translation (KeyMap)                  │
├──────────────────────────────────────────────────────┤
│  JNI Bridge                                          │
│  - RMConnection: init, keepActive, checkValidUser    │
│  - RemoteVideo: native methods (44 functions)        │
├──────────────────────────────────────────────────────┤
│  Native Library (libiKVM64.so)                       │
│  - RFBProtocol: handshake, message dispatch          │
│  - RFBScreen: video decode, framebuffer management   │
│  - RFBKeyboard: key translation, send                │
│  - RFBMouse: pointer events, send                    │
│  - RFBKMCryto: AES-128-CBC for input encryption      │
│  - RFBPrivilege: session/privilege management         │
│  - NtwStream: buffered network I/O                   │
│  - Video decoders: AST, Hermon, Yarkon, Pilot3       │
├──────────────────────────────────────────────────────┤
│  TCP Socket (TcpSocket)                              │
└──────────────────────────────────────────────────────┘

Connection Lifecycle

Client                                    BMC (Server)
  │                                          │
  │──────── TCP connect (KVM port) ─────────>│  InitHandShake
  │                                          │
  │<──────── "RFB 003.008\n" ───────────────│  ProcVersion
  │──────── "RFB 003.008\n" ───────────────>│
  │                                          │
  │<──── security_types_count(1) ───────────│  ProcSecurity
  │<──── security_type(1) × N ─────────────│
  │──── selected_type(1) ──────────────────>│
  │                                          │
  │<──── challenge(24) ────────────────────│  Authenticate
  │──── username(24) ─────────────────────>│  (ATEN plaintext auth)
  │──── password(24) ─────────────────────>│
  │<──── auth_result(4) ───────────────────│  0 = success
  │                                          │
  │──── client_init: shared_flag=0x00(1) ──>│  ProcClientInit
  │                                          │
  │<──── ServerInit (standard RFB) ────────│  ProcServerInit
  │<──── ATEN extensions (session+config) ─│
  │                                          │
  │<──── PrivilegeInfo (type 0x39) ────────│  First server message
  │                                          │
  │──── FramebufferUpdateRequest ──────────>│  Main loop begins
  │<──── FramebufferUpdate (type 0x00) ────│
  │<──── CursorPosition (type 0x04) ───────│
  │──── KeyEvent / PointerEvent ───────────>│
  │<──── KeyboardInfo+MouseInfo (0x35) ────│
  │<──── KeepAlive (type 0x16) ────────────│
  │──── KeepAlive ACK ─────────────────────>│
  │         ... (repeat) ...                 │

Handshake

The handshake is split into two phases:

1. InitHandShake (RFBProtocol::InitHandShake at 0x00117fa0): Creates the NtwStream, establishes the TCP connection, then runs ProcVersion + ProcSecurity.
2. Authenticate (RFBProtocol::Authenticate at 0x00118190): Performs the ATEN-specific username/password authentication, then runs ProcClientInit + ProcServerInit.

Version Exchange

Server sends exactly 12 bytes: RFB 003.008\n

Client reads this, parses RFB %03d.%03d\n, verifies major=3, minor=8, then echoes back the identical 12-byte string.

Security Negotiation

Server sends:

┌──────────────────────┐
│ count: u8            │  Number of security types
├──────────────────────┤
│ types: u8[count]     │  Available security type IDs
└──────────────────────┘

Client selects a security type (the implementation always picks the last one offered) and sends it back as a single byte.

The native code does not implement any security-type-specific logic (e.g., VNC DES challenge-response). It blindly selects the last type offered and sends it back. The actual authentication is ATEN-proprietary (see below), not standard RFB security types. The security type negotiation appears to be a formality to satisfy the RFB 3.8 protocol structure.

ATEN Authentication

After security negotiation, the Authenticate function performs ATEN-specific plaintext credential authentication. Important: On success, Authenticate also runs ProcClientInit and ProcServerInit internally -- from the Java caller's perspective, checkValidUser() returning true means the entire handshake (including ClientInit, ServerInit, and ATEN privilege setup) is already complete.

Client                                    BMC (Server)
  │                                          │
  │<──── challenge: u8[24] ──────────────────│  Server sends 24-byte challenge
  │                                          │
  │──── username: u8[24] ───────────────────>│  Client sends username (null-padded)
  │──── password: u8[24] ───────────────────>│  Client sends password (null-padded)
  │                                          │
  │<──── auth_result: u32 (big-endian) ─────│  0 = success, non-zero = failure
  │                                          │
  │  If auth_result != 0:                    │
  │<──── error_len: u32 (big-endian) ───────│
  │<──── error_msg: u8[error_len] ──────────│
  │  Connection should be closed.            │
  │                                          │
  │  If auth_result == 0:                    │
  │──── ClientInit ─────────────────────────>│  (sent inside Authenticate)
  │<──── ServerInit ────────────────────────│  (read inside Authenticate)
  │                                          │

Wire format:

┌─────────────────────────────────────────────┐
│ From server:                                │
│   challenge:    u8[24]                      │  24-byte challenge/nonce (discarded)
├─────────────────────────────────────────────┤
│ From client (sent atomically via buffered   │
│ write -- both in one StreamWriteFlush):     │
│   username:     u8[24]                      │  Null-padded username
│   password:     u8[24]                      │  Null-padded password
├─────────────────────────────────────────────┤
│ From server:                                │
│   auth_result:  u32 (big-endian)            │  0 = success
│ If auth_result != 0:                        │
│   error_len:    u32 (big-endian)            │  Length of error message
│   error_msg:    u8[error_len]               │  Error description string
└─────────────────────────────────────────────┘

Verified from decompilation of RFBProtocol::Authenticate at 0x00118190: The Userinfo_t struct (48 bytes, passed by value) contains username at offset 0x00 (24 bytes) and password at offset 0x18 (24 bytes). The JNI wrapper checkValidUser at 0x00119c60 copies credentials from Java UserInfo fields using strcpy into local buffers (username: 24 bytes, password: 96 bytes on stack but only 24 bytes sent on wire).

Security note: The username and password are sent in plaintext (no encryption, no hashing). The 24-byte challenge from the server is read into a local stack buffer and never referenced again -- it is genuinely discarded (confirmed by binary analysis: no DES, no XOR, no hash of the challenge). Maximum username length is 24 bytes; maximum password length is also 24 bytes (both truncated and null-padded to exactly 24 bytes). The error message on auth failure is read but also discarded by the native code -- Java shows a hardcoded "Authentication failed" dialog instead.

Step-by-Step Wire Bytes for Full Handshake

For client implementors, here is the exact byte sequence from TCP connect through the first FramebufferUpdateRequest. All multi-byte integers are big-endian unless noted.

Step 1: TCP connect to BMC KVM port (e.g., 5900)
  → (no application data, just SYN/SYN-ACK/ACK)

Step 2: Read server RFB version (12 bytes)
  ← 52 46 42 20 30 30 33 2E 30 30 38 0A     "RFB 003.008\n"
  Parse: sscanf(buf, "RFB %03d.%03d\n", &major, &minor)
  Verify: major == 3, minor == 8
  Send same string back:
  → 52 46 42 20 30 30 33 2E 30 30 38 0A     "RFB 003.008\n"

Step 3: Read security types
  ← [count:u8]                                e.g. 01
  ← [type:u8] × count                         e.g. 02  (VNC auth)
  Client MUST select one type and send it back (ATEN always picks the LAST offered):
  → [selected_type:u8]                        e.g. 02

Step 4: Authentication
  ← [challenge:24 bytes]                      Read and DISCARD (not used)
  → [username:24 bytes, null-padded]           e.g. "admin\x00\x00...\x00"
  → [password:24 bytes, null-padded]           e.g. "password\x00...\x00"
  ← [auth_result:u32]                         00 00 00 00 = success
  If auth_result != 0:
    ← [error_len:u32]                         Length of error string
    ← [error_msg:error_len bytes]             Human-readable error
    Connection should be closed.

Step 5: ClientInit
  → [shared_flag:u8]                          00 (exclusive access)

Step 6: Read ServerInit (standard RFB, all discarded by native code)
  ← [fb_width:u16]
  ← [fb_height:u16]
  ← [pixel_format:16 bytes]
     bits_per_pixel:u8, depth:u8, big_endian:u8, true_colour:u8,
     red_max:u16, green_max:u16, blue_max:u16,
     red_shift:u8, green_shift:u8, blue_shift:u8,
     padding:3 bytes
  ← [name_length:u32]
  ← [name:name_length bytes]

Step 7: Read ATEN ServerInit extensions
  ← [padding:4 bytes]                         (skip)
  ← [session_id:u32]
  ← [privilege_config:4 bytes]                (read as 4 individual u8s)
  The session_id and privilege_config are passed to ViewerConfig()
  which stores them in globals (g_session_id, g_config) and calls
  Java setViewerConfig(session_id, config[4]).

  privilege_config byte meanings (verified from Java setViewerConfig):
    [0] = video_access      (0 = denied -> show error and exit)
    [1] = keyboard_mouse    (0 = disabled)
    [2] = kick_ability      (0 = disable kick button)
    [3] = virtual_storage   (0 = disabled)

Step 8: Server sends PrivilegeInfo (type 0x39) -- first server message
  ← 39
  ← [session_word_lo:u32]
  ← [session_word_hi:u32]
  ← [privilege_data:256 bytes]                (read byte-by-byte)
  The 256-byte payload is forwarded to Java privilegeCtrl(lo, hi, data).
  It is NOT used for AES key setup -- the AES key is hardcoded.
  Java interprets it based on session_word_hi:
    hi=1: user joined (data = "<session_id> <username> <ip>")
    hi=2: user left (data = session_id string)
    hi=3: forced logout
    hi=4: session kicked
    hi=7: LED status sync (data[3] bits: b0=NumLock, b1=CapsLock, b2=ScrollLock)

Step 9: Client sends first FramebufferUpdateRequest
  → 03 00 00 00 00 00 00 00 [width:u16] [height:u16]
  (type=0x03, incremental=0, x=0, y=0, width, height)

Step 10: Server sends first FramebufferUpdate (type 0x00)
  ← 00                                     type byte
  ← [skip:3 bytes]                         padding(1) + num_rects(2), skip all 3
  ← [x:u16] [y:u16]                        rectangle position
  ← [w:i16] [h:i16]                        rectangle size (can be negative, abs'd)
  ← [encoding:u32]                         ATEN encoding (0x57-0x61)
  ← [frame_number:u32]                     1 for first frame, 0 thereafter
  ← [data_length:u32]                      compressed data size
  ← [data:data_length bytes]               compressed frame data
  frame_number will be 1 for the very first frame.
  The encoding_type (0x57-0x61) determines which decoder to use.

Main loop: continue exchanging FramebufferUpdateRequests,
KeepAlive ACKs, KeyEvents, PointerEvents, and server messages.

ClientInit

Client sends a single byte:

┌──────────────────────┐
│ shared_flag: u8      │  0x00 = exclusive access
└──────────────────────┘

The ATEN implementation always sends 0x00 (non-shared / exclusive).

ServerInit

Server sends the standard RFB ServerInit followed by ATEN-proprietary extensions.

Confirmed from decompilation of RFBProtocol::ProcServerInit at 0x00118040:

Standard RFB ServerInit (read and discarded by native code):
┌─────────────────────────────────────┐
│ fb_width:         u16 (big-endian)  │  StreamRead16 -- discarded
│ fb_height:        u16              │  StreamRead16 -- discarded
│ pixel_format:     16 bytes         │
│   bits_per_pixel: u8               │  StreamRead8 -- discarded
│   depth:          u8               │  StreamRead8 -- discarded
│   big_endian:     u8               │  StreamRead8 -- discarded
│   true_colour:    u8               │  StreamRead8 -- discarded
│   red_max:        u16              │  StreamRead16 -- discarded
│   green_max:      u16              │  StreamRead16 -- discarded
│   blue_max:       u16              │  StreamRead16 -- discarded
│   red_shift:      u8               │  StreamRead8 -- discarded
│   green_shift:    u8               │  StreamRead8 -- discarded
│   blue_shift:     u8               │  StreamRead8 -- discarded
│   padding:        3 bytes          │  StreamReadSkip(3) -- discarded
│ name_length:      u32              │  StreamRead32 -> uVar1
│ name:             u8[name_length]  │  StreamRead(buf, uVar1) -- read into local_58
└─────────────────────────────────────┘

ATEN Extensions:
┌─────────────────────────────────────┐
│ padding:          4 bytes (skip)    │  StreamReadSkip(4) -- discarded
│ session_id:       u32              │  StreamRead32 -> uVar2
│ aes_key_seed:     u8[4]            │  StreamRead8 x4 -> local_5c..local_59
└─────────────────────────────────────┘

Important: The standard RFB ServerInit fields (width, height, pixel format) are all read and discarded. The native code does not use them -- the actual resolution is determined later from the first FramebufferUpdate.

The ATEN extension fields are used as follows:

1. session_id (u32) is passed to RFBPrivilege::ViewerConfig(session_id, aes_key_seed_ptr) which stores it in global g_session_id via storeViewerConfig()
2. aes_key_seed (4 bytes read individually) is also passed to ViewerConfig which stores it in global g_config via strncpy(&g_config, ptr, 4)

After reading the ATEN extensions, ProcServerInit creates a new RFBPrivilege object (0x120 bytes), calls ViewerConfig (vtable offset 0x28) to store the session_id and key seed, then immediately calls the virtual destructor (vtable offset 0x00) to destroy the temporary privilege object. This temporary RFBPrivilege is separate from the one later created in RMDesktop.

Note: The RFBPrivilege constructor allocates an RFBKMCryto object (0x2110 bytes) at offset 0x118 within the privilege object. This crypto instance is available but the SetCipherKey method is a no-op stub (returns 1 without doing anything), confirming that the actual AES key is entirely hardcoded, not derived from the server-provided key material.

After ServerInit, the server sends a ProcPrivilegeInfo message (type 0x39) containing session/privilege management data. The PrivilegeInfo received immediately after ServerInit has the same format as subsequent PrivilegeInfo messages -- there is no special "first-time" variant.

Privilege Info (from server, post-handshake)

Sent as server message type 0x39 (see PrivilegeInfo in Server-to-Client Messages).

Important: Despite the field name "aes_key_material" in early documentation, the 256-byte payload is NOT used for AES key setup. The AES encryption key is entirely hardcoded in the native code. The 256-byte payload is forwarded to Java's privilegeCtrl callback, which interprets it as session management data (user join/leave notifications, LED sync, forced logout) based on the session_word_hi value.

┌─────────────────────────────────────┐
│ type:             u8 = 0x39         │
│ session_word_lo:  u32              │  Stored at this+0x10
│ session_word_hi:  u32              │  Stored at this+0x14; action code
│ privilege_data:   u8[256]          │  Stored at this+0x18 (read byte-by-byte)
└─────────────────────────────────────┘

session_word_hi action codes (from Java privilegeCtrl):
  1 = user joined session (data = "<session_id> <username> <ip>")
  2 = user left session (data = session_id string)
  3 = forced logout by another user
  4 = session kicked (privilege denied)
  7 = LED status sync (data[3]: bit0=NumLock, bit1=CapsLock, bit2=ScrollLock)

Confirmed from decompilation of RFBPrivilege::ProcPrivilegeInfo at 0x0011a120:

1. StreamRead32() -> store at this+0x10 (session_word_lo)
2. StreamRead32() -> store at this+0x14 (session_word_hi)
3. Compute combined 64-bit value: *(long*)(this+0x10) and compare against 0x400000001 (which is session_word_hi=4, session_word_lo=1 in little-endian)
4. Call SetThreadNormaleStart(result):
   • If (lo, hi) == (0x00000001, 0x00000004): passes 0 (NOT normal start)
   • If (lo, hi) != (0x00000001, 0x00000004): passes 1 (normal start)
5. Read 256 bytes one at a time (StreamRead8() in a loop of 0x100 iterations) into this+0x18 through this+0x117
6. Call virtual method at vtable offset 0x18 (which is ExePrivilegeCtrl())

ExePrivilegeCtrl at 0x0011a1b0 simply calls privilegeControl(lo, hi, key_ptr), which is a JNI helper at 0x00121580 that:

1. Attaches to JVM thread
2. Creates a Java byte array of 256 bytes
3. Copies the 256-byte key material into the Java byte array
4. Calls RemoteVideo.privilegeCtrl(int lo, int hi, byte[] keyData) via JNI

The 256-byte key material is forwarded to Java for AES key setup. However, as noted in the AES encryption section, the actual AES key used for mouse event encryption is hardcoded in the binary (the NIST test vector), so the server-provided key material appears to be used only on the Java side for session management purposes, not for actual cryptographic operations in the native layer.

The SetThreadNormaleStart function at 0x00121640 calls the Java callback RemoteVideo.setNormalStart(int flag) to signal whether this is a normal session start or a reconnection/takeover.

Message Framing

All messages are big-endian unless noted. Each message begins with a 1-byte type identifier. Multi-byte integers are big-endian (network byte order) as per standard RFB, sent through the NtwStream::StreamWrite{8,16,32} helpers.

Server-to-Client Messages

The RFBProtocol::ProtocolHandler() at 0x001183d0 dispatches server messages via a jump table at 0x00127790 supporting types 0x00 through 0x3C (61 entries).

The jump table was fully decoded from the binary. Non-default handlers exist for exactly 7 types:

Type	Hex	Name	Handler	Description
0	0x00	FramebufferUpdate	RFBScreen::ScreenDecode (0x0010E7E8)	Video frame data
4	0x04	CursorPosition	RFBScreen::ScreenCursorPosProc (0x0010E7F8)	Hardware cursor position + shape
22	0x16	KeepAlive Request	(inline: reads 1 byte)	Server ping, expects ACK
53	0x35	KeyboardInfo + MouseInfo	RFBKeyboard::ProcKeyboardInfo (0x0011A720) then RFBMouse::ProcMouseInfo (0x001198C0)	Combined keyboard + mouse config
55	0x37	MouseInfo	RFBMouse::ProcMouseInfo (0x001198C0)	Mouse-only config update
57	0x39	PrivilegeInfo	RFBPrivilege::ProcPrivilegeInfo (0x0011A120)	Session/AES key material
60	0x3C	GetScreenUILang	RFBScreen::GetScreenUILang (0x00118CB0)	OSD language config response

All 54 other type values (1-3, 5-21, 23-52, 54, 56, 58-59, etc.) hit the default handler at 0x001183f5 which simply returns the type byte value to the caller (the Java runImage loop). The byte is not consumed or processed further by the native code.

Corrections from previous analysis: The earlier documentation listed types 0x31 (MouseInfo), 0x33 (KeyboardInfo), 0x35 (PrivilegeInfo), and 0x38 (ATEN Extended) as separate server message types. Binary analysis of the jump table reveals this was incorrect:

• Types 0x31 and 0x33 do NOT exist in the dispatch table
• Type 0x35 handles BOTH keyboard and mouse info in a single message
• Type 0x37 handles mouse info only (a separate mouse-only variant)
• Type 0x38 has NO server-to-client handler (it is client-to-server only: SendKickRequest)
• Type 0x39 is the actual PrivilegeInfo handler (not 0x35)

FramebufferUpdate (type 0x00)

The standard RFB FramebufferUpdate, but with ATEN-proprietary encoding.

Verified from decompilation of RFBScreen::ScreenDecode at 0x00119090 -- exact sequence of StreamRead calls after the type byte:

┌─────────────────────────────────────────────────┐
│ type:           u8 = 0x00                       │  (consumed by ProtocolHandler)
│                                                 │
│ --- ScreenDecode reads the following ---         │
│                                                 │
│ padding:        3 bytes (StreamReadSkip(3))      │  Standard RFB: padding(1) + num_rects(2)
│                                                 │  ATEN always sends 1 rect; client skips all 3
│ x_position:     u16 (StreamRead16)              │
│ y_position:     u16 (StreamRead16)              │
│ width:          i16 (StreamRead16)              │  Stored at RFBScreen+0x10 (as int)
│ height:         i16 (StreamRead16)              │  Stored at RFBScreen+0x14 (as int)
│ encoding_type:  u32 (StreamRead32)              │  Stored at RFBScreen+0x20 (0x57-0x61)
│ frame_number:   u32 (StreamRead32)              │  Stored at RFBScreen+0x50
│ data_length:    u32 (StreamRead32)              │
│ data:           u8[data_length] (StreamRead)    │  Read into compressed_data_buf
└─────────────────────────────────────────────────┘

Total wire bytes after type: 3 + 2 + 2 + 2 + 2 + 4 + 4 + 4 + data_length = 23 + data_length.

Byte-by-byte wire dump (for a frame with encoding 0x58, 640x480):

00                      type = FramebufferUpdate
XX XX XX                3 bytes skipped (padding + num_rects)
00 00                   x_position = 0
00 00                   y_position = 0
02 80                   width = 640
01 E0                   height = 480
00 00 00 58             encoding_type = 0x58 (AST2100+)
00 00 00 01             frame_number = 1 (first frame)
00 01 23 45             data_length = 0x12345 (example)
[data_length bytes]     compressed frame data

Important for implementors: The ATEN server always sends exactly ONE rectangle per FramebufferUpdate. The "3 bytes padding" is actually the standard RFB padding(1) + number_of_rectangles(2) fields, but the native client calls StreamReadSkip(3) -- it does not parse the rectangle count. A client implementation should read these 3 bytes and can expect num_rects to always be 1.

Frame processing logic (verified from decompilation):

1. After reading encoding_type, RMDecoder::GetDecoder() is called to create/retrieve the singleton decoder for this encoding type.
2. x, y, width, height, and encoding are stored into the decoder's info struct.
3. frame_number is read:
   • If frame_number == 1 AND this is the very first frame ever received (RFBScreen+0x50 was 0): set resolution_changed flag, clear cursor flag.
4. data_length is read:
   • If data_length > 0 AND frame_number == 0: decode immediately by calling decoder->Decode(UpdateBlocks_t&). Width/height are abs()'d before decode.
   • If data_length > 0 AND frame_number != 0: set decoder's "deferred decode" flag (decoder offset 0x1c = 1). The actual decode happens on the next call with frame_number == 0.
   • If data_length == 0: no data to read, returns immediately.

KeepAlive Request (type 0x16)

┌───────────────────────────────┐
│ type:         u8 = 0x16       │  (consumed by dispatcher)
│ status:       u8              │  Read and discarded by native code
└───────────────────────────────┘

Total: 2 bytes.

The server sends type 0x16 periodically as a heartbeat. The ProtocolHandler's inline handler at 0x00118488 reads one additional byte via StreamRead8() and discards it (the return value is not stored or checked). The byte's value has no significance to the native code -- it is consumed solely to advance the stream position.

The Java side responds by calling sendKeepAliveAck(), which invokes ProcAlive(1) via vtable offset 0x28. The ACK is always sent with value 1 (true). See KeepAlive ACK below.

CursorPosition (type 0x04)

Server message type 0x04 triggers RFBScreen::ScreenCursorPosProc() at 0x00118fb0 (thunked via 0x0010E7F8), which reads the hardware cursor position and optionally the cursor image data.

Note: Type 0x04 is dual-purpose by direction -- server-to-client is CursorPosition, client-to-server is KeyEvent. They are separate streams.

┌─────────────────────────────────────────────┐
│ type:              u8 = 0x04                │  (consumed by dispatcher)
│ cursor_x:          u32 (BE)                 │  Cursor X position
│ cursor_y:          u32 (BE)                 │  Cursor Y position
│ cursor_width:      u32 (BE)                 │  Cursor image width
│ cursor_height:     u32 (BE)                 │  Cursor image height
│ cursor_type:       u32 (BE)                 │  1 = cursor data follows
│ If cursor_type == 1:                        │
│   compositing_mode:  u32 (BE)               │  0=XOR/overlay, 1=alpha-blend
│   cursor_pixels:     u8[w * h * 2]          │  16bpp ARGB4444 cursor
└─────────────────────────────────────────────┘

Total: 21 bytes minimum (type + 5 x u32), plus variable cursor pixel data if cursor_type == 1. The cursor_type value is NOT stored to the object -- only the consequence matters (whether pixel data was read).

When cursor_type != 1: Only the position is updated. No pixel data is read or stored. The cursor_has_data flag (RFBScreen+0x55) is not set.

When cursor_type == 1: The compositing_mode (stored at RFBScreen+0x2070) determines how the cursor is composited onto the framebuffer:

Cursor pixel format: ARGB4444 (16 bits per pixel, big-endian):

• Bits 15-12: Alpha / control flags
• Bits 11-8: Blue (4 bits)
• Bits 7-4: Green (4 bits)
• Bits 3-0: Red (4 bits)

Compositing mode 0 (XOR/Overlay) -- ASTVideoDecoder::MixedCursor at 0x0010f990: Each 16-bit cursor pixel is processed as follows:

• Bit 15 clear: Direct color. ARGB4444 expanded to 8-bit RGB channels (R = low nibble << 4, G = nibble 1 << 4, B = nibble 2 << 4)
• Bit 15 set, bit 14 clear: Transparent. Background pixel shows through.
• Bits 15 and 14 both set: XOR/Invert. Background pixel channels are bitwise inverted (~byte), creating a visible cursor on any background.

Compositing mode 1 (Alpha blend): True alpha blending using 4-bit alpha:

alpha = (pixel >> 12) & 0xF      // 4-bit alpha (0-15)
R_out = (bg_R * (15 - alpha) + (pixel & 0xF) * 16 * alpha) / 15
G_out = (bg_G * (15 - alpha) + ((pixel>>4) & 0xF) * 16 * alpha) / 15
B_out = (bg_B * (15 - alpha) + ((pixel>>8) & 0xF) * 16 * alpha) / 15

Note: Only ASTVideoDecoder::MixedCursor implements compositing. HermonVideoDecoder, YarkonVideoDecoder, and Pilot3VideoDecoder all have stub implementations that return 1 without doing anything.

KeyboardInfo + MouseInfo (type 0x35)

Type 0x35 is a combined message that carries both keyboard and mouse configuration. The handler calls ProcKeyboardInfo() first, then falls through to ProcMouseInfo().

┌───────────────────────────────┐
│ type:         u8 = 0x35       │  (consumed by dispatcher)
│ kb_encrypt:   u8              │  Keyboard encryption flag (-> kb offset 0x18)
│ kb_type:      u8              │  Keyboard type (-> kb offset 0x14)
│ mouse_encrypt:u8              │  Mouse encryption (0=clear, !0=AES) (-> mouse 0x18)
│ mouse_mode:   u8              │  Mouse mode (-> mouse offset 0x14)
│ mouse_config: u8              │  Mouse additional config (-> mouse offset 0x1c)
└───────────────────────────────┘

Total: 6 bytes.

The ProcKeyboardInfo function (0x0011A720) reads:

• byte 1 (u8) -> stored at RFBKeyboard offset 0x18 (encryption_enabled)
• byte 2 (u8) -> stored at RFBKeyboard offset 0x14 (keyboard_type)
• Sets config_received flag at offset 0x10

The ProcMouseInfo function (0x001198C0) reads:

• byte 1 (u8) -> stored at RFBMouse offset 0x18 (encryption_enabled) Controls mouse event encryption in SendMouse (0=cleartext, non-zero=AES)
• byte 2 (u8) -> stored at RFBMouse offset 0x14 (mouse_mode)
• byte 3 (u8) -> stored at RFBMouse offset 0x1c (additional config)
• Sets config_received flag at offset 0x10

MouseInfo (type 0x37)

Type 0x37 is a mouse-only configuration update. It calls ProcMouseInfo() without the keyboard info prefix.

┌───────────────────────────────┐
│ type:         u8 = 0x37       │  (consumed by dispatcher)
│ mouse_encrypt:u8              │  Mouse encryption (0=clear, !0=AES) (-> offset 0x18)
│ mouse_mode:   u8              │  Mouse mode (-> offset 0x14)
│ mouse_config: u8              │  Mouse additional config (-> offset 0x1c)
└───────────────────────────────┘

Total: 4 bytes. Uses the same ProcMouseInfo handler as type 0x35.

Confirmed from decompilation of RFBMouse::ProcMouseInfo at 0x001198c0:

1. StreamRead8() -> store at this+0x18 as uint (encryption_enabled)
2. StreamRead8() -> store at this+0x14 as uint (mouse_mode)
3. StreamRead8() -> store at this+0x1c as uint (additional config)
4. Set this[0x10] = 1 (config received flag)

Encryption control: The field at RFBMouse offset 0x18 is the encryption enable flag. In SendMouse, the check is if (*(int*)(param_1 + 0x18) != 0) -- when this value is non-zero, mouse events are AES-encrypted; when zero, they are sent in cleartext. This is the first byte read by ProcMouseInfo, which is the server telling the client whether to encrypt mouse events.

PrivilegeInfo (type 0x39)

┌─────────────────────────────────────┐
│ type:             u8 = 0x39         │  (consumed by dispatcher)
│ session_word_lo:  u32              │  Stored at RFBPrivilege+0x10
│ session_word_hi:  u32              │  Stored at RFBPrivilege+0x14
│ aes_key_material: u8[256]          │  Stored at RFBPrivilege+0x18..0x117
└─────────────────────────────────────┘

Total: 265 bytes. Format is identical whether received immediately after ServerInit or later during the session.

The handler (RFBPrivilege::ProcPrivilegeInfo at 0x0011A120):

1. StreamRead32() -> this+0x10 (session_word_lo)
2. StreamRead32() -> this+0x14 (session_word_hi)
3. Compares the combined 64-bit value at this+0x10 against 0x400000001 (little-endian: lo=1, hi=4):
   • If NOT equal: calls SetThreadNormaleStart(1) (normal start)
   • If equal: calls SetThreadNormaleStart(0) (not normal start)
4. Reads 256 bytes one-at-a-time via StreamRead8() in a loop (0x100 iterations), storing each byte at this+0x18 through this+0x117
5. Calls ExePrivilegeCtrl() via vtable offset 0x18, which forwards (session_word_lo, session_word_hi, key_material_ptr) to Java via the privilegeCtrl(int, int, byte[]) JNI callback

The 256-byte key material is forwarded to Java for session/privilege management. The actual AES encryption key used in the native layer is hardcoded (see AES Encryption section).

GetScreenUILang (type 0x3C)

Server response to a client's ProcGetScreenUI request (client sends type 0x3C):

┌───────────────────────────────┐
│ type:         u8 = 0x3C       │  (consumed by dispatcher)
│ lang_id:      u32             │  Language identifier
│ lang_sub:     u32             │  Language sub-identifier
└───────────────────────────────┘

Total: 9 bytes.

The handler (RFBScreen::GetScreenUILang at 0x00118CB0) reads two u32 values and passes them to Java via the getScreenUILangConf callback. Contains debug printf statements ("doris getscreen ui lang", "doris %d %d").

Client-to-Server Messages

All 19 client-to-server message types, verified by searching every StreamWrite8 call in the binary. Each function was decompiled from /liblinux_x86_64__V1.0.5/libiKVM64.so-730126.

Type	Hex	Name	Size	Sender Function (Address)	Description
3	0x03	FramebufferUpdateRequest	10	RFBScreen::ScreenUpdate (0x118850)	Request screen update
4	0x04	KeyEvent	18	RFBKeyboard::Sendkey (0x11a670)	Keyboard event (ATEN extended)
5	0x05	PointerEvent	18	RFBMouse::SendMouse (0x1194f0)	Mouse event (ATEN extended)
7	0x07	MouseSync	3	RFBMouse::MouseSync (0x119730)	Mouse synchronization
8	0x08	MouseReset	2	RFBMouse::MouseReset (0x119780)	Mouse device reset
21	0x15	ScreenSetPosition	9	RFBScreen::ScreenSetPosition (0x1189b0)	Set screen position
22	0x16	KeepAlive ACK	2	RFBProtocol::ProcAlive (0x118230)	Response to server keepalive
23	0x17	ScreenCalibration	2	RFBScreen::ScreenCalibration (0x118960)	Trigger screen recalibration
25	0x19	GetCursorPos	1	RFBScreen::GetCursorPos (0x118d60)	Request cursor position
26	0x1A	SetPowerOnOff	2	RFBScreen::SetPowerOnOff (0x118900)	Power control
50	0x32	ScreenSetInfo	5	RFBScreen::ScreenSetInfo (0x118b80)	Set resolution
53	0x35	KeyboardUpdateInfo	1	RFBKeyboard::KeyboardUpdateInfo (0x11a610)	Request keyboard config refresh
54	0x36	MouseSetPT	3	RFBMouse::MouseSetPT (0x119820)	Set pointer type on server
55	0x37	MouseUpdateInfo	1	RFBMouse::MouseUpdateInfo (0x1197d0)	Request mouse config refresh
56	0x38	SendKickRequest	73	RFBPrivilege::SendKickRequest (0x11a1f0)	Privilege kick request
58	0x3A	MouseHotPlug	1	RFBMouse::MouseHotPlug (0x119910)	Mouse hot-plug reset
59	0x3B	QoS	13	RFBProtocol::ProcQos (0x118290)	Quality of Service params
60	0x3C	GetScreenUI	1	RFBProtocol::ProcGetScreenUI (0x1183a0)	Request OSD language config
61	0x3D	SetScreenUILang	5-9	RFBProtocol::ProcSetScreenUI (0x118320)	Set UI language

FramebufferUpdateRequest (type 0x03)

Standard RFB FramebufferUpdateRequest:

┌─────────────────────────────────┐
│ type:         u8 = 0x03         │
│ incremental:  u8                │  0 = full, 1 = incremental
│ x:            u16               │
│ y:            u16               │
│ width:        u16               │
│ height:       u16               │
└─────────────────────────────────┘

Total: 10 bytes.

KeyEvent (type 0x04) — ATEN Extended

IMPORTANT: Keyboard events are NEVER encrypted. Unlike PointerEvents, the Sendkey() function always sends keyboard events in cleartext. The RFBKeyboard object does allocate an RFBKMCryto instance (at this+0x20), but Sendkey() never calls it. Only mouse events support AES-128-CBC encryption.

┌─────────────────────────────────┐
│ type:         u8 = 0x04         │
│ padding:      u8 = 0x00         │  Always zero (no encryption flag)
│ down_flag:    u8                │  0 = up, 1 = down
│ padding:      2 bytes (zero)    │
│ keycode:      u32               │  USB HID keycode
│ padding:      9 bytes (zero)    │
└─────────────────────────────────┘

Total: 18 bytes. Always sent in cleartext.

The down_flag values:

• 0 = key released (up)
• 1 = key pressed (down)

The keycode field contains USB HID keycodes. The Java client's KeyMap.VKtoHID() maps Java VK codes to HID keycodes before passing them to the native layer. The native processVK() tables documented below show the intermediate X11 keysym representation used internally, but the BMC firmware ultimately interprets the value field as a HID keycode.

Keyboard send flow (RFBKeyboard::Sendkey at 0x0011a670):

1. StreamWriteStart() - begin write batch
2. StreamWrite8(0x04) - type byte
3. Clear the keyboard mode field at this+0x18 to 0
4. StreamWrite8(0x00) - padding byte (always 0, no encryption flag)
5. StreamWrite8(down_flag) - key state
6. StreamWriteSkip(2) - 2 zero bytes padding
7. StreamWrite32(keycode) - the HID keycode value
8. StreamWriteSkip(9) - 9 zero bytes padding
9. StreamWriteFlush() - send

Lock key handling (in keyboardAction JNI at 0x00121000):

For CapsLock (VK 0x14), ScrollLock (VK 0x91), and NumLock (VK 0x90), the JNI entry point reads the local X11 keyboard indicator state via XkbGetIndicatorState():

• Bit 0 (0x01) of indicator state = ScrollLock active
• Bit 1 (0x02) of indicator state = NumLock active
• Bit 2 (0x04) of indicator state = CapsLock active

(Verified from RemoteVideo_init at 0x0011fbd0: & 1 -> scrollLock, & 2 -> numLock, & 4 -> capsLock)

If the corresponding lock is NOT currently active on the local machine, the VK code is modified: the low byte is OR'd with 0xFF to signal a toggle request. For example, CapsLock VK 0x14 becomes 0xFF14 when CapsLock is not active.

The three special VK codes checked are:

• VK 0x14 (CapsLock) -> checks capsLock_status
• VK 0x90 (NumLock) -> checks numLock_status
• VK 0x91 (ScrollLock) -> checks scrollLock_status

After lock key processing, the modified VK code is passed through processVK() which translates it to the final keysym sent on the wire.

Extended key handling (in keyboardAction JNI):

If the extendedKey flag is set in the KeyInfo_t struct, 0x100 is added to both the VK code and the scan code fields. This distinguishes keys like right-Ctrl (VK 0x111) from left-Ctrl (VK 0x11), or numpad-Enter (VK 0x10D) from main Enter (VK 0x0D).

processVK Key Translation (RFBKeyboard::processVK at 0x0011a920)

The processVK() function uses three std::map<int, uint> lookup tables populated in the RFBKeyboard constructor (0x0011abf0). It performs a cascading lookup: Table 1, then Table 2 (with 0x100 toggle fallback), then Table 3. The first non-zero result is used.

Lookup algorithm:

1. If the extendedKey flag is set, add 0x100 to the VK code
2. Table 1 (this+0x28): Look up the VK code directly
   • If found and value != 0, return value & 0xFFFF
3. Table 2 (this+0x58): Look up the VK code (with extended flag)
   • If not found, try with the 0x100 bit toggled (XOR 0x100)
   • If found and value != 0, return value & 0xFFFF
4. Table 3 (this+0x88): Look up the scan code
   • If found and value != 0, return value & 0xFFFF
5. If all lookups return 0, the key event is dropped (keysym = 0)

Special lock key handling in processVK:

Before the table lookups, if iStack0000000000000014 == 0 (action field is 0), three special cases are checked:

• VK 0x14 (CapsLock): returns ~(-(capsLock_status == 0)) + 0x2FFE6
  • If CapsLock active: returns 0x2FFE5 (X11 Caps_Lock keysym)
  • If not active: returns 0x2FFE6
• VK 0x90 (NumLock): returns ~(-(numLock_status == 0)) + 0x2FF80
  • If NumLock active: returns 0x2FF7F (X11 Num_Lock keysym)
  • If not active: returns 0x2FF80
• VK 0x91 (ScrollLock): returns ~(-(scrollLock_status == 0)) + 0x2FF15
  • If ScrollLock active: returns 0x2FF14 (X11 Scroll_Lock keysym)
  • If not active: returns 0x2FF15

Table 1 (this+0x28, 48 entries): Printable character VK codes

Direct mapping of Java VK codes to ASCII character codes:

VK Code	Keysym	Key	VK Code	Keysym	Key
0x20	0x20	Space	0x30-0x39	0x30-0x39	0-9
0x41-0x5A	0x41-0x5A	A-Z	0xBA	0x3A	: (colon)
0xBB	0x2B	+ (plus)	0xBC	0x2C	, (comma)
0xBD	0x5F	_ (underscore)	0xBE	0x2E	. (period)
0xBF	0x2F	/ (slash)	0xC0	0x60	` (backtick)
0xDB	0x5B	[ (bracket)	0xDC	0x5C	\ (backslash)
0xDD	0x5D	] (bracket)	0xDE	0x22	" (quote)

For A-Z and 0-9, the VK code equals the keysym (identity mapping).

Table 2 (this+0x58, 89 entries): Function/control/navigation keys

Maps VK codes (with extended bit) to X11 keysyms:

VK Code	Keysym	Key	VK Code	Keysym	Key
0x08	0xFF08	BackSpace	0x09	0xFF09	Tab
0x0C	0xFF0B	Clear	0x0D	0xFF0D	Return
0x13	0xFF13	Pause	0x1B	0xFF1B	Escape
0x12E*	0xFFFF	Delete	0x124*	0xFF50	Home
0x125*	0xFF51	Left	0x126*	0xFF52	Up
0x127*	0xFF53	Right	0x128*	0xFF54	Down
0x121*	0xFF55	Page_Up	0x122*	0xFF56	Page_Down
0x123*	0xFF57	End	0x029	0xFF60	Select
0x02C	0xFF61	Print	0x02B	0xFF62	Execute
0x12D*	0xFF63	Insert	0x02F	0xFF6A	Help
0x103*	0xFF6B	Break	0x70-0x87	0xFFBE-0xFFD5	F1-F24
0x10D*	0xFF8D	KP_Enter	0x24	0xFF95	KP_Home
0x25	0xFF96	KP_Left	0x26	0xFF97	KP_Up
0x27	0xFF98	KP_Right	0x28	0xFF99	KP_Down
0x21	0xFF9A	KP_Page_Up	0x22	0xFF9B	KP_Page_Down
0x23	0xFF9C	KP_End	0x0C	0xFF9D	KP_Begin
0x2D	0xFF9E	KP_Insert	0x2E	0xFF9F	KP_Delete
0x6A	0xFFAA	KP_Multiply	0x6B	0xFFAB	KP_Add
0x6C	0xFFAC	KP_Separator	0x6D	0xFFAD	KP_Subtract
0x6E	0xFFAE	KP_Decimal	0x16F*	0xFFAF	KP_Divide
0x60-0x69	0xFFB0-0xFFB9	KP_0-KP_9
0x10	0xFFE1	Shift_L	0x10	0xFFE2	Shift_R
0x11	0xFFE3	Control_L	0x111*	0xFFE4	Control_R
0x12	0xFFE9	Alt_L	0x112*	0xFFEA	Alt_R
0x5B	0xFFEB	Super_L	0x5C	0xFFEC	Super_R
0x5D	0xFF67	Menu	0x19	0xFF21	Kanji
0x15	0xFF2E	Hangul

Entries marked with * have the extended bit (0x100) set.

Note: Some VK codes appear twice with different keysyms (e.g., VK 0x10 maps to both Shift_L 0xFFE1 and Shift_R 0xFFE2; VK 0x24 maps to both Home 0xFF95 and KP_Home). The std::map will keep only one entry per key, so the last insert wins during constructor initialization. The non-extended variants (0x24, 0x25, 0x26, 0x27, 0x28, etc.) thus map to the KP_* variants since those are inserted after the navigation key versions.

Table 3 (this+0x88, 32 entries): Scan code based mapping

Maps raw scan codes to keysyms. This table handles multimedia keys, Japanese keyboard keys, and other special keys not covered by Tables 1-2:

Scan Code	Keysym	Description	Scan Code	Keysym	Description
0x15E*	0x8181	JP key	0x15F*	0x8282	JP key
0x163*	0x8383	JP key	0x073	0x8787	JP key
0x070	0x8888	JP Katakana/Hiragana	0x07D	0x8989	JP Yen
0x079	0x8A8A	JP Henkan	0x07B	0x8B8B	JP Muhenkan
0x05C	0x8C8C	JP key	0x0F2	0x9090	Hanja
0x0F1	0x9191	Hangeul	0x078	0x9292	JP key
0x077	0x9393	JP key	0x076	0x9494	JP key
0x119*	0x00B5	Media Next Track	0x110*	0x00B6	Media Prev Track
0x124*	0x00B7	Media Stop	0x122*	0x00CD	Media Play/Pause
0x120*	0x00E2	Volume Mute	0x130*	0x00E9	Volume Up
0x12E*	0x00EA	Volume Down	0x16D*	0x0183	Media Select
0x16C*	0x018A	Mail	0x121*	0x0192	Calculator
0x16B*	0x0194	My Computer	0x165*	0x0221	WWW Search
0x132*	0x0223	WWW Home	0x16A*	0x0224	WWW Back
0x169*	0x0225	WWW Forward	0x168*	0x0226	WWW Stop
0x167*	0x0227	WWW Refresh	0x166*	0x022A	WWW Favorites

Entries marked with * have the extended bit (0x100) set.

PointerEvent (type 0x05) — ATEN Extended

Two formats depending on whether AES encryption is enabled:

Unencrypted (when mouse.encryption_flag == 0):

┌─────────────────────────────────┐
│ type:          u8 = 0x05        │
│ encrypt_flag:  u8 = 0x00        │
│ button_mask:   u8               │
│ x_position:    u16              │
│ y_position:    u16              │
│ padding:       11 bytes (zero)  │
└─────────────────────────────────┘

AES-encrypted (when mouse.encryption_flag != 0):

┌─────────────────────────────────────┐
│ type:          u8 = 0x05            │
│ encrypt_flag:  u8 = 0x01            │
│ encrypted:     u8[16]               │  AES-128-CBC encrypted block
└─────────────────────────────────────┘

Total: 18 bytes in both cases.

Encrypted payload (16 bytes before encryption):

┌─────────────────────────────────────┐
│ button_mask:  u8                    │  byte 0: from param_3 low byte
│ x_position:   u16 (big-endian,      │  bytes 1-2: PsudoStreamSwap16(x)
│                     byte-swapped)    │
│ y_position:   u16 (big-endian,      │  bytes 3-4: PsudoStreamSwap16(y)
│                     byte-swapped)    │
│ random_pad:   11 bytes              │  bytes 5-15: random fill (from 4x rand() calls)
└─────────────────────────────────────┘

The encrypted path (RFBMouse::SendMouse at 0x001194f0):

1. Fills a 16-byte local_48 buffer with random data (4 calls to rand(), each filling 4 bytes = 16 bytes total)
2. Overwrites byte 0 with button_mask (low byte of param_3)
3. Overwrites bytes 1-2 with PsudoStreamSwap16(x_position) (big-endian u16)
4. Overwrites bytes 3-4 with PsudoStreamSwap16(y_position) (big-endian u16)
5. Bytes 5-15 remain as the random data (11 bytes of random padding)
6. Calls EnCryto(local_48, output, 0x10) through the RFBKMCryto vtable at *(this+0x20) offset 0x18 -- this is the RFB_AES128_EventCryto() method
7. Sends: type=0x05, encrypt_flag=0x01, followed by the 16 encrypted bytes
8. StreamWriteFlush() to send

The unencrypted path:

1. Sends: type=0x05, encrypt_flag=0x00
2. StreamWrite8(button_mask) -- 1 byte
3. StreamWrite16(x_position) -- 2 bytes big-endian
4. StreamWrite16(y_position) -- 2 bytes big-endian
5. StreamWriteSkip(11) -- 11 zero bytes padding
6. StreamWriteFlush() to send

Button mask bits:

• Bit 0 (0x01): Left button
• Bit 1 (0x02): Middle button
• Bit 2 (0x04): Right button

Mouse wheel (RFBMouse::MouseAction at 0x00119660):

The MouseAction function receives the full MouseInfo_t struct (button_mask, x, y, wheel_rotation). The wheel rotation value is in the upper 32 bits of param_3 (i.e., param_3 >> 32).

• If wheel_rotation == 0: calls SendMouse() once (normal move/click).
• If wheel_rotation != 0: takes the absolute value of wheel_rotation, then calls SendMouse() twice per unit of wheel rotation (press + release pair). The loop runs abs(wheel_rotation) iterations, each iteration calling SendMouse() two times. The sign of wheel_rotation determines the scroll direction (the button_mask passed to SendMouse encodes scroll-up vs scroll-down as different button bits set by the Java caller).

Total mouse events per wheel tick: 2 (one press, one release). For a wheel rotation of N ticks, 2*|N| PointerEvent messages are sent.

MouseSync (type 0x07)

┌─────────────────────────────────┐
│ type:         u8 = 0x07         │
│ value:        u16 = 0x0780      │  Fixed value (1920 decimal)
└─────────────────────────────────┘

Total: 3 bytes. Confirmed from decompilation of RFBMouse::MouseSync at 0x00119730:

1. StreamWriteStart()
2. StreamWrite8(0x07) -- type byte
3. StreamWrite16(0x0780) -- hardcoded value 1920 (max framebuffer width)
4. StreamWriteFlush()

Sent to synchronize mouse position with the remote host. The value 0x0780 (1920) represents the maximum horizontal resolution and is always sent unchanged regardless of the actual display resolution.

MouseReset (type 0x08)

┌─────────────────────────────────┐
│ type:         u8 = 0x08         │
│ mouse_type:   u8                │  Current mouse type (from this+0x14)
└─────────────────────────────────┘

Total: 2 bytes. Confirmed from RFBMouse::MouseReset at 0x00119780. Sends the current mouse type byte stored at RFBMouse offset 0x14 (received from the server via ProcMouseInfo).

MouseUpdateInfo (type 0x37, client-to-server)

┌─────────────────────────────────┐
│ type:         u8 = 0x37         │
└─────────────────────────────────┘

Total: 1 byte. Sent by RFBMouse::MouseUpdateInfo at 0x001197d0. Clears the "config received" flag (offset 0x10 = 0) and sends type 0x37 to the server to request a fresh MouseInfo update. The server responds with a type 0x37 server-to-client message containing the current mouse configuration.

KeyboardUpdateInfo (type 0x35, client-to-server)

┌─────────────────────────────────┐
│ type:         u8 = 0x35         │
└─────────────────────────────────┘

Total: 1 byte. Sent by RFBKeyboard::KeyboardUpdateInfo at 0x0011a610. Clears the "config received" flag (offset 0x10 = 0) and sends type 0x35 to the server to request a fresh keyboard + mouse info update. The server responds with a type 0x35 server-to-client message containing both keyboard and mouse config.

Note: This is the client-to-server counterpart of the server-to-client type 0x35 (KeyboardInfo + MouseInfo). The type 0x35 is dual-purpose: when sent by the client it is a 1-byte request; when sent by the server it is a 6-byte configuration message.

MouseSetPT (type 0x36)

┌─────────────────────────────────┐
│ type:         u8 = 0x36         │
│ mouse_mode:   u8                │  Mode value (from this+0x18)
│ mouse_type:   u8                │  Type value (from this+0x14)
└─────────────────────────────────┘

Total: 3 bytes. Sent by RFBMouse::MouseSetPT at 0x00119820. This function sets the pointer type on the server side. It first compares the requested mode/type against the current values (offsets 0x18 and 0x14); if both match, the message is suppressed (returns without sending). Otherwise, it stores the new values and sends the type 0x36 message.

KeepAlive ACK (type 0x16)

┌─────────────────────────────────┐
│ type:         u8 = 0x16         │
│ ack:          u8                │  Boolean: always 1 in practice
└─────────────────────────────────┘

Total: 2 bytes. Confirmed from RFBProtocol::ProcAlive at 0x00118230:

StreamWriteStart(ntw);
StreamWrite8(ntw, 0x16);    // type
StreamWrite8(ntw, param_1); // ack value (bool parameter)
StreamWriteFlush(ntw);

The param_1 parameter is typed as bool in the decompilation. The sendKeepAliveAck JNI function at 0x00121a00 always passes 1 (true). A client implementation should send 0x16 0x01 in response to every server KeepAlive.

ScreenSetPosition (type 0x15)

┌─────────────────────────────────┐
│ type:         u8 = 0x15         │
│ x_position:   u32               │  Screen X offset
│ y_position:   u32               │  Screen Y offset
└─────────────────────────────────┘

Total: 9 bytes. Sends a screen position/offset to the server. The x and y values are packed as a 64-bit value split into two 32-bit words. (RFBScreen::ScreenSetPosition at 0x001189b0)

ScreenCalibration (type 0x17)

┌─────────────────────────────────┐
│ type:         u8 = 0x17         │
│ flag:         u8 = 0x01         │
└─────────────────────────────────┘

Triggers a screen recalibration on the BMC.

SetPowerOnOff (type 0x1A)

┌─────────────────────────────────┐
│ type:         u8 = 0x1A         │
│ action:       u8                │  Power action code
└─────────────────────────────────┘

Total: 2 bytes.

Power action codes (confirmed from binary analysis of JNI functions):

Code	Constant	JNI Function	Address	Description
0	POWER_OFF	setPowerOff (0x00121a70)	Calls SetPowerOnOff(0)	Hard power off
1	POWER_ON	setPowerOn (0x00121a50)	Calls SetPowerOnOff(1)	Power on
2	POWER_RESET	setPowerReset (0x00121ab0)	Calls SetPowerOnOff(2)	Hard reset (reboot)
3	SOFT_OFF	setSoftPowerOff (0x00121a90)	Calls SetPowerOnOff(3)	ACPI soft power off

All four JNI functions call SetPowerOnOff(action) through the RFBScreen vtable (vtable index 7, offset 0x38 from vtable base). RFBScreen::SetPowerOnOff at 0x00118900 simply writes: type 0x1A, then the action byte, then flushes.

ScreenSetInfo (type 0x32)

┌─────────────────────────────────┐
│ type:         u8 = 0x32         │
│ width:        u16               │
│ height:       u16               │
└─────────────────────────────────┘

Total: 5 bytes.

Sent when the client detects a resolution change and informs the server. The function (RFBScreen::ScreenSetInfo at 0x00118b80) only sends the message if the resolution has actually changed (compares new height against stored value at RFBScreen offset 0x18). If unchanged, the message is suppressed.

When the resolution changes, it also updates the internal ScreenInfo_t structure (offsets 0x10-0x30) with the new screen parameters before sending.

SendKickRequest (type 0x38)

┌─────────────────────────────────────┐
│ type:         u8 = 0x38             │
│ action:       u32                   │  Kick action code (param_1)
│ reserved:     u32 = 0               │  Always zero (hardcoded)
│ data:         u8[64]                │  Kick data payload (param_2)
└─────────────────────────────────────┘

Total: 73 bytes. Confirmed from decompilation of RFBPrivilege::SendKickRequest at 0x0011a1f0:

1. StreamWriteStart()
2. StreamWrite8(0x38) -- type byte (0x38 = '8')
3. StreamWrite32(param_1) -- action code (from Java sendPrivilegeCtrl)
4. StreamWrite32(0) -- hardcoded zero (reserved field)
5. StreamWrite(param_2, 0x40) -- 64 bytes of kick data
6. StreamWriteFlush()

The sendPrivilegeCtrl JNI function at 0x00121970 extracts a 64-byte array from Java (via GetByteArrayRegion) and passes it along with the action code integer to SendKickRequest through the RFBPrivilege vtable (offset 0x20).

Used for privilege control -- kicking other users off the KVM session. The action code and 64-byte data payload are defined by the Java application layer.

MouseHotPlug (type 0x3A)

┌─────────────────────────────────┐
│ type:         u8 = 0x3A         │
└─────────────────────────────────┘

Total: 1 byte. Confirmed from decompilation of RFBMouse::MouseHotPlug at 0x00119910:

1. StreamWriteStart()
2. StreamWrite8(0x3A) -- type byte (0x3A = ':')
3. StreamWriteFlush()

Single byte message to trigger mouse device re-enumeration on the BMC. No additional payload. Called from the hotPlug JNI method at 0x00121a30.

QoS (type 0x3B)

┌─────────────────────────────────┐
│ type:         u8 = 0x3B         │
│ param1:       u32               │  QoS parameter 1 (video quality)
│ param2:       u32               │  QoS parameter 2 (compression)
│ param3:       u32               │  QoS parameter 3 (bandwidth)
└─────────────────────────────────┘

Total: 13 bytes. Confirmed from decompilation of RFBProtocol::ProcQos at 0x00118290:

1. StreamWriteStart()
2. StreamWrite8(0x3B) -- type byte (0x3B = ';')
3. StreamWrite32(param1) -- first QoS parameter
4. StreamWrite32(param2) -- second QoS parameter
5. StreamWrite32(param3) -- third QoS parameter
6. StreamWriteFlush()

Quality-of-service parameters for video compression tuning. The three u32 parameters are passed directly from the Java setQosParameter JNI method at 0x00121c00, which takes three int arguments from the Java side. The JNI function calls through RMConnection::QosControl at 0x00112b40 (vtable offset 0x40) which delegates to RFBProtocol::ProcQos.

The base class RMProtocol::ProcQos at 0x0011fa80 is a stub (returns 1 without sending anything), so only the RFBProtocol override actually sends the message. The exact semantics of the three parameters are determined by the Java application and BMC firmware (video quality level, compression ratio, and bandwidth limit are the likely interpretations based on the Java UI option frames).

GetScreenUI (type 0x3C)

┌─────────────────────────────────┐
│ type:         u8 = 0x3C         │
└─────────────────────────────────┘

Single byte message requesting the server's current OSD language configuration. The server responds with a type 0x3C server-to-client message containing the lang_id and lang_sub values. (RFBProtocol::ProcGetScreenUI at 0x001183a0)

SetScreenUILang (type 0x3D)

┌─────────────────────────────────┐
│ type:         u8 = 0x3D         │
│ lang_id:      u32               │  Language identifier (param_1)
│ lang_sub:     u32 (conditional) │  Only if FW protocol flag != 0
└─────────────────────────────────┘

Total: 5 bytes (without lang_sub) or 9 bytes (with lang_sub).

Confirmed from decompilation of RFBProtocol::ProcSetScreenUI at 0x00118320:

1. StreamWriteStart()
2. StreamWrite8(0x3D) -- type byte (0x3D = '=')
3. StreamWrite32(param_1) -- language ID (always sent)
4. If this[0x48] != 0 (the fw_protocol_flag at RFBProtocol offset 0x48): StreamWrite32(param_2) -- language sub-ID (conditionally sent)
5. StreamWriteFlush()

The FW protocol flag (this[0x48]) is a single byte set by RFBProtocol::SetFWProtocol(uchar) at 0x00118530. This flag is set from the Java side through the vtable. When the flag is zero (default), only 5 bytes are sent (type + lang_id). When non-zero, 9 bytes are sent (type + lang_id + lang_sub).

The setScreenUILang JNI method at 0x00121b00 takes two int parameters from Java and calls ProcSetScreenUI(param_3, param_4) through the RFBProtocol vtable (offset 0x30).

The corresponding getScreenUILang JNI method at 0x00121b50 calls ProcGetScreenUI() through the vtable (offset 0x38), which sends a 1-byte type 0x3C request. The server responds with a type 0x3C message.

Video Encoding

Encoding Types

The encoding_type field in FramebufferUpdate identifies the BMC chipset's video compression format:

ID	Hex	Decoder Class	BMC Chipset
87	0x57	ASTVideoDecoder	ASPEED AST2050
88	0x58	ASTVideoDecoder	ASPEED AST2100/2300/2400/2500
89	0x59	HermonVideoDecoder	Nuvoton WPCM450 (Hermon)
96	0x60	YarkonVideoDecoder	Yarkon
97	0x61	Pilot3VideoDecoder	ServerEngines Pilot III

AST Video Compression (0x57, 0x58)

ASPEED BMCs use a proprietary JPEG-like compression scheme:

• AST2050 (0x57): Uses the ast2100 decoder class with DCT, Huffman coding, and VQ (Vector Quantization) decompression. The decoder allocates ~1MB (0x1029E8 bytes) of state.

• AST2100+ (0x58): Uses the ast_jpeg decoder class. Pure JPEG-variant with standard DCT/IDCT, Huffman decoding, and YUV→RGB color conversion. Allocates 0x588 bytes of state.

Both decoders:

• Accept the compressed data buffer and produce RGBA pixel output
• Support incremental updates (dirty rectangles via UpdateBlocks_t)
• Track resolution changes and report them back to the Java layer via changeResolution callback

Decoder Selection (RMDecoder::GetDecoder at 0x00112560)

The GetDecoder function creates a decoder based on the encoding type from the FramebufferUpdate. It is a singleton -- once created, it is reused:

Encoding	Decoder Class	Alloc Size
0x57	ASTVideoDecoder	0x58
0x58	ASTVideoDecoder	0x58
0x59	HermonVideoDecoder	0x58
0x60	YarkonVideoDecoder	0x60
0x61	Pilot3VideoDecoder	0x458

GetDecoder is a singleton factory -- once a decoder is created, it is cached in the global DecoderHandle and reused for all subsequent frames. If the encoding type does not match any of 0x57, 0x58, 0x59, 0x60, or 0x61, the function returns NULL (there is no default/fallback decoder). A NULL decoder pointer will cause ScreenDecode to skip decoding (it checks lVar9 != 0 before accessing the decoder). The switch statement has no default case.

ASTVideoDecoder::Decode (at 0x0010f8b0) then creates the inner decoder:

• Encoding 0x58: allocates ast_jpeg (0x588 bytes)
• Encoding 0x57: allocates ast2100 (0x1029E8 bytes)

AST JPEG Wire Format (encoding 0x58)

The compressed data blob within a FramebufferUpdate has a 16-byte header followed by a stream of tile command words.

Header (first 16 bytes of the data blob, read as big-endian u64 words):

Offset  Size  Description
0x00    8     Header word 0 (big-endian u64): total compressed size (bits [55:0])
0x08    8     Header word 1 (big-endian u64): parameters

The byte-swap in the decoder reads the first two 8-byte words from the buffer and reinterprets them as big-endian 64-bit integers.

Header word 1 bit fields:

Bits	Field	Description
[31]	Advance mode	If set, negates the advance counter
[10:8]	Y quantization level	Luminance quant (3 bits)
[22:21]	CbCr quant level	Chrominance quant (3 bits)
[20:16]	CbCr tile size	Chrominance tile size (5 bits, 0 = 16)
[28:24]	Y tile size	Luminance tile size (5 bits, 0 = 16)

If Y tile size == 0, defaults to 16x16 tiles. If CbCr tile size == 0, defaults to 16x16 tiles. The tile size controls the IDCT block dimensions.

After the header, additional data at offset 0x10 provides initial tile positions and the bitstream data.

Tile command stream:

After the header, the data is processed as a stream of tile commands. Each command word is read from the bitstream with the top 2 bits indicating the command type:

Bits [31:30]	Type	Description
0 (0b00)	Same position, alt QT	Decode tile at current position with QT_TableSelection=2
1 (0b01)	Same position	Decode tile at current position with QT_TableSelection=0
2 (0b10)	New position, alt QT	Jump to new position, decode with QT_TableSelection=2
3 (0b11)	New position	Jump to new position, decode with QT_TableSelection=0

For types 2 and 3 (new position), the tile position is embedded in the command word:

Bits	Field	Description
[29:23]	tile_x	X tile index (7 bits)
[22:16]	tile_y	Y tile index (7 bits)

For types 0 and 1 (same position), the previous tile position is reused and only 2 bits are consumed from the bitstream for the command.

Tile decoding (per tile, 6 Huffman-coded DCT blocks):

Each tile is decoded as a 4:2:0 YCbCr macroblock:

1. 4 luminance (Y) blocks (8x8 each, stored at byTileYuv+0x00, +0x40, +0x80, +0xC0):

   • process_Huffman_data_unit() -> get_DCT() -> IDCT_transform(QT=0)

2. 1 Cb block (8x8, stored at byTileYuv+0x100):

   • process_Huffman_data_unit() -> get_DCT() -> IDCT_transform(QT=1)

3. 1 Cr block (8x8, stored at byTileYuv+0x140):

   • process_Huffman_data_unit() -> get_DCT() -> IDCT_transform(QT=1)

Total: 6 blocks per tile = 4Y + 1Cb + 1Cr (standard 4:2:0 JPEG subsampling)

YUV to RGB conversion (ast_jpeg::YUVToRGB at 0x0011e4e0):

After IDCT, the 6 decoded blocks are in byTileYuv buffer (0x180 = 384 bytes). The YUVToRGB() function converts the tile from YCbCr to 32bpp RGBA and writes it into the framebuffer at the tile position.

Screen output (ast_jpeg::ScreenResolution at 0x0011e770):

Two modes based on the tile_size field at this+0x564:

• 16x16 tile mode: 256 pixels per tile, written as 16bpp to the output buffer with color scaling using the pixel format masks
• Non-16 mode: 256 pixels per tile, written as 32bpp (BGRX) to the output buffer with a boundary check against 0x7E9000 bytes (framebuffer limit)

Tile index advancement (ast_jpeg::MoveBlockIndex at 0x0011e700):

After decoding each tile:

• Increment X tile index by 1
• If X >= (aligned_width / tile_size): wrap to X=0, increment Y tile index
• If Y >= (aligned_height / tile_size): reset to Y=0
• Track total decoded pixels in a counter (incremented by 0x100 = 256 per tile)
• Stop when counter >= total pixels (header word 0 << 8)

Huffman tables:

The decoder uses fixed Huffman tables (not embedded in the stream):

• DC luminance table: DC_LUMINANCE_HUFFMANCODE at 0x001304C0
• DC chrominance table: DC_CHROMINANCE_HUFFMANCODE at 0x00130480
• AC luminance table: AC_LUMINANCE_HUFFMANCODE at 0x001303E0
• AC chrominance table: AC_CHROMINANCE_HUFFMANCODE at 0x00130320
• General Huffman constants: HUFFMANCODE at 0x0012FAA0

These are standard JPEG Huffman tables stored in a pre-computed format.

Quantization:

Quantization levels are set per-frame from header word 1 bits. The decoder_fun::set_QuantLevel() function configures three QT table selections (0, 1, 2) based on these parameters. QT_TableSelection=0 is used for standard luminance, QT_TableSelection=1 for chrominance, and QT_TableSelection=2 for an alternate quality level (used in command types 0 and 2).

AST2050 Video Compression (encoding 0x57)

The AST2050 uses the ast2100 class (~1 MB state, 0x1029E8 bytes) with a hybrid VQ (Vector Quantization) + Huffman + IDCT decoder, optionally preceded by RC4 decryption.

Data blob layout (within FramebufferUpdate):

The first 4 bytes of the compressed data blob are parsed by SetOptions():

Offset  Size  Description
0x00    1     Mode byte 0 (param_3, maps to YUV/grayscale mode)
0x01    1     Mode byte 1 (param_4, maps to quality level)
0x02    2     Chip revision (big-endian u16, byte-swapped via PsudoStreamSwap16)
0x04    ...   Compressed tile bitstream data

Chip revision (2 bytes at offset 0x02):

• 0x01A6: AST2050 mode - tile size 16x16, YUV 4:2:0 mode (mode_flag=1)
• 0x01BC: AST2050 alternate - tile size 8x8, grayscale mode (mode_flag=0)

After the 4-byte header, the remainder is the compressed bitstream.

RC4 decryption (conditional):

If the encrypt_flag (this+0x101f6d) is set to 1, the entire data payload (after the 4-byte header) is decrypted with RC4 before tile decoding.

• RC4 key: The hardcoded string "fedcba9876543210" (at address 0x00230500), expanded to 256 bytes by Keys_Expansion() which repeats the key cyclically
• RC4 state: Standard RC4 KSA (Key Scheduling Algorithm) implemented in DecodeRC4_setup(), producing a 256-entry S-box
• Decryption: RC4_crypt() XORs the data with the RC4 PRGA stream

The RC4 encryption appears to be optional obfuscation of the video stream data to prevent casual interception. The key is hardcoded and not derived from any session material.

Tile bitstream format:

The bitstream is processed as a sequence of 32-bit command words. The top 4 bits (uVar6 >> 28, called "nibble code") select the command type:

Nibble	Description
0x0	Huffman+IDCT decode at current position (no QT change)
0x4	Huffman+IDCT decode with alternate QT (param_4=2)
0x5	VQ decode, 1 color cache entry (no position update)
0x6	VQ decode, 2 color cache entries (no position update)
0x7	VQ decode, 4 color cache entries (no position update)
0x8	Set new tile position + Huffman+IDCT decode
0x9	End of frame (return immediately)
0xC	Set new tile position + Huffman+IDCT alternate QT
0xD	Set new tile position + VQ 1-entry decode
0xE	Set new tile position + VQ 2-entry decode
0xF	Set new tile position + VQ 4-entry decode

For position-setting commands (0x8, 0xC, 0xD, 0xE, 0xF), the position is encoded in the command word:

Bits [27:20]  = Y block coordinate (row)
Bits [19:12]  = X block coordinate (column)

Then 20 bits are consumed from the bitstream (4 nibble + 8+8 position bits).

For non-position commands (0x0, 0x4, 0x5, 0x6, 0x7), only 4 bits are consumed for the command nibble.

VQ (Vector Quantization) decompression (VQ_Decompress at 0x0011d490):

The COLOR_CACHE is a 4-entry palette (each entry is a YCbCr triple: Y, Cb, Cr). Initialized by VQ_Initialize() with default values:

• Entry 0: Y=0x00, Cb=0x80, Cr=0x80 (black)
• Entry 1: Y=0xFF, Cb=0x80, Cr=0x80 (white)
• Entry 2: Y=0x80, Cb=0x80, Cr=0x80 (gray)
• Entry 3: Y=0xC0, Cb=0x80, Cr=0x80 (light gray)

For VQ commands, additional bits from the bitstream update the color cache:

• Bit 31 set: A new 24-bit color (YCbCr) is read from the bitstream and stored into the cache slot indicated by bits [30:29] (2-bit index)
• Bit 31 clear: Only a 3-bit cache index is consumed, selecting an existing color from the cache

VQ fills 64 pixel values (an 8x8 block) by either:

• Flat fill (cache entry 0x24 == 0): fill all 64 pixels with one color
• Per-pixel indexing (cache entry 0x24 != 0): read cache indices from bitstream

Huffman+IDCT decompression (Decompress at 0x0011d150):

Two modes based on mode_flag (this+0x101f6f):

• Mode 0 (8x8 tiles): 1 Y block + 1 Cb block + 1 Cr block (YCbCr 4:4:4)
• Mode 1 (16x16 tiles): 4 Y blocks + 1 Cb block + 1 Cr block (YCbCr 4:2:0)

Each block is Huffman-decoded via process_Huffman_data_unit() and then IDCT transformed via IDCT_transform(). The IDCT output is 64 coefficients per 8x8 block (standard JPEG-like DCT).

YUV to RGB conversion (ast2100::YUVToRGB at 0x0011cec0):

Uses pre-computed lookup tables stored in the ast2100 object:

• Cr_tab (offset 0x044): Cr to R contribution
• Cb_tab (offset 0x444): Cb to G contribution
• Cr_Cb_green_tab (offset 0x844): combined CrCb to G contribution
• Y_tab (offset 0x1044): Y to luminance
• rlimit_table (offset 0x101a60): clamping/range-limit table

Output format: 32bpp BGRX (byte order: [0]=unused, [1]=B, [2]=G, [3]=R)

Tile advancement (MoveBlockIndex at 0x0011d590):

After each tile is decoded, the block position advances:

• Mode 0 (8x8): X increments by 1; wraps at (width / 8), then Y increments
• Mode 1 (16x16): X increments by 1; wraps at (width / 16), then Y increments

Dirty rectangles (x, y, w, h) are recorded into the UpdateBlocks_t array:

• Mode 0: each block is 8x8 pixels
• Mode 1: each block is 16x16 pixels

Frame termination: The loop terminates when either:

• Nibble code 0x9 is encountered (explicit end marker)
• The iteration counter exceeds (width * height) / 64 (minimum 0x1000)

Hermon Video Compression (0x59)

Nuvoton WPCM450 BMCs use the HermonVideoDecoder with raw pixel tiles and simple pixel format conversion.

Multi-session check: If encoding_type == 0x10001 (multi-session flag), the decoder throws an exception: "Not Support Multi-Session!"

Data blob wire format (HermonVideoDecoder::Decode at 0x0010fee0):

Offset  Size  Description
0x00    1     Frame type: 0x01 = full frame, other = incremental
0x01    1     Color depth flag: 0x00 = 16bpp (RGB555), non-zero = 8bpp (palette)
0x02    4     Tile count (big-endian u32, bytes [2..5])
0x06    4     Total data length (big-endian u32, bytes [6..9])
0x0A    ...   Tile data (format depends on frame_type)

Full frame (frame_type == 0x01):

• The entire tile data starting at offset 10 contains raw pixel data for the complete screen
• ConvertVierwerPixelFormat() converts from source bpp to 32bpp BGRX
• The converted data is copied to both the working and display framebuffers
• UpdateBlocks reports a single full-screen rectangle (0xFFFF, 0xFFFF, 16, 16)

Incremental update (frame_type != 0x01): Each tile in the stream has a 6-byte header followed by raw pixel data:

Per-tile format:
Offset  Size  Description
0x00    4     Reserved / unknown
0x04    1     Tile Y position (in units of 16 pixels)
0x05    1     Tile X position (in units of 16 pixels)
0x06    N     Raw pixel data (N = 0x200 for 16bpp or 0x100 for 8bpp)

N depends on the color depth flag:

• 16bpp (flag == 0): N = 512 bytes (16x16 pixels x 2 bytes each)
• 8bpp (flag != 0): N = 256 bytes (16x16 pixels x 1 byte each)

Each tile is passed through ConvertVierwerPixelFormat() and then SetRect() copies the converted 16x16 tile into the framebuffer at (x16, y16).

Pixel format conversion (ConvertVierwerPixelFormat at 0x0010fd60):

Two modes based on the color_depth_flag parameter:

16bpp mode (RGB555, flag == 0): Input is 2 bytes per pixel

Input:  byte[0] = GGGBBBBB, byte[1] = xRRRRRGG
Output: [0]=0x00, [1]=B<<3, [2]=(byte[1]<<6)|(byte[0]&0xE0)>>2, [3]=(byte[1]&0x7C)<<1

This is RGB555 -> BGRX32 conversion.

8bpp mode (2-bit-per-channel palette, flag != 0): Input is 1 byte per pixel

Input:  byte = xxRRGGBB
Output: [0]=0x00, [1]=BB<<6, [2]=GG<<4, [3]=RR<<2

This is a simple 2-bit-per-channel to 8-bit expansion.

SetRect (HermonVideoDecoder::SetRect at 0x0010fe30):

Copies a decoded 16x16 tile (64 bytes per row, 16 rows) into the framebuffer at position (x * 64_bytes, y * pitch * 16). The pitch is clamped to min(screen_width, 1280).

Yarkon Video Compression (0x60)

The Yarkon decoder uses a combination of Huffman compression, RLE compression, and RFB Hextile-style encoding.

Multi-session check: Same as Hermon -- throws "Not Support Multi-Session!"

Data blob wire format (YarkonVideoDecoder::Decode at 0x00110630):

The data blob starts with a 1-byte compression indicator:

Offset  Size  Description
0x00    1     Compression type: 0xFF = Huffman, 0xFE = RLE, other = raw

Huffman-compressed (byte[0] == 0xFF):

0x00    1     0xFF marker
0x01    4     Compressed size (big-endian u32)
0x05    4     Decompressed size (big-endian u32)
0x09    ...   Huffman-compressed payload

The payload is decompressed via Huffman_Uncompress() into a working buffer before tile parsing.

RLE-compressed (byte[0] == 0xFE):

0x00    1     0xFE marker
0x01    4     Compressed size (big-endian u32)
0x05    4     (unused for RLE, but present for alignment)
0x09    ...   RLE-compressed payload

Decompressed via RLE_Uncompress().

Raw/uncompressed (other values): Data is used directly.

Decompressed tile stream format:

After decompression, the tile stream has:

Offset  Size  Description
0x00    1     Frame type (0x00 = full frame, other = incremental)
0x01    2     Tile count (big-endian u16, bytes [1..2])
0x03    4     Data length (big-endian u32, bytes [3..6])
0x07    ...   Per-tile records

Per-tile record:

0x00    1     Tile X position (in tiles)
0x01    1     Tile Y position (in tiles)
0x02    2     Tile pixel data length (big-endian u16)
0x04    N     Hextile-encoded tile data

Each tile is decoded via HextileDecoder() which implements the RFB Hextile encoding:

Hextile sub-encoding flags (first byte of tile data):

Bit	Flag	Description
0	Raw	Tile is raw pixel data (no hextile encoding)
1	BackgroundSpecified	2-byte background color follows
2	ForegroundSpecified	2-byte foreground color follows
3	AnySubrects	Subrectangle count byte follows
4	SubrectsColoured	Each subrect has its own 2-byte color

When bit 0 (Raw) is set, the tile is raw 16bpp pixel data, converted via ConvertVierwerPixelFormat() and placed via SetRect().

Otherwise, the tile is filled with the background color, then subrectangles (if present) are drawn on top. Each subrect is encoded as:

[optional 2-byte color]  -- if SubrectsColoured flag set
1 byte: (x << 4) | y    -- position within tile (4 bits each)
1 byte: (w << 4) | h    -- size within tile (4 bits each, value+1)

Yarkon pixel format conversion is identical to Hermon's 16bpp mode (RGB555): input is 2 bytes per pixel, output is 4 bytes BGRX.

Full frame indicator: When frame_type == 0x00 (first byte of decompressed stream), the UpdateBlocks reports a single full-screen update (0xFFFF, 0xFFFF).

RLE_Uncompress (RLE_Uncompress at 0x0011a520):

Simple RLE with an escape character (first byte of input):

• Literal bytes (not equal to escape): copied directly
• Escape + count (0..2): output count+1 copies of the escape character itself
• Escape + count (3..127) + value: output count+1 copies of value
• Escape + count (128..255) + value: count is extended to 16 bits ((count & 0x7F) << 8 | next_byte), then output count+1 copies of value

Pilot3 Video Compression (0x61)

ServerEngines Pilot III BMCs use the Pilot3VideoDecoder with a complex multi-mode compression scheme supporting 1bpp, 2bpp, 4bpp, 8bpp, 16bpp, and 32bpp color depths, and both full-frame and incremental tile updates.

Data blob wire format (Pilot3VideoDecoder::Decode at 0x00110d20):

Offset  Size   Description
0x00    1      Frame type: 0x01 = full, other = incremental
0x01    1      Color depth code (bpp_code): 1,2,3,4,5 (see below)
0x02    1      Sub-mode: 0x00 = standard, 0x02 = planar, other = uncompressed
0x06    4      Total data length (big-endian u32, bytes [6..9])

Color depth codes (byte at offset 0x01):

Code	Bits per pixel	Pixel format
1	8bpp	Palette-indexed (uses 1024-byte palette)
2	16bpp	RGB555
3	Uncompressed	Raw pixel data
4	32bpp	BGRX (4 bytes per pixel)
5	8bpp variant	8bpp with palette at offset 0x0A

Full frame (frame_type == 0x01):

For bpp_code==1 or bpp_code==5: A 1024-byte (256-entry x 4 bytes) color palette is read at offset 0x0A, stored at this+0x30.

The pixel data follows and is decompressed through a custom RLE scheme with escape markers:

• Byte 0x55 is the primary escape:
  • 0x55 0x00: literal 0x55
  • 0x55 0x01: literal 0xAA
  • 0x55 count value: repeat value count+1 times
• Byte 0xAA is the secondary escape:
  • 0xAA value: repeat value 3 times

The decompressed data is further processed through planar-to-chunky conversion for 32bpp and 16bpp modes, reconstructing interleaved pixel data from separate bitplanes.

For bpp_code==4 (32bpp), each pixel has 4 planes; for bpp_code==2 (16bpp), each pixel has 2 planes.

Incremental update (frame_type != 0x01):

Tile records start after a tile-count header:

Offset   Size  Description
0x0A     2     Tile count (big-endian u16)
0x0C+N*4 ...   Per-tile headers (4 bytes each)

Per-tile header (for sub-modes 2 and 4):

0x00     1     Tile row (Y)
0x01     1     Tile column (X)
0x02     1     Tile width (in pixels)
0x03     1     Tile height (in pixels)

Each tile's compressed data follows, using the same RLE scheme (0x55/0xAA escapes) with 4-byte-aligned chunk boundaries. A 4-byte "chunk length" header precedes each compressed segment.

Planar mode (sub-mode == 0x00 with bpp_code 2 or 4):

In this mode, the data is stored as bitplanes. After RLE decompression, the decoder reconstructs chunky pixel data from the planes using bit manipulation. For 32bpp, 4 planes are combined into BGRA; for 16bpp, 2 planes are combined into RGB555.

Sub-mode 0x02 (planar with change-list):

A special full-frame mode where the data contains a change-list. Each entry specifies an 8x16 character cell position and a bitmask indicating which of the 16 rows have changed. Each bit in the bitmask corresponds to a row; if set, that row's pixels (8 pixels wide) are set to white (0xFF,0xFF,0xFF,0x00).

Pilot3 pixel format conversion (ConvertVierwerPixelFormat at 0x00110a30):

Five modes based on the format byte:

Format	Input	Conversion
0x04	4bpp	Each byte = 2 pixels (low nibble, high nibble)
		Palette lookup through 1024-byte table at this+0x30
0x08	8bpp	Each byte = 1 pixel, palette lookup at this+0x30
0x10	16bpp	RGB565 -> BGRX32 (B<<3, G<<5
0x20	32bpp	ARGB -> BGRX (byte swap: [3]->[0], [0]->[1], etc)

For palette modes (0x04, 0x08), the output pixel is constructed from 4 consecutive bytes in the palette table at this+0x30 + index*4.

SetRect (Pilot3VideoDecoder::SetRect at 0x00110c20):

Copies a decoded 32x32 tile (128 bytes per row, 32 rows) into the framebuffer. Tile size is larger than Hermon/Yarkon (32x32 vs 16x16).

Framebuffer Format

The decoded framebuffer uses:

• 32 bits per pixel (BGRX in memory)
• Color masks: R=0x000000FF, G=0x0000FF00, B=0x00FF0000
• Max resolution: 1920 × 1200
• Default resolution: 640 × 480

The framebuffer is allocated as 0x600000 bytes (6,291,456 = 1920 × 1200 × ~3.4, rounded up, or simply a large fixed buffer).

The Java side wraps this in a DataBufferInt → WritableRaster → BufferedImage for rendering.

AES-128-CBC Encryption

Mouse events can be optionally encrypted using AES-128-CBC when enabled by the server. Keyboard events are never encrypted -- they are always sent in cleartext despite the protocol having an RFBKMCryto instance allocated for the keyboard object.

The encryption is controlled per-input-type:

• Mouse: Encrypted when *(int*)(RFBMouse + 0x18) != 0. This field is encryption_enabled (set from the first byte read by ProcMouseInfo -- the server tells the client whether to encrypt). In SendMouse at 0x001194f0, the check is if (*(int*)(param_1 + 0x18) != 0) -- when non-zero, encryption is enabled; when zero, events are sent in cleartext. The RFBMouse constructor initializes offset 0x18 to 0 (unencrypted by default).
• Keyboard: Never encrypted. RFBKeyboard::Sendkey() does not check any encryption flag and always sends in cleartext.

Hardcoded Key

Key: 2B 7E 15 16 28 AE D2 A6 AB F7 15 88 09 CF 4F 3C

Initialization Vector

IV:  00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F

Additional Key Schedule Data

The RFBKMCryto class contains additional hardcoded key schedule material:

Block 1: 8E 73 B0 F7 DA 0E 64 52 C8 10 F3 2B 80 90 79 E5
Block 2: 62 F8 EA D2 52 2C 6B 7B
Block 3: 60 3D EB 10 15 CA 71 BE 2B 73 AE F0 85 7D 77 81
Block 4: 1F 35 2C 07 3B 61 08 D7 2D 98 10 A3 09 14 DF F4

These are AES-128 expanded round key constants. The encryption uses SW_AES_CBC() — a software AES-CBC implementation.

Encryption Flow

Mouse events only (RFBMouse::SendMouse at 0x001194f0):

When encryption_enabled (RFBMouse offset 0x18, set by server via ProcMouseInfo first byte) is non-zero:

1. Fill a 16-byte buffer with random data using rand() (4 calls to rand(), filling 4 x 4-byte words)
2. Place button_mask (1 byte) at offset 0
3. Place byte-swapped x_position (2 bytes) at offset 1-2 via PsudoStreamSwap16()
4. Place byte-swapped y_position (2 bytes) at offset 3-4 via PsudoStreamSwap16()
5. Bytes 5-15 remain as random data (11 bytes of random padding)
6. Call EnCryto(plaintext_16, ciphertext_16, 0x10) through the RFBKMCryto vtable
   • This calls RFB_AES128_EventCryto() which: a. Sets up the hardcoded AES key and IV on the stack (see above) b. Calls SW_AES_CBC(mode=0, keysize=0, input, blocks=1, key, output, iv) c. mode=0 = encryption, blocks=1 = one 16-byte block
7. Send: type=0x05, encrypt_flag=0x01, followed by the 16 encrypted bytes
8. StreamWriteFlush() to send

When encryption_enabled is zero (unencrypted):

1. Send: type=0x05, encrypt_flag=0x00, button_mask, x_position (u16), y_position (u16)
2. StreamWriteSkip(11) - 11 zero bytes padding
3. StreamWriteFlush() to send

SW_AES_CBC implementation (RFBKMCryto::SW_AES_CBC at 0x0010e2e8):

Parameters: (this, mode, keysize, input, num_blocks, key, output, iv)

• mode=0: Encrypt. XOR each block with IV (first block) or previous ciphertext, then aes_encrypt(). Standard CBC encrypt.
• mode=1: Decrypt. aes_decrypt() each block, then XOR with IV (first block) or previous ciphertext. Standard CBC decrypt.
• keysize: Passed as keysize * 64 + 128 to aes_set_key() (for keysize=0, this is 128 bits).

Keyboard events (RFBKeyboard::Sendkey at 0x0011a670):

Keyboard events are always sent unencrypted. Despite having an RFBKMCryto* at RFBKeyboard offset 0x20, the Sendkey() function never references it. The message is simply: type 0x04, padding 0x00, down_flag, 2 zero bytes, keycode (u32), 9 zero bytes.

Connection Flow (Java Side)

Startup

1. KVMMain.main() parses arguments: IP, username, password, KVM_port, hostname, VM_port (default 623), company_id, board_id, language
2. Static initializer loads native libraries: System.loadLibrary("iKVM64") and System.loadLibrary("SharedLibrary64")
3. RMConnection.init(ConnInfo.class, UserInfo.class) initializes JNI field IDs
4. RMConnection.keepActive(connInfo) establishes TCP connection:
   • Creates RMConnection object
   • Calls RFBProtocol::InitHandShake() which: a. Creates NtwStream and establishes TCP connection via NtwStream::Connect() b. Runs ProcVersion() (version exchange) c. Runs ProcSecurity() (security type negotiation)
5. RMConnection.checkValidUser(userInfo) authenticates:
   • JNI layer (checkValidUser at 0x00119C60) extracts username (max 24 chars) and password (max 96 chars, but only 24 sent) from Java UserInfo object
   • Calls RMConnection::CheckVaildUser() which delegates to RFBProtocol::Authenticate(): a. Reads 24-byte challenge from server (appears to be discarded) b. Sends 24-byte username (null-padded) c. Sends 24-byte password (null-padded) d. Reads auth result (u32): 0 = success, else reads error message e. On success: runs ProcClientInit() then ProcServerInit()
6. On success, creates Viewer which initializes the GUI

Video Loop

The Viewer creates a RemoteVideo panel, which starts three threads:

1. CatchThread: Calls catchLoop() → native RFBProtocol::ProtocolHandler() in a loop, processing all incoming server messages

2. DecodeThread: Calls runImage() in a loop:

   • Calls RFBProtocol::ProtocolHandler() to receive next frame
   • Calls ScreenDecode() to decompress the video data
   • Calls ScreenGetFreame() to get decoded pixels
   • Triggers changeResolution callback on resolution changes
   • Reports dirty rectangles via addClipBounds

3. LazyWorker: Periodic tasks including:

   • refresh() → sends FramebufferUpdateRequest
   • updateImage() → copies framebuffer to Java BufferedImage
   • FPS calculation and display

Input Flow

Keyboard: processKeyEvent() -> KeyMap.VKtoHID(vkCode, location) -> keyboardAction(virtualKey, scanKey, extendedKey, action, 0, 0, 0) -> native JNI at 0x00121000 (lock key check via X11 LED query) -> RFBKeyboard::KeyboardAction(KeyInfo_t) -> RFBKeyboard::processVK(KeyInfo_t) (VK -> X11 keysym via 3 lookup maps) -> RFBKeyboard::Sendkey(keysym, downFlag) (always cleartext, type 0x04)

Mouse: mousePressed/Released/Moved/Dragged() -> doMouseAction() -> mouseAction(x, y, buttonMask, wheelRotation) -> JNI Java_..._mouseAction at 0x001212d0 (null-checks desktop pointer) -> RFBMouse::MouseAction() at 0x00119660 via vtable (offset 0x10 of desktop+8):

• If wheelRotation == 0: single SendMouse() call
• If wheelRotation != 0: abs(wheelRotation) iterations, each calling SendMouse() twice (press + release pair) -> RFBMouse::SendMouse() at 0x001194f0 (AES-encrypted if mouse_mode != 0, cleartext if mouse_mode == 0, type 0x05)

Mouse Modes

Three mouse modes controlled by the server:

• Normal (mode 0): Sends absolute coordinates, client syncs mouse position before each action with MouseSync
• Absolute (mode 1): Sends absolute coordinates directly
• Single (mode 2): Relative mode — sends delta from center of screen, Java side warps mouse cursor back to center after each event

Class Hierarchy (Native)

TcpSocket
  └─ NtwStream (buffered I/O over TCP)

RMProtocol (virtual base)
  └─ RFBProtocol
       ├─ InitHandShake()    → TCP connect + ProcVersion + ProcSecurity
       ├─ Authenticate()     → ATEN auth + ProcClientInit + ProcServerInit
       ├─ ProcVersion()
       ├─ ProcSecurity()
       ├─ ProcClientInit()
       ├─ ProcServerInit()
       ├─ ProtocolHandler()  ← main message dispatch (jump table 0x00-0x3C)
       ├─ ProcAlive()
       ├─ ProcQos()
       ├─ ProcSetScreenUI()
       ├─ ProcGetScreenUI()
       └─ SetFWProtocol()

RMScreen → RFBScreen
  ├─ ScreenUpdate()         → send FramebufferUpdateRequest
  ├─ ScreenDecode()         → receive and decode FramebufferUpdate (type 0x00)
  ├─ ScreenCursorPosProc()  → handle cursor position (type 0x04)
  ├─ ScreenGetRect()        → copy rect from framebuffer
  ├─ ScreenSetInfo()        → send resolution (client type 0x32)
  ├─ ScreenSetPosition()    → send position (client type 0x15)
  ├─ ScreenCalibration()    → trigger recalibration
  ├─ SetPowerOnOff()        → power control
  ├─ GetScreenUILang()      → handle OSD lang response (type 0x3C)
  └─ GeFrontGround()        → get foreground/cursor compositing

RMKeyboard → RFBKeyboard
  ├─ Sendkey()              → send KeyEvent message
  ├─ processVK()            → translate VK/HID to keysym
  ├─ ProcKeyboardInfo()     → handle server keyboard config
  └─ KeyboardSetType()      → set keyboard type

RMMouse → RFBMouse
  ├─ SendMouse()            → send PointerEvent message (type 0x05)
  ├─ MouseAction()          → process mouse action (+ wheel repeat)
  ├─ MouseSync()            → send MouseSync message (type 0x07, value 0x0780)
  ├─ MouseReset()           → send MouseReset (type 0x08, mouse_type)
  ├─ MouseUpdateInfo()      → request mouse config (client type 0x37)
  ├─ MouseGetPT()           → get pointer type info (16 bytes from this+0x10)
  ├─ MouseSetPT()           → set pointer type (type 0x36, mode+type)
  ├─ MouseHotPlug()         → send hot-plug reset (type 0x3A)
  └─ ProcMouseInfo()        → handle server mouse config (3 bytes)

RMPrivilege → RFBPrivilege
  ├─ ProcPrivilegeInfo()    → handle privilege/AES key data (type 0x39)
  ├─ ExePrivilegeCtrl()     → forward privilege data to Java
  ├─ ViewerConfig()         → store session_id + config
  └─ SendKickRequest()      → kick user (type 0x38)

RMDecoder
  ├─ ASTVideoDecoder (0x57, 0x58)
  │   ├─ ast2100  (AST2050 — VQ + Huffman + IDCT)
  │   └─ ast_jpeg (AST2100+ — JPEG variant + YUV→RGB)
  ├─ HermonVideoDecoder (0x59)
  ├─ YarkonVideoDecoder (0x60)
  └─ Pilot3VideoDecoder (0x61)

RFBKMCryto
  └─ AES-128-CBC encryption for mouse events only (keyboard is always cleartext)

RMConnection
  ├─ ConnKeepActive()       → establish connection
  ├─ CheckVaildUser()       → authenticate
  └─ QosControl()           → QoS settings

RMDesktop (composition)
  ├─ RFBKeyboard  (offset 0x00)
  ├─ RFBMouse     (offset 0x08)
  ├─ RFBScreen    (offset 0x10)
  └─ RFBPrivilege (offset 0x18)

Wire Format Summary

All multi-byte values are big-endian (network byte order).

Client-to-Server Quick Reference

KeyEvent (18 bytes, always cleartext):
  04 00 [down] 00 00 [keycode:4] 00 00 00 00 00 00 00 00 00

PointerEvent unencrypted (18 bytes):
  05 00 [buttons] [x:2] [y:2] 00 00 00 00 00 00 00 00 00 00 00

PointerEvent encrypted (18 bytes):
  05 01 [aes_encrypted:16]

FramebufferUpdateRequest (10 bytes):
  03 [incremental] [x:2] [y:2] [w:2] [h:2]

MouseSync (3 bytes):
  07 07 80                    (hardcoded 0x0780 = 1920)

MouseReset (2 bytes):
  08 [mouse_type]

ScreenSetPosition (9 bytes):
  15 [x_pos:4] [y_pos:4]

KeepAlive ACK (2 bytes):
  16 01                     (always ack=1 in ATEN implementation)

ScreenCalibration (2 bytes):
  17 01                     (flag always 0x01)

GetCursorPos (1 byte):
  19

SetPowerOnOff (2 bytes):
  1A [action]  (0=off, 1=on, 2=reset, 3=soft-off)

ScreenSetInfo (5 bytes):
  32 [width:2] [height:2]

MouseSetPT (3 bytes):
  36 [mouse_mode] [mouse_type]

KeyboardUpdateInfo (1 byte):
  35                          (request keyboard+mouse config from server)

MouseUpdateInfo (1 byte):
  37                          (request fresh mouse config from server)

SendKickRequest (73 bytes):
  38 [action:4] [reserved:4=0] [data:64]

MouseHotPlug (1 byte):
  3A

QoS (13 bytes):
  3B [param1:4] [param2:4] [param3:4]

GetScreenUI (1 byte):
  3C

SetScreenUILang (5-9 bytes):
  3D [lang_id:4] [lang_sub:4 if FW flag]

Server-to-Client Quick Reference

FramebufferUpdate (type 0x00):
  00 [pad:3] [x:2] [y:2] [w:2] [h:2] [encoding:4] [frame:4] [len:4] [data:len]

CursorPosition (type 0x04):
  04 [cursor_x:4] [cursor_y:4] [width:4] [height:4] [cursor_type:4]
  If cursor_type == 1: [extra:4] [pixels:width*height*2]

KeepAlive (type 0x16):
  16 [status:1]

KeyboardInfo + MouseInfo (type 0x35):
  35 [kb_encrypt] [kb_type] [mouse_encrypt] [mouse_mode] [mouse_config]

MouseInfo (type 0x37):
  37 [mouse_encrypt] [mouse_mode] [mouse_config]

PrivilegeInfo (type 0x39):
  39 [session_lo:4] [session_hi:4] [key_data:256]

GetScreenUILang (type 0x3C):
  3C [lang_id:4] [lang_sub:4]

RFBScreen Internals

RFBScreen Object Layout

The RFBScreen object is large (~0x2080 bytes) and contains the framebuffer management state, cursor data, and decoder references:

Offset   Size   Type             Field                 Notes
0x00     8      vtable*          vtable ptr            Points to 0x0022f8d0
0x08     8      RMProtocol*      protocol              Network stream access
0x10     4      int              screen_width          Current width in pixels
0x14     4      int              screen_height         Current height in pixels
0x18     4      int              stored_width          Width for change detection
0x1C     4      int              stored_height         Height for change detection
0x20     4      uint             encoding_type         Current encoding (0x57-0x61)
0x28     8      uint64           update_blocks         Dirty rect tracking
0x30-0x37       ...              (screen info fields)
0x38     8      uchar*           framebuffer           Main decoded pixel buffer
0x40     8      uchar*           compressed_data_buf   Receive buffer for encoded data
0x48     8      uchar*           cursor_buffer         Cursor pixel data (Java ByteBuffer)
0x50     4      int              frame_number          Frame sequence counter
0x54     1      byte             resolution_changed    Flag: need to notify Java
0x55     1      byte             cursor_data_valid     Flag: cursor data available
0x56     varies  uchar[]         cursor_pixel_data     Inline cursor image storage
0x2058   4      uint             cursor_x              Hardware cursor X position
0x205C   4      uint             cursor_y              Hardware cursor Y position
0x2060   4      int              cursor_width          Cursor image width
0x2064   4      int              cursor_height         Cursor image height
0x2070   4      uint             cursor_extra          Additional cursor info
0x2078   8      RMDecoder*       current_decoder       Active video decoder instance

ScreenDecode (0x00119090) -- Video Frame Receive

This function handles incoming FramebufferUpdate (type 0x00) messages. The ProtocolHandler reads the type byte; ScreenDecode reads the rest:

1. Skip 3 bytes (standard RFB padding + num_rects, always 1)
2. Read x_position (u16), y_position (u16)
3. Read width (i16) -> stored at this+0x10; height (i16) -> stored at this+0x14
4. Read encoding_type (u32) -> stored at this+0x20
5. Call RMDecoder::GetDecoder(compressed_data_buf, framebuffer) to get/create the appropriate decoder. This is a singleton -- only one decoder is ever created. If encoding_type is unknown (not 0x57-0x61), returns NULL and no decoding occurs.
6. Store x, y, width, height, encoding into the decoder's info struct (offsets 0x08-0x18)
7. Read frame_number (u32):
   • If frame_number == 1 and this is the first frame (this+0x50 == 0): set resolution_changed flag (this+0x54 = 1), clear cursor flag (this+0x55 = 0)
8. Store frame_number at this+0x50
9. Read data_length (u32)
10. If data_length > 0:
    • Make width/height absolute (negate if negative)
    • Read data_length bytes into the compressed_data_buf (this+0x40)
    • Clear update_blocks (this+0x28 = 0)
    • If frame_number == 0 (not first frame): call decoder->Decode(UpdateBlocks_t) to decompress update stored_width/height from decoder's output dimensions
    • If frame_number != 0 (first frame): set decoder's "deferred decode" flag (decoder offset 0x1c = 1)

ScreenUpdate (0x00118850) -- Send FramebufferUpdateRequest

Sends a standard RFB FramebufferUpdateRequest (type 0x03). The ScreenReqInfo_t parameter struct is passed by value on the stack:

ScreenReqInfo_t layout (on stack):
  offset 0x00: u16 x_position
  offset 0x04: u16 y_position
  offset 0x08: u16 width
  offset 0x0C: u16 height
  offset 0x10: u8  incremental_flag

Wire bytes:

1. Write type 0x03
2. Write incremental flag (u8) -- 0=full update, 1=incremental
3. Write x (u16), y (u16), width (u16), height (u16)
4. Flush
5. If resolution_changed flag (this+0x54) is set:
   • Call MixedCursor() through vtable[0x60] to composite cursor
   • Clear the resolution_changed flag (this+0x54 = 0)

ScreenGetFreame (0x00118d40) -- Get Framebuffer Pointer

Simply returns the framebuffer pointer at this+0x38. This is the decoded pixel data that the Java side maps as a DirectByteBuffer.

ScreenGetRect (0x00118a20) -- Copy Rectangle from Framebuffer

Copies a rectangular region from the framebuffer (this+0x38) to an output buffer. Performs bounds checking against screen_width and screen_height. Copies row by row using memcpy, with stride = screen_width * 4 (32bpp).

ScreenSetFrameBuff (0x00118d50) -- Set Framebuffer Pointers

Sets the framebuffer pointer (this+0x38) only if it hasn't been set yet (is NULL). Always sets the cursor buffer pointer (this+0x48). Called during initialization to establish the shared memory between Java ByteBuffers and the native decoder.

ScreenCursorPosProc (0x00118fb0) -- Cursor Position Update

Handles server message type 0x04. Reads:

1. cursor_x (u32) -> this+0x2058
2. cursor_y (u32) -> this+0x205C
3. cursor_width (u32) -> this+0x2060
4. cursor_height (u32) -> this+0x2064
5. cursor_type (u32):
   • If cursor_type == 1: reads cursor_extra (u32), then reads width * height * 2 bytes of 16bpp cursor pixel data into this+0x56 Sets cursor_data_valid flag
6. If frame_number indicates first frame received (this+0x50 != 1):
   • Calls MixedCursor() through vtable to composite cursor onto framebuffer
   • Calls getQuickCursor() JNI callback to notify Java

GeFrontGround (0x00118da0) -- Get Cursor Composite Buffer

Returns the cursor buffer (this+0x48) with metadata prepended:

• Writes frame_number, cursor_x, cursor_y, cursor_width, cursor_height (byte-swapped to big-endian) into the first 20 bytes of the cursor buffer
• If cursor data is valid and frame_number != 0: allocates a temporary buffer, copies the cursor rect via ScreenGetRect, calls decoder's MixedCursor(), then frees the temp buffer
• Returns pointer to the cursor buffer (or 0 if no framebuffer)

Network I/O Layer

NtwStream Object Layout (0x628 bytes)

Confirmed from constructor at 0x0010efb0:

Offset   Size   Field                    Description
0x000    0x28   pthread_mutex_t mutex     Write mutex (pthread_mutex_init)
0x028    0x5F0  uint8_t write_buf[1520]   Internal write buffer
0x618    0x08   uint8_t *write_cursor     Current write position (init: this+0x28)
0x620    0x08   TcpSocket *socket         Underlying socket object

Reads are NOT buffered -- every StreamRead call goes directly to TcpSocket::read() via virtual dispatch (vtable +0x18). The 1520-byte buffer is exclusively for writes.

Write batching: StreamWriteStart() locks the mutex, subsequent StreamWrite*() calls accumulate into the buffer via memcpy, and StreamWriteFlush() sends all buffered data in a single write() call and unlocks the mutex. This ensures protocol messages are sent atomically.

If a StreamWrite() would overflow the buffer, the existing buffer is flushed first, then the data is sent directly to the socket.

StreamReadSkip/StreamWriteSkip: Both allocate a temporary heap buffer, zero-fill it, read/write through it, then free it. Wasteful but functional.

Endianness: All StreamRead16/32 and StreamWrite16/32 functions use big-endian (network byte order). Confirmed from decompilation:

• StreamRead32 returns (b0 << 24) | (b1 << 16) | (b2 << 8) | b3
• StreamWrite32 writes val>>24, val>>16, val>>8, val

TcpSocket Object Layout (0x60 bytes)

Confirmed from constructor at 0x00117290:

Offset   Size   Field                    Description
0x00     0x08   void **vtable             Virtual function table
0x08     0x34   char hostname[52]         Hostname (copied via strcpy)
0x3C     0x04   int port                  Port number
0x40     0x04   int record_flag           Debug: write recv'd data to file
0x44     0x04   int playback_flag         Debug: read from file instead of socket
0x48     0x08   FILE *record_file         Recording file handle
0x50     0x08   FILE *playback_file       Playback file handle
0x58     0x04   int socket_fd             Connected socket file descriptor
0x5C     0x04   int is_server             1=server (listen/accept), 0=client

Socket options set during connect (CreateScok at 0x001179c0):

• SO_SNDBUF = 0x40000 (256 KB)
• SO_RCVBUF = 0x40000 (256 KB)
• No TCP_NODELAY -- enableNagles() is a no-op stub (returns 1)
• No SO_KEEPALIVE set

Connection uses getaddrinfo() + socket() + connect() with ai_socktype = SOCK_STREAM, ai_flags = AI_PASSIVE.

Network error handling: Both TcpSocket::read() and write() use select() with a 30-second timeout (tv_sec = 0x1e). On error or timeout, they throw C++ exceptions (ErrMsg, 0x44 bytes: int error_code + char[64] message):

Error Code	Message	Condition
-1	"Read: Socket closed"	recv() returns 0 or negative
-1	"Socket Read Failed"	select() returns negative
-2	"Socket Read Timeout"	select() timeout (30s)
-1	"Write: Socket closed"	send() returns negative
-2	"Socket Write Timeout"	select() timeout (30s)

SIGPIPE (signal 13) is handled by BrokenPipe_handle which prints a message and returns (prevents JVM crash on broken pipe).

Connection Teardown

There is NO graceful disconnect message. The connection is torn down by:

1. Java_..._RemoteVideo_destory() at 0x0011ffd0 (note: typo "destory")
2. Nulls the global desktop and connection pointers
3. Destroys RMDesktop -> destroys sub-objects (keyboard, mouse, screen, privilege)
4. Destroys RMConnection -> destroys RFBProtocol -> destroys NtwStream
5. NtwStream::~NtwStream -> destroys TcpSocket
6. TcpSocket::EndSock() at 0x00117aa0:
   • shutdown(socket_fd, SHUT_RDWR) (2 = both directions)
   • close(socket_fd)

A client implementation can simply close the TCP socket. No protocol-level goodbye message is needed.

Notable Implementation Details

• PsudoStreamSwap16() performs byte-swapping for 16-bit values (host→network or network→host order conversion)
• The native code uses XkbGetIndicatorState() to read X11 keyboard LED state for CapsLock/NumLock/ScrollLock synchronization
• The processVK() function uses three std::map<int, uint> lookup tables to translate VK codes to X11 keysyms, falling through from table 1 → 2 → 3
• Video frame decode is done in-place into the shared framebuffer memory mapped by the Java ByteBuffer
• The sendViewerConfig() function sends a 4-byte g_config value and a g_session_id to the Java side via setViewerConfig callback
• Keyboard events are always cleartext -- only mouse events support AES encryption
• The GetCursorPos() function at 0x00118d60 sends type 0x19 to request cursor position from the server (an undocumented client-to-server message type)
• The setMouseMode JNI method at 0x001214a0 calls MouseSetPT (vtable offset 0x30) followed by MouseUpdateInfo (vtable offset 0x38) to configure the mouse mode on the server
• The changeLEDstate JNI method at 0x00120cd0 does NOT send any network message. It is purely local X11 manipulation: it opens an X11 display, simulates a key press+release for CapsLock (VK 0x39 -> keysym 0xFFE5), ScrollLock (VK 0x47 -> keysym 0xFF14), or NumLock (VK 0x53 -> keysym 0xFF7F) using XTestFakeKeyEvent, then closes the display. This synchronizes the client's local keyboard LED state with the remote server's state.
• The updateInfo JNI method at 0x00120be0 calls MouseUpdateInfo() via vtable on the RFBMouse object (desktop+0x08, vtable offset 0x28), which sends a type 0x37 MouseUpdateInfo message to request fresh mouse configuration.
• The changeScreenInfo JNI method at 0x00120c00 calls ScreenGetRect() then ScreenSetInfo() via vtable on the RFBScreen object (desktop+0x10). If the resolution has changed, ScreenSetInfo sends a type 0x32 message with the new width/height. It also refreshes the JNI frame buffer references.
• The doCatch JNI method at 0x00121570 simply sets the global filterFlag variable and returns. It does NOT process any messages.
• The catchLoop JNI method at 0x001216c0 is a no-op (empty function body). The actual message receive loop is driven by runImage, not catchLoop.
• The refresh JNI method at 0x00121510 does JNI buffer management (MonitorEnter/MonitorExit on frameObj) and resets the flag global. It does NOT send a FramebufferUpdateRequest on the wire -- that is done by the Java side calling ScreenUpdate() through the vtable.

Error Handling

ErrMsg Exception Class

The native library uses C++ exceptions (__cxa_throw) for error propagation. The ErrMsg class (RTTI at 0x0022df70, typeinfo-name at 0x00122415) is the single exception type used throughout libiKVM64.so.

ErrMsg structure (0x44 bytes = 68 bytes):

Offset  Size  Type     Field
0x00    4     int      error_code
0x04    64    char[]   error_message (null-terminated, strncpy'd with max 0x40)

Error Codes

Error codes are negative integers (stored as the first 4 bytes of the ErrMsg):

Code	Decimal	Source	Message
0xFFFFFFFF	-1	TcpSocket::read	"Read: Socket closed"
0xFFFFFFFF	-1	TcpSocket::read	"Socket Read Failed"
0xFFFFFFFF	-1	TcpSocket::write	"Write: Socket closed"
0xFFFFFFFF	-1	TcpSocket::write	"Socket Write Failed"
0xFFFFFFFE	-2	TcpSocket::read	"Socket Read Timeout"
0xFFFFFFFE	-2	TcpSocket::write	"Socket Write Timeout"
0xFFFFFFFD	-3	HermonVideoDecoder::Decode	"Not Support Multi-Session!"
0xFFFFFFFD	-3	YarkonVideoDecoder::Decode	"Not Support Multi-Session!"
0xFFFFFFFC	-4	TcpSocket::read	"Read: File closed"
0xFFFFFFFB	-5	TcpSocket::read	"Read: File maybe curropted" [sic]

Error Propagation to Java

Errors are propagated from native code to Java via the errorHandler JNI callback. The callback is registered during RemoteVideo.init with signature (I)V -- it takes a single integer parameter (the error code).

Registration (in Java_tw_com_aten_ikvm_ui_RemoteVideo_init at 0x0011fbd0):

errorHandlerMid = GetMethodID(class, "errorHandler", "(I)V")

The errorHandlerMid global (at BSS 0x04923730) stores the Java method ID. If registration fails (returns 0), the init function prints "Java_tw_com_aten_ikvm_RemoteVideo_init failed" and continues.

Exception flow:

1. Socket I/O errors in TcpSocket::read/write throw ErrMsg exceptions via __cxa_throw(exception, &ErrMsg::typeinfo, 0)
2. Video decoder errors (e.g., multi-session not supported) also throw ErrMsg
3. The runImage JNI function (0x001201e0) has try/catch blocks (CatchHandler at 0x00120515) that catch these exceptions
4. The catch handler extracts the error code and calls errorHandler(code) on the Java RemoteVideo object via JNI CallVoidMethod
5. The Java errorHandler method handles cleanup (closing connections, showing error dialogs, etc.)

Note: The checkValidUser and keepActive JNI functions also have try/catch blocks for ErrMsg exceptions at their respective catch handler addresses (0x00119e36 and 0x00119be8), but their catch handlers return false rather than calling errorHandler -- the error is indicated by the boolean return value to the Java caller.

Socket Timeout

The socket timeout for both reads and writes is 30 seconds (tv_sec = 0x1e). TcpSocket::read and TcpSocket::write use select() with a 30-second timeout. If select() returns 0 (timeout), an ErrMsg with code -2 is thrown. If select() returns negative (error), an ErrMsg with code -1 is thrown.

Minimal Client Implementation Guide

This section summarizes the minimum protocol interactions needed for a working KVM client. All multi-byte integers are big-endian (network byte order).

(a) Establishing a KVM Session

1. TCP connect to BMC KVM port (e.g., 5900)

2. Version exchange:
   recv(12)  -> "RFB 003.008\n"
   send(12)  -> "RFB 003.008\n"

3. Security negotiation:
   recv(1)   -> count
   recv(count) -> security_type bytes
   send(1)   -> last security_type received (pick any; ATEN ignores it)

4. Authentication:
   recv(24)  -> challenge (read and discard)
   send(24)  -> username (null-padded to 24 bytes)
   send(24)  -> password (null-padded to 24 bytes)
   recv(4)   -> auth_result (u32, 0 = success)
   If auth_result != 0:
     recv(4) -> error_len
     recv(error_len) -> error message string
     ABORT

5. ClientInit:
   send(1)   -> 0x00 (shared_flag = exclusive)

6. ServerInit (read and discard):
   recv(2)   -> fb_width (u16, discard)
   recv(2)   -> fb_height (u16, discard)
   recv(16)  -> pixel_format (discard all)
   recv(4)   -> name_length (u32)
   recv(name_length) -> name (discard)

7. ATEN ServerInit extensions:
   recv(4)   -> skip (padding)
   recv(4)   -> session_id (u32, store for reference)
   recv(1)   -> aes_key_seed[0]
   recv(1)   -> aes_key_seed[1]
   recv(1)   -> aes_key_seed[2]
   recv(1)   -> aes_key_seed[3]

8. Server sends PrivilegeInfo (type 0x39):
   recv(1)   -> 0x39
   recv(4)   -> session_word_lo (u32)
   recv(4)   -> session_word_hi (u32)
   recv(256) -> aes_key_material (read byte-by-byte or bulk)
   (Store these for session management; native AES key is hardcoded)

9. Session is now established. Begin main loop.

(b) Receiving and Decoding Video

The main loop receives server messages and processes them:

LOOP:
  recv(1) -> type_byte

  SWITCH type_byte:
    0x00 (FramebufferUpdate):
      recv(3)   -> skip (padding + num_rects)
      recv(2)   -> x (u16)
      recv(2)   -> y (u16)
      recv(2)   -> width (i16, take abs)
      recv(2)   -> height (i16, take abs)
      recv(4)   -> encoding_type (u32)
      recv(4)   -> frame_number (u32)
      recv(4)   -> data_length (u32)
      if data_length > 0:
        recv(data_length) -> compressed_data
        if frame_number == 0:
          decode(compressed_data, encoding_type, x, y, width, height)
        else:
          // First frame: store resolution, defer decode to next frame
      // After processing, send FramebufferUpdateRequest for next frame:
      send: 03 01 00 00 00 00 [width:2] [height:2]

    0x04 (CursorPosition):
      recv(4) -> cursor_x (u32)
      recv(4) -> cursor_y (u32)
      recv(4) -> cursor_width (u32)
      recv(4) -> cursor_height (u32)
      recv(4) -> cursor_type (u32)
      if cursor_type == 1:
        recv(4) -> cursor_extra (u32)
        recv(cursor_width * cursor_height * 2) -> cursor_pixels (16bpp)

    0x16 (KeepAlive):
      recv(1)  -> status (discard)
      // Send ACK:
      send: 16 01

    0x35 (KeyboardInfo + MouseInfo):
      recv(1) -> kb_mode
      recv(1) -> kb_type
      recv(1) -> mouse_mode     (0=normal, 1=absolute, 2=single)
      recv(1) -> mouse_type
      recv(1) -> mouse_extra    (encryption flag)

    0x37 (MouseInfo):
      recv(1) -> mouse_mode
      recv(1) -> mouse_type
      recv(1) -> mouse_extra

    0x39 (PrivilegeInfo):
      recv(4)   -> session_word_lo
      recv(4)   -> session_word_hi
      recv(256) -> aes_key_material

    0x3C (GetScreenUILang):
      recv(4) -> lang_id
      recv(4) -> lang_sub

    DEFAULT (types > 0x3C or unhandled):
      // Unknown type: the native code returns the type byte to the
      // caller without consuming further data. This is DANGEROUS --
      // if the server sends an unexpected type, the stream will
      // desynchronize. A robust client should disconnect on unknown types.

Encoding types (the encoding_type field in FramebufferUpdate):

Encoding	Decoder	BMC Chipset	Notes
0x57	AST2050 (VQ+DCT)	ASPEED AST2050	Optional RC4 decryption layer
0x58	AST JPEG	ASPEED AST2100+	Most common; JPEG-like 4:2:0
0x59	Hermon	Nuvoton WPCM450	Raw 16bpp/8bpp tiles
0x60	Yarkon	Yarkon	Huffman/RLE + Hextile
0x61	Pilot3	Pilot III	Multi-mode RLE + planar

For most modern ATEN/ASPEED BMCs, expect encoding 0x58. See the "AST JPEG Wire Format" section for full decoder details.

(c) Sending Keyboard Input

Keyboard events are always sent in cleartext (never encrypted):

send: 04 00 [down_flag:1] 00 00 [keycode:4] 00 00 00 00 00 00 00 00 00

Total: 18 bytes. The keycode is a USB HID keycode (e.g., 0x28 for Enter, 0x04 for 'A'). The down_flag is 0 for key-up and 1 for key-down.

Example -- pressing and releasing the 'a' key (HID keycode 0x04):

send: 04 00 01 00 00 00 00 00 04 00 00 00 00 00 00 00 00 00   (key down, keycode=0x04)
send: 04 00 00 00 00 00 00 00 04 00 00 00 00 00 00 00 00 00   (key up, keycode=0x04)

(d) Sending Mouse Input

Mouse events support two formats. In normal mode (mouse_mode == 0), events are unencrypted:

send: 05 00 [button_mask:1] [x:2] [y:2] 00 00 00 00 00 00 00 00 00 00 00

Total: 18 bytes. Button mask bits: 0x01=left, 0x02=middle, 0x04=right.

In absolute/single mode (mouse_mode != 0), events are AES-encrypted:

Plaintext (16 bytes): [button_mask:1] [x:2 big-endian] [y:2 big-endian] [random:11]
Encrypt with AES-128-CBC:
  Key: 2B 7E 15 16 28 AE D2 A6 AB F7 15 88 09 CF 4F 3C
  IV:  00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
send: 05 01 [ciphertext:16]

Total: 18 bytes either way.

Mouse wheel: For each wheel tick, send TWO PointerEvents (press + release) with the appropriate scroll button bit set in button_mask.

Object Allocation Sizes (for Implementation Reference)

All sizes verified from operator_new calls in constructors:

Class	Alloc Size	Hex	Notes
TcpSocket	96	0x60	Underlying TCP socket
NtwStream	1,576	0x628	Buffered network I/O (includes mutex, 1520-byte write buffer)
RFBProtocol	112	0x70	Protocol state, sub-object pointers
RMConnection	16	0x10	Vtable + RFBProtocol pointer
RMDesktop	48	0x30	Composition of keyboard+mouse+screen+privilege
RFBKeyboard	184	0xB8	Includes 3 std::map lookup tables
RFBMouse	40	0x28	Includes RFBKMCryto pointer
RFBScreen	8,320	0x2080	Large: cursor data, framebuffer refs
RFBPrivilege	288	0x120	256-byte key material + RFBKMCryto pointer
RFBKMCryto	8,464	0x2110	AES state: key schedule, S-boxes
ErrMsg	68	0x44	Exception: int error_code + char[64]

Additionally, the RFBScreen constructor allocates a 0x600000-byte (6 MB) compressed data buffer for incoming encoded video frames.

Disconnection

There is no graceful disconnect message in this protocol. To disconnect:

1. Simply close the TCP socket (shutdown + close)
2. The server detects the disconnect via TCP RST/FIN

The server will also disconnect the client if KeepAlive ACKs stop arriving (timeout behavior depends on BMC firmware implementation).